Skip to main content

7-6 Solution Implementation


Approving Plans and Results of Contractor HF Empirical Evaluations. The FAA HF Coordinator (HFC) ensures that appropriate HF empirical evaluations are planned and completed. 

  • Step 10. Providing input and approval for the plans developed by the Contractor HFC for cognitive walkthroughs, Early User Interface Events (EUIEs) and other appropriate methods for testing and evaluation (FAA Testing), including the data collection instruments for the evaluation of features in the GUI/CHI and associated software functionality in the context of the proposed workflow and procedures, as well as compatibility of the GUI/CHI and associated software functionality with the broader hardware and software environment

Scenario Selection. The FAA HFC reviews and approves the scenario selection and design for use in testing and evaluation. Since only a limited number of scenarios can be selected for evaluation in cognitive walkthroughs and EUIEs, selection should be based on consideration of the frequencies of the embedded subtasks in actual operations, the criticality of successful completion of these subtasks, and any uncertainties regarding the most effective CHI designs and associated functionalities. A further consideration is to ensure coverage of not only nominal scenarios but also the inclusion of important off-nominal scenarios.

The elements of these scenarios can be guided by the results of the task analyses and associated Critical Task Analysis Reports (see p.B-7) provided in the transfer package as well as by the contents of the operator human engineering design approach documents (HEDADs) produced by the Vendor responsible for SI (described below).

It is important to note, however, that the CHI design and supporting software and hardware functionality (including automation) as developed by the SI may introduce new critical tasks not considered during the Research for Service Analysis and earlier AMS stages and can thus identify additional scenarios that require consideration. 

Selection of Operators for Testing. The selection of the operators to include in cognitive walkthroughs and EUIEs also is very important. The participants need to be selected to ensure adequate coverage in order to get feedback on design decisions from the full range of prospective users. Three points merit emphasis:

  • First, although an effort is typically made to include a range of different users to serve as Subject Matter Experts (SMEs) on the CHI team for a project, it is often the case that these SMEs are among the more experienced of the relevant user population. Such representation is important, as such experts are likely to have a greater breadth and depth of experience
  • Second, the CHI team of necessity has a limited number of user representatives, potentially making it impossible to fully represent the full range of different user groups
  • Third, since the CHI team supports development over an extended period of time, like the staff responsible for the design and implementation of the software, the members of the CHI team may become too familiar with the design and therefore may have difficulty seeing it with “fresh eyes”
  • The transfer package from the Research for Service Analysis should provide insights into the range of users that should be considered in testing

The conclusion regarding the selection of participants for testing is therefore twofold:

  • First, the members of the CHI team are an important subset of the users that need to be included in evaluations
  • It is important to use the results of the user analysis (drawing upon findings from the Research for Service Analysis and any additional data collected early in the SI process) to determine what additional participants in testing should be obtained from the user community in order to reflect the full set of differences across users and to guide decisions about participant selection. And tests with these additional participants should be separate from tests with the CHI team members in order to get independent assessments

In addition, it is important to recognize that it is not just the immediate user of a tool who needs to be considered. As an example, for certain tools for ATC, it is not sufficient to assume that systems for ATC can be designed without an assessment of impacts resulting from the performance of pilots. Thus, for some ATC tools, it is important to get input from the broader set of people who are indirectly affected by the introduction of a new capability, not just the immediate users.

Back to top