Checklist for Expert Evaluation of HMIs of Automated Vehicles—Discussions on Its Value and Adaptions of the Method within an Expert Workshop

Introduction

The safety and usability of Autonomous Driving Systems (ADS) rely heavily on effective Human-Machine Interfaces (HMIs). The background of this study is the Federal Automated Vehicles Policy provided by the U.S. National Highway Traffic Safety Administration (NHTSA), which outlines minimum requirements for AV HMIs. These requirements stipulate that the HMI must inform the user about the system functioning properly, being engaged in automated driving mode, being unavailable, experiencing a malfunction, and requesting a control transition from the ADS to the operator. However, NHTSA did not provide details on how entities should assess and validate conformity to these requirements. Prior to the publication of the underlying research, no standardized tools existed for the assessment of ADS HMI usability and safety.

The purpose of this paper is to summarize the results of an expert workshop held to discuss and further improve a heuristic evaluation methodology that uses a checklist to support the development of HMIs for AVs. This heuristic assessment method is a component of a larger test procedure developed to evaluate the conformity of SAE Level 3 (conditional automation) ADS HMIs with NHTSA policy requirements. The primary goal of the workshop was to bring together human factors experts to discuss the method, promote the development of HMI guidelines, and accelerate convergence toward robust guidelines for the industry.

Methodology

The study summarizes the review and adaptation of an existing heuristic evaluation methodology performed during a workshop.

Research Design and Participants: The workshop was held on 27 June 2019, in Santa Fe, New Mexico, and included 14 participants. Participants were selected based on their expertise as practitioners or academics in the automotive domain with previous experience in HMI design, development, or evaluation methods for driver assistance systems. The group consisted of representatives from the automotive industry, scientific institutes, and national agencies.

Data Collection Procedures: The methodology centered on hands-on application and discussion of the heuristic checklist.

1. System Experience: Participants were split into small groups of 3–4 people. They applied the method while riding as passengers in two vehicles equipped with Level 2 (L2) driving automation systems: a Tesla Model 3 with Autopilot (AP) and a Cadillac with GM Supercruise (SC). The use of L2 systems was justified as most of the heuristics refer to general design guidelines applicable to all automated systems, and L3-equipped vehicles may also operate in L2 modes.
2. Procedure: The participants experienced both systems during a 30-minute drive on the interstate. Due to safety and insurance reasons, the workshop organizers drove, meaning the evaluation did not strictly follow the proposed procedure where evaluators should drive. The drivers executed a fixed set of use cases (e.g., system activation, driving for a longer interval, experiencing the driver monitoring system, and planned system limits).
3. Documentation: After experiencing both vehicles, participants filled out the checklist based on the second system experienced.

Analytical Methods: The heuristic assessment method involves using a checklist comprised of 20 items summarizing key HMI design principles derived from existing norms, standards, and empirical research. These items cover visual, auditory, and vibrotactile HMIs. Each checklist item is rated using five categories: “major concerns,” “minor concerns,” “no concerns,” “measurement necessary”/”subject to verification,” or “not applicable”. Following individual assessment, the experts engaged in a joint discussion to reach a unanimous rating and discuss methodological improvements

Results

The study confirmed the value of the heuristic method while identifying key areas for adaptation and improvement based on expert consensus.

Value and Limitations of the Method: The workshop participants agreed that the developed method is a useful tool primarily for facilitating the assessment of system usability and ensuring compliance with the NHTSA minimum requirements. Compared to user studies, the heuristic evaluation provides a quicker and more efficient way to identify potential usability issues early in the product cycle, helping to reduce late-stage design changes. However, experts noted that for a comprehensive evaluation, the method must complement empirical approaches such as usability testing with real users to assess complex aspects like system logic, system trust, controllability, and user experience (UX).

Key Adaptions and Checklist Revisions: Based on the discussions, several adaptions were decided upon for future iterations of the checklist:

1. 1. Structure and Usability: The checklist structure will be revised to group items into more global categories and subcategories to improve expert usability
   2. Measurable Items: Items regarding objectively measurable aspects (e.g., color contrasts, line of sight, text sizes) will remain in the checklist, but they will be assessed subjectively by experts for confirmation, with absolute measurable numbers removed from the examples.
   3. New Complexity Category: A new, multi-dimensional category focused on perceived complexity will be included. This category will cover visual demands, cognitive demands (system logic complexity), motoric demands (operational device arrangement), and the ease of learning system interaction.
   4. New Specific Items: Two new items will be added: one concerning the appropriate design of other display elements (like steering wheel light bars) and another concerning the content of a warning/take-over request, particularly addressing the identification of the hazard, means to avoid it, and consequences of inaction.

   Test Procedure Confirmation: The proposed test procedure—using two experts to experience the system during defined use cases in real drives, documenting compliance independently, and then reaching a unanimous decision through joint discussion—was rated as a reasonable approach. It was suggested that experts should adopt the perspective of “naive users” (persons with no more prior knowledge than a later customer would have) to ensure the requirements are valid for the average population.

   References

   1. ANSI Z535.4-2011 (R2017). American National Standard for Product Safety Signs and Labels.
   2. Borojeni, S.S.; Chuang, L.; Heuten, W.; Boll, S. Assisting Drivers with Ambient Take-Over Requests in Highly Automated Driving. In Automotive’ UI: Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications; Association for Computing Machinery: New York, NY, USA, 2016; pp. 237–244.
   3. Hoff, K.A.; Bashir, M. Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust. Hum. Factors 2015, 57, 407–434.
   4. ISO 26262. Road Vehicles Functional Safety; International Organization for Standardization: Geneva, Switzerland, 2008.
   5. Linja-aho, M. Creating a framework for improving the learnability of a complex system. Hum. Technol. 2006, 2, 202–224.
   6. National Highway Traffic Safety Administration. Automated Driving Systems 2.0: A Vision for Safety; NHTSA: Washington, DC, USA, 2017.
   7. Naujoks, F.; Hergeth, S.; Keinath, A.; Wiedemann, K.; Schömig, N. Development and application of an expert based assessment for evaluating the usability of SAE Level 3 ADS HMIs. In Proceedings of the ESV Conference 2019, Eindhoven, The Netherlands, 10–13 June 2019.
   8. Naujoks, F.; Hergeth, S.; Wiedemann, K.; Schömig, N.; Forster, Y.; Keinath, A. Test procedure for evaluating the human–machine interface of vehicles with automated driving systems. Traffic Inj. Prev. 2019, 20, 146–151.
   9. Naujoks, F.; Hergeth, S.; Wiedemann, K.; Schömig, N.; Keinath, A. Use cases for assessing, testing, and validating the human machine interface of automated driving systems. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Philadelphia, PA, USA, 1–5 October 2018.
   10. Naujoks, F.; Wiedemann, K.; Schömig, N.; Hergeth, S.; Keinath, A. Towards guidelines and verification methods for automated vehicle HMIs. Transp. Res. Part F Traffic Psychol. Behav. 2019, 60, 121–136.
   11. Nielsen, J. Usability Engineering; Academic Press: Boston, MA, USA, 1993.
   12. RESPONSE Consortium. Code of Practice for the Design and Evaluation of ADAS; RESPONSE 3: A PReVENT Project; 2006.
   13. Schömig, N., Wiedemann, K., Hergeth, S., Forster, Y., Muttart, J., Eriksson, A., Mitropoulos-Rundus, D., Grove, K., Krems, J., Keinath, A., Neukum, A., & Naujoks, F. (2020). Checklist for Expert Evaluation of HMIs of Automated Vehicles—Discussions on Its Value and Adaptions of the Method within an Expert Workshop. Information, 11(4), 233.
   14. UN ECE R79: Uniform Provisions Concerning the Approval of Vehicles with Regard to Steering Equipment. 2017.
   15. UNECE. Uniform Provisions Concerning the Approval of Vehicles with Regard to Automated Lane Keeping System. Informal Document ACSF-25-03.
   16. Wickens, C.D. Processing resources in attention. In Varieties of Attention; Parasuraman, R., Davies, R., Eds.; Academic Press: New York, NY, USA, 1984; pp. 63–101.