Zhang Heng, Yuanbo Sun, Zhang Boyang, Yinan Che (2022). The Verification of Human-Machine Interaction Design of Intelligent Connected Vehicles Based on Augmented Reality. In: Human Factors in Virtual Environments and Game Design. AHFE Open Access, vol. 50, pp. 219–226. https://doi.org/10.54941/ahfe1002081
Firstly, compared with the traditional vehicle interactive verification methods, it is concluded that the real vehicle test scheme has the disadvantages of high cost, poor experience of pure physical bench scheme and lack of force feedback of pure virtual reality scheme. Augmented reality plus physical bench scheme has potential. Through the disassembly of automobile interactive verification process, the factors affecting the effect of automobile interactive verification are summarized. Secondly, conduct investigation and interview on the automobile active interactive users and the subjects of verification experiment, summarize the form and scope of automobile human-computer interaction design, as well as the design strategies and difficulties of interactive verification experiment, so as to determine the functional requirements of automobile interactive design verification bench. Then the feasibility of augmented reality technology applied to automotive human-computer interaction design verification is discussed and practiced.
Vehicle man-machine interface is a common configuration in the luxury car market and has gradually become a common configuration of medium and high-end cars. In many cases, users must use various functions of the onboard man-machine interface while driving. Therefore, different from other man- machine interfaces, the onboard man-machine interface must be designed as simple and easy to operate as possible, and try not to affect the driver’s driving behavior, so as to prevent driving safety problems. Testing and evaluation are essential steps in the research and development of vehicle man-machine interfaces. It is necessary to conduct user-centered Scientific and accurate usa- bility tests and evaluations on the onboard man-machine interface, so as to ensure the satisfaction of users in using the onboard man-machine interface and improve the competitiveness of the overall product. Driving simulators is a kind of system equipment that can simulate the running condition of the
car more realistically and get close to the driving feeling of the real car. It is an important tool for new automobile product development, traffic beh- avior characteristics, road optimization, driving characteristics research of different driver groups, and driver training. It is typical hardware in the loop simulation equipment. With the development of virtual reality, augmented reality, hybrid reality and other related technologies and testing technologies, the construction and application of immersive virtual evaluation systems for automotive human-computer interaction design becomes possible. However, the immersive virtual reality system displays the virtual scene for the user in a closed space through the helmet-mounted display and other devices, and clo- ses the user’s vision, hearing, and touch in the same space, which has a better sense of immersion than the traditional hardware verification bench. Howe- ver, this scheme completely relies on the virtual scene in the helmet-mounted display and a small number of hardware devices for interactive simulation and lacks tactile feedback for the actual interaction effect and grasp of the field spatial scale. Therefore, it is of great significance to study the vehicle interactive verification scheme based on augmented reality.
EXISTING VERIFICATION SCHEMES The basic idea of user-centered design and evaluation is to put users in the first place in all processes all the time. In the initial stage of the product life cycle, the product strategy should take meeting the needs of users as the basic motivation and ultimate goal. In the subsequent product design and development process, the research and understanding of users should be used as the basis for various decisions. At the same time, the evaluation information of the product at each stage should also come from the feedback of users. Therefore, the concept of the user is the core of the whole design and evaluation idea. The contradiction between the complexity of product functions and people’s demand for simplicity of use leads to the widespread usability problems of human-computer interface products. When designing a human-machine interface, designers need to consider three aspects: human, machine, and environment. User-centered interface design can only be ach- ieved if usability is met. At present, usability testing and evaluation have been widely used in the design process of various human-computer interfaces abroad. Sony, Nokia, and other interface product manufacturers have their own usability evaluation centers, which conduct professional usability tests and evaluations for their interface products. In addition, there are a certain number of professional usability organizations at home and abroad. It can be said that only the products evaluated by usability tests can be successfully used by people.
At present, three verification methods for automotive human-computer interaction design schemes have emerged in the industry. The first is the more traditional real vehicle testing, that is, design verification in a highly sophisti- cated physical vehicle. This verification method is a verification method after the manufacturing process. It cannot pre-test and evaluate interaction design scenarios prior to manufacture. The second way is to verify the interaction design by making a hardware prototype platform. This scheme simulates the
The Verification of Human-Machine Interaction Design of Intelligent Connected Vehicles 221
design part, simulates the interaction scene, and predicts the problems that may occur after the design scheme is implemented. The third solution is to use virtual reality technology to reconstruct the in-vehicle scene in the virtual world to simulate the interactive scene at a lower cost.
Prototype Bench With the development of automobile technology, users have high require- ments and expectations for a good automobile interactive experience, and the automobile human-computer interaction scheme is constantly updated. At the same time, designers also need to constantly verify and improve the auto- mobile human-computer interaction interface (HMI). When carrying out real vehicle tests and evaluations, the man-machine relationship between human and vehicle and between human and interface can be accurately conside- red, and the tested person will have the most realistic response. However, the real vehicle test cycle is long, the cost is high, and it consumes a lot of manpower, material resources, and other resources. Therefore, the traditio- nal evaluation method is mainly to make a hardware prototype bench and then conduct real-life user tests. With the development of simulation techno- logy, the pure hardware platform driving scheme cannot meet the verification requirements of the visual interaction-based automotive human-computer interaction design scheme. Gradually, driving simulators have been widely used for vehicle interaction verification.
The driving simulator is a kind of system equipment that can simulate the running condition of the car more realistically and get close to the driving feeling of the real car. It is an important tool for new automobile product development, traffic behavior characteristics, road optimization, driving cha- racteristics research of different driver groups, and driver training. It can simulate the external stimuli such as vision, sound, touch, and motion feeling felt by drivers under real working conditions, and create a realistic simu- lation environment. In the process of operating the simulator, the display screen, operating system, and computer-aided system provide the user’s sense of vision, hearing, and touch; The user completes the complete man-machine driving interaction process through corresponding driving operations. The development of simulator originated in the mid-20th century, which was ini- tially applied to simply imitate driving movements; With the comprehensive development of virtual reality technology, multimedia technology, multi- degree of freedom motion system and other technologies and integrated into the simulator, the simulator developed so far not only provides users with a highly immersive driving experience on the premise of safe driving environ- ment and energy-saving and pollution-free driving process, Its application scope has been extended to the scientific research practice of automobile engineering and live entertainment. Compared with other types of vehicle simulation environments, the driving simulator has many advantages. Com- pared with the real vehicle test, the driving simulator can simulate some extreme working conditions safely and at a low cost. Moreover, the driving simulator can accurately reproduce the experimental process, which is not available in real vehicle tests. Compared with hardware, driving simulation
Figure 1: Research driving simulators & Training driving simulators & Entertainment driving simulators.
not only introduces human factors, but also provides people with a near- real driving experience, which makes the simulation process closer to the real driving, and makes the simulation results more meaningful. In summary, the driving simulator can completely control the experimental environmental factors, efficiently collect experimental data, provide a safe and controllable experimental environment for the experiments of human extreme physiolo- gical response, vehicle slip, collision, and other dangerous situations, and the experimental cost is extremely economical.
According to different functions, driving simulators are roughly divi- ded into three types: training type, research type, and entertainment type (Figure 1). The focus of training simulators is to restore real driving scena- rios and provide users with driving training; research simulators are mostly used in the development and research of automotive engineering technology, as well as in the early stages of relatively close and complex vehicle develo- pment such as vehicle ergonomics analysis; entertainment driving simulations The purpose of the device is to provide users with an exciting and interesting driving experience. Due to the pursuit of driving pleasure, it does not require a strict level of simulation of real driving.
Virtual Reality Verification Scheme The experiments of Nobuyuki Uchida et al. have proved that augmented reality-based virtual reality technology has development potential in the research of vehicle ergonomics, but this system is limited by the research environment and its application field is relatively limited. In addition, the immersive virtual reality system displays the virtual scene for the user in a closed space through the helmet display and other equipment, and the user’s vision, hearing, and touch are enclosed in the same space, which has a better immersion than other virtual reality systems. It is not limited by the venue and can be used in the construction of virtual interactive platforms. Regarding the human-computer interaction equipment of this system, Daniele Spor- tillo et al. analyzed the interaction differences between the subjects using the real steering wheel and the 6-DOF handle and concluded that the driving simulator hardware is better than the driving simulator hardware in satisf- ying the user’s real sense of interaction. Conclusion for the VR controller. Garrett Weinberg and Bret Harsham used a steering wheel, desktop virtual reality system, and eye-tracking device to build a low-cost human-computer interaction evaluation platform.
The Verification of Human-Machine Interaction Design of Intelligent Connected Vehicles 223
Verification Schemes Summary Research on vehicle interaction verification mainly focuses on real vehi- cle testing, adjustable man-machine verification benches, and virtual reality simulation systems. Some automobile companies have researched and deve- loped the adjustable and reusable product physical model, applied it to the product development practice, and achieved good results. The more typi- cal one is Ford’s integrated layout design and evaluation system. As far as the adjustable man-machine verification bench is concerned, some research institutions have conducted similar research and development. For example, some research institutions in China have developed test benches for simula- ting car or aircraft cockpits and analyzing and verifying driver ergonomics. The current adjustable ergonomic test benches still have the following pro- blems: x Usually, they are mainly used for some kind of ergonomic test research, and are rarely used for the presentation of actual product design schemes; y A few test benches are mainly used for product verification. Veri- fication for a specific vehicle model, the same bench cannot be verified for different types of vehicles. z Generally, it does not have the function of verif- ying the interior design. Even if it has the function of verifying the interior, it is usually only suitable for installing the interior of a certain model, but not suitable for installing the interior of different passenger models, so the function of interior verification is limited.
VEHICLE HUMAN-COMPUTER INTERACTION USABILITY TESTING PARTICULARITY Usability (refers to the effectiveness, ease, and efficiency of using a product or service. With the progress and development of science and technology, the functions of the product have shown the characteristics of diversifica- tion and complexity, which also affects the human-machine interface of the product. More serious requirements are put forward. Products with more complex functions, higher difficulty in getting started, and higher professi- onal level require an interface that can reduce the user’s psychological and physiological load and improve the user’s work efficiency; while popular pro- ducts require man-machine The interface is easy to be seen by users, easy to understand, easy to operate, etc. Interfaces in different fields have diffe- rent requirements for designers, which makes the usability of human-machine interfaces a popular and focused research direction.
The more comprehensive and precise definition of usability by the famous usability guru Jacob Nielson (1993) complements the missing part of Har- tons’ definition. His famous definition of usability includes the following attributes: Learnability: Is the product easy to use? Can you quickly start operating and completing tasks even without training? Efficiency, can users complete tasks and achieve goals with high efficiency? Memorability, after users, stop using the product for a period of time, can they still remember how to operate? Error rate, is it prone to errors in the operation? Satisfaction, are users satisfied with the product? How satisfied are you?
The International Organization for Standardization (ISO) has made a spe- cific definition of usability in the standard ISO 9241-11: Usability refers to
the effectiveness, efficiency, and satisfaction of the interaction process when users use a product to complete a specific task in a specified environment. According to the definition of usability, there are three main attributes in the concept of usability: effectiveness, efficiency, and satisfaction. Effectiveness generally refers to whether the product can achieve specific functions and the effectiveness of interface information transmission; efficiency refers to the performance level of completing operational tasks through the product, including fault tolerance, ease of learning, and easy to remember; finally, satisfaction refers to the user Subjective satisfaction with the product. The definition of usability promotes user-centered design thinking to a certain extent and makes people more and more aware of the importance of user experience.
Usability testing is a method that testers invite users to use design prototy- pes or products to complete operation tasks, and evaluate the usability of the interface by observing, recording, and analyzing user behavior and relevant data. Usability testing can comprehensively evaluate the usability of the inter- face, which is one of the most commonly used methods. It is applicable to the evaluation of product interface and interface prototype in the middle and later stages of interface design. Utility refers to whether the interactive inter- face can effectively realize the function of the product, that is, the accuracy and integrity of users to complete specific tasks. Efficiency means that users can use the least resources to achieve specific tasks, including less time and less energy. Satisfaction refers to the subjective satisfaction and acceptance of the product, including comfort and willingness to use the product.
Due to the particularity of driving scenarios, car human-computer inte- raction design is also different from general human-computer interaction design. When the driver is driving, the overly complex interaction method will divert the driver’s attention from driving, and in the process of driving, distraction is an extremely dangerous thing, which greatly affects the driving experience. safety. Second, the driver can only interact in certain gestures while driving the car. For example, when it comes to hand interaction, the driver can often only use a relatively free hand for touch or gesture intera- ction, which also limits the range of motion for human-machine interaction in cars. Finally, due to the complexity of driving scenarios, the functions covered by car human-computer interaction are relatively more and more complex. Devices with more complex interaction methods in life, such as mobile pho- nes, can already meet more functional requirements. However, most of the functions of mobile phones in driving scenarios must be replaced by cars. What’s more, in the car, the main task of the driver is to ensure the safety of driving, so it is conceivable that the functions of the car’s human-computer interaction cover a wide range of complexities.
Therefore, the usability evaluation indicators suitable for automotive HMI (Table 1) mainly include practicality, efficiency, satisfaction, and driving com- patibility. Among them, driving compatibility includes the operation logic of the car HMI, which determines whether the interaction process will con- fuse users and affect driving safety. Driving compatibility also includes the rationality of ergonomics, which determines whether the user’s actions will be restricted during the interaction process in the car; in addition, legibility
The Verification of Human-Machine Interaction Design of Intelligent Connected Vehicles 225
Table 1. Usability evaluation indicators suitable for automotive HMI.
Table available in the original PDF paper.
Main Indicators Indicator Description Utility Visibility, Discrimination, Discrimination, Accessibility Efficiency Comprehension, Consistency, Intensity, Information Conciseness, Error Tolerance, Sensitivity, Fastness Satisfaction Comfort, quality of expression, degree of customization, aesthetics Driving Compatibility Operational logic, ergonomic rationality, easy identification, focus
determines whether the user is easy to understand and respond when reco- gnizing interactive instructions; the last point is attention, which determines whether interactions within the vehicle interfere with driving behavior.
USER NEEDS AND FUNCTIONAL DESIGN The relevant users involved in the verification system are mainly divided into master guides, observers, and evaluators. The task of the lead facilitator is to guide the evaluator through the assigned tasks and answer any questions posed by the evaluator. The task of the observer is to record the assessor’s assessment results, problems found, and assessments. Evaluators can be divi- ded into two categories, the first category is ordinary consumers, that is, the target group of human-machine interface products, and the second category is expert consumers with certain professional knowledge. Ordinary consu- mers need to give subjective feelings and evaluations on behalf of the target group, but they may not be able to give credible reasons for evaluation when the evaluation standards are measured. The difference is that expert consu- mers can give more accurate results and reasons for the evaluation criteria and can find some usability problems that ordinary consumers cannot find and give solutions to.
The verification system needs to meet the following functions: simulating car functions, simulating driving scene simulation, network connection in the local area network, and collecting user data. The system needs to be able to basically simulate the functions of a typical car, and combine the initial state of the car as well as the location and number of intelligent vehicles to form driving scenarios in different traffic environments. After the network conne- ction is realized, the verification system can have greater flexibility. As long as it is under the same local area network, it can access the driving vehicle arbitrarily, realize multi-vehicle testing, and simulate the real traffic environ- ment more realistically. In addition, the system also needs video observation, and the driver or other personnel can see the real-time video of the driver’s manipulation of the hardware device, the global map of the scene, and the local map of the current driving vehicle on the observation screen. It also has other equipment such as EEG, skin electricity, and eye tracker to collect user behavior and responses.
In order to meet the above system functional requirements, an adju- stable and flexible research driving simulator combined with augmented
reality technology is a feasible solution. Augmented reality technology can attach more flexible interactive components to hardware entities. For exam- ple, when exploring new steering wheel interactions in the future, a simple prototype can be 3D printed to test indicators related to human-machine dimensions. Also, attach the steering wheel texture and screen to the har- dware mockup in a virtual reality head-mounted display with a camera (such as the HTC Vive Pro) , which allows evaluators to both have physi- cal hardware to touch or click in reality and see a screen-based interface in a helmet-mounted display.
CONCLUSION
With the application and development of advanced technologies such as the Internet, big data, and artificial intelligence in the field of transportation, from the perspective of design, the car has developed from a vehicle to a design object including personal space, public space, and social space. In this context, automotive HMI designs are becoming more and more complex and mature, and corresponding design verification methods also need to meet new requirements. A research driving simulator combined with augmented reality could be a viable option. It can test the utility, efficiency, satisfaction, and driving compatibility of automotive HMI, and can test as many interaction models as possible.
REFERENCES
Booher, Harold, ed. (2003). Handbook of human systems integration. New Jersey: Wiley. Booher, H.R., Minninger, J. (2003) “Human systems integration in army systems
acquisition”, in: Handbook of human systems integration, Booher, Harold (Ed.). pp. 663–698. M. R. Zofka et al. (2018) “Traffic Participants in the Loop: A Mixed Reality-Based Interaction Testbed for the Verification and Validation of Autonomous Vehicles,” 2018 21st International Conference on Intelligent Transportation Systems (ITSC). pp. 3583–3590. N. Uchida, T. Tagawa and K. Sato. (2017). “Development of an Augmented Reality
Vehicle for Driver Performance Evaluation”. in IEEE Intelligent Transportation Systems Magazine, vol. 9, no. 1, pp. 35–41. X. Wang, H. Peng, S. Zhang and K. -H. Lee, (2021) “An Interaction-aware Evaluation Method for Highly Automated Vehicles”, 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). pp. 394–401.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Virtual Environments and Game Design
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-26-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional