Nicolas Daniel Herzberger, Marcel Usai, Michael Preutenborbeck, Ronald Meyer, Gina Wessel, Frank Flemisch (2022). Biologically Inspired Automotive User Interfaces for Partially and Highly Automated Maneuver Gestures: Final Results and Outlook. In: Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems. AHFE Open Access, vol. 22. https://doi.org/10.54941/ahfe100975
Automated driving puts severe challenges on the design and testing of automotive user interfaces. In partially automated driving, the driver is still responsible for the vehicle control, but is strongly supported by technology. In highly automated driving, the driver can give control to the automation for a certain time and can get control back e.g., when the automation encounters limits. Despite great technological progress, a truly intuitive way to interact with these automated driving modes is still under research. Project Vorreiter is addressing this by using the inspiration of a rider and a horse to provide intuitive steering gestures on the steering wheel or an alternative device, which initiate maneuvers executed by the automation. These can be supervised, influenced or interrupted by the driver. The gestures are built up in a universal design approach, which helps all drivers, including beginners and drivers with disabilities. After an introduction into the overall philosophy and concept, the contribution focuses on a final step in the project, an overall evaluation of the concept in a driving simulator and presents new data especially on the comparison of swiping gestures and pushing gestures regarding false or true positive or negative detected gestures. Finally, a brief outlook sketches next steps with a new Wizard-of-Oz / theater vehicle.
Based on the three levels of a driving task (stabilization/control, guidance and navigation (Donges, 1995), Flemisch et al. (2006) described how the control of a vehicle can be shared between a driver and a cooperative automation. The authors pointed out, how the transition between two states can be implemented, where in one state the driver controls the lower stabilization level and in another state, only maneuvers are selected. Winner et al. (2006) described the idea of maneuver-based driving in order to shift the execution of vehicle stabilization from the driver to an automation system while granting the driver access to maneuver control of that system. These concepts of the different layers of control were later extended by the cooperational layer (Flemisch et al. 2019a).
With emerging cooperation between driver and automation system, the need for alternative controls is growing. Early works suggest the use of graphical user interfaces for maneuver selections (e.g. Kauer et al. 2010) and later comparing the applications of direct touch to voice and hand gesture interfaces, while notably trying to keep the touch interface close to the steering wheel (Detjen et al. 2020). Different alternative control devices such as sidesticks (e.g. Saupp et al. 2019) have been tested, as have alterations of the steering wheel (e.g. Mok et al. 2017) to better adapt it to its potential change of purpose in partially and highly automated driving. The trend is towards intuitive and non-distracting interfaces, e.g. the combination of head-up displays (HUD) with a hands-free interface for non-driving related tasks (NDRT) to support lower levels of automation (Tippey et al. 2017).
Some of the state of the art described above, were influenced by a biologically
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 inspired design metaphor, which compares automated driving to the relationship of a rider and a horse. This H(orse)-Metaphor was initially described by Flemisch et al. (2003), instantiated into a multimodal way of driving (e.g. Altendorf et al. 2015), in an interaction scheme of cooperative automation and shared & cooperative guidance and control (e.g. Flemisch et al. 2016 & Flemisch et al. 2019a).
GENERAL IDEA: FROM H-METAPHOR TO STEERING GESTURES One essential aspect of the H-Metaphor is to look at assistance and automation not as black and white, i.e. manual or autonomous, but as a spectrum of control distribution between humans and machines, which can be continuous, but can also be grouped in modes. These modes were initially formulated as tight rein, loose rein and secured rein, translated into technical terms as assisted, partially and highly automated driving described by a group led by the BASt (Gasser et al. 2012), and later extended by the SAE into the internationally known SAE levels of driving automation (SAE, 2018). What the levels of automation do not describe, however, is how human and automation interact in higher levels of automation - especially how longer sequences of a driving behavior can be initiated and still be controlled in an intuitive way by the driver.
Figure 1. Shared and cooperative control in a rider-horse system, and in a human-automation system (extended from Donges (1995), see also Schartmüller et al. (2019)), and in the Vorreiter project focused on maneuver gestures.
One of the keys to control horses is that continuous interaction, e.g., by holding a rein, is complemented with discrete interaction e.g. on the stirrups to start or stop complex sequences of behavior (Fig. 1). In vehicles, these sequences are called maneuvers, and can be initiated in a similar, discrete interaction by hand or foot, combined with other visual or acoustic interactions: A combination which we call maneuver gestures. It became clear that the interaction is more likely to be successful, if (a) the co- system/automation offers possible maneuvers e.g., by displaying these on a head down or head up display, (b) the driver initiates a maneuver with a steering gesture, and then (c) the automation executes the maneuver, while the maneuver can be felt,
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 influenced or interrupted by the driver through haptic interaction.
OVERVIEW OF THE PROJECT VORREITER Encouraged by these early simple versions of maneuver based control, in 2016 a consortium of IAW – RWTH Aachen University, Fraunhofer IAO, IAT - University of Stuttgart, HWR Berlin and Valeo GmbH set out to explore the concept of maneuver gestures. A detailed overview of the project can be found in a book chapter (Flemisch et al. 2020). The final evaluation as well as the results are published for the first time below. In the following, the concept and the development process will be described:
In order to find a maneuver gesture design that satisfies users as well as technical and legal requirements, the consortium used a structured iterative exploration process (Flemisch et al 2019b). The use- and design-space of maneuver gestures were explored systematically in several workshops (Wessel, 2017), covering user groups including novice, frequent, elderly and disabled drivers. Based on the exploration and first experimental results, the use-space was structured into a catalogue of eleven use- cases (Flemisch et al. 2020). Based on this use-case-catalogue, a set of abstract gestures was designed that intuitively adapts to all use-cases. These gestures are defined as a ‘pars pro toto’-movement in at least one of the general directions of movement of the vehicle, i.e., ‘to front’, ‘to back’, ‘to left’ or ‘to right’.
Based on the abstract gesture set and inspired by the H-Metaphor, two sets of steering wheel gestures were developed: Push/ Twist gestures, where the hand and the inceptor interact by forces and torques, and Swipe Gestures, where the interaction is just by gentle touch or stroking.
Both gesture sets are designed for the use in partially and highly automated vehicles. For the use of steering gestures in automated vehicles, an interaction pattern was designed (see Flemisch et al. (2020), figure 8.2): While driving in partially or highly automated driving mode, the vehicle constantly checks available maneuvers, the vehicle keeps lane while exploring that changing to left or right lane is possible. All options are presented to the driver, e.g., using a trajectory display on a contact analogue head up display. At any given time, the driver may input a steering gesture to signal his or her will to execute a maneuver in given direction. While the driver attempts and continues the gesture input, the machine evaluates the driver’s input and maneuvers connected to it. Based on the driving situation and driver’s input, the correct maneuver is chosen. Based on the current vehicle automation level and its safety and legal evaluation results, the maneuver requested by the driver is then executed or not executed. The driver may abort the gesture input at any time by stopping gesture input or even the maneuver execution by applying a gesture in the opposite direction of that of the current maneuver.
The application of this pattern requires a cooperational automation system which is based on maneuvers and trajectories. For the technical application of the developed
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 design pattern for maneuver-based driving with both developed gesture sets, Valeo developed a steering wheel with integrated capacitive touch detection (Flemisch et al. 2020). The steering wheel is divided into multiple zones required to detect swipe gestures. Combined with LEDs along the steering wheel, visual feedback to the driver is provided. The visual feedback includes feedback on the current situation, gesture inputs as well as mode awareness.
For the application of swipe gestures to the Vorreiter steering wheel and possibly other haptic input devices, the IAW developed a pattern detection algorithm. The algorithm is described in more depth in [20].
In the development and implementation of steering gestures, special emphasis was also placed on legal feasibility. In this context, particular attention was paid to the legal admissibility and whether such systems could possibly help people with disabilities to obtain a driving license more easily. A general overview of the legal aspects and its impact on the steering gestures can be found in Flemisch et al. (2020). The developed steering gestures (Meyer et al. 2020) were examined in several studies: In two VR studies at IAO with a holographic steering wheel (Holowheel), swipe gestures on a capacitive steering wheel were tested against push gestures on a sidestick (Flemisch et al. 2020).
FINAL EVALUATION IN A DRIVING SIMULATOR
Method
The study was conducted in the driving simulator in the Exploroscope (Flemisch et al. 2013) of IAW. SILAB 6.0 was used as the driving simulator software. For vehicle controls, the capacitive steering wheel from Valeo and the IAW gesture recognition software developed by IAW were used. Eight driving scenarios were implemented that enabled the selected gestures: Rural road with T-intersection, highway entrance, highway, construction site, highway exit, again rural road with T-intersection and finally a parking lot. Five conditions were compared in a 2x2+1 Design: Manual control (baseline) versus twist/push versus swiping gesture both in SAE 2 and SAE 3/4.
N = 26 subjects participated in the study (26.9% female, 73.1% male). A valid driver's license and no uncorrected visual impairment were required for participation. The age of the subjects ranged from 19 to 64 years (M = 28.96 years, SD = 13.25 years).
Each experimental run began with an introductory drive (SAE Level 0) so that participants could familiarize themselves with the simulator. In addition, subjects were told that they would go through five test blocks, always starting with a Naive Run (without prior explanation), followed by a short questionnaire, an explanation of the current system, a Trained Run, and an associated questionnaire. The Naive Runs
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 were implemented in order to assess intuitive comprehensibility as well as learnability. After all test blocks were completed, a final evaluation followed. Each trial lasted approximately four hours per participant.
Results
OBJECTIVE EVALUATION To get an overview of the performance of the implemented gestures, the recognition rate of the individual gesture activations was evaluated (see Error! Reference source not found.).
Figure 2. Distribution of gesture activations and situation solutions based on a modified Signal Detection Theory (Macmillan, 2002). Recorded was the co-automation’s reaction to human input and its observed classification (Based on Usai et al. 2021).
The left side of Fig. 2 depicts the distribution of gesture activations. All attempts of participants to activate a gesture were counted and it was evaluated whether they were successfully recognized and also activated (true positive), recognized but rightfully not implemented because they were considered unsafe by the automation (true negative), recognized but obviously unintended by the driver, or different maneuvers intended by the driver (false positive) or not recognized or refused even if the maneuver would have been possible (false negative).
Data shows that the majority of responses to gesture inputs were true positive or true negative. The false negatives were, in this fragile risk balance between false positive and false negative, on the safer side. An exception was the longitudinal swipe gestures in Level 2, with had a relatively high rate of false positives. In this case, many participants struggled with the double allocation of the steering wheel, which serves both as an input device for vehicle stabilization and as a gesture input device.
Fig. 2, right depicts the evaluation whether the driving situation were solved overall, regardless how many attempts of individual steering gestures. A situation ended whenever a given opportunity, e.g. to take a turn, has passed or the desired maneuver
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 has been executed or the driver lost interest in initiating a maneuver. Data shows that more than 75 % of all situations in any case were solved as they should have been.
SUBJECTIVE PREFERENCES AND LEGAL SITUATION Fig. 3 depicts the subjective preferences and legal admissibility of the steering concepts. As part of the final overall evaluation, each participant could indicate which of the systems was preferred. Multiple responses were possible. Regardless of the level of automation, twist/push gestures were preferred over swipe gestures. The twist/push gestures in SAE level 3/4 were preferred over the baseline.
Figure 3. Overall subjective preferences and legal admissibility of the control gestures.
Results and Discussion
The results indicate that gesture control concepts and maneuver-based systems may be a real alternative in the future. Nevertheless, the SAE level 2 systems show clear difficulties: One challenge was the fact that at least one hand was required on the steering wheel. This sometimes led to a mixing of the maneuver gesture and the stabilization task. For swipe gestures, this could lead to an error in parallel gesture input and ‘hands-on’. In this case, a two-handed gesture input was incorrectly detected. However, twist/push gestures also struggled with some difficulties in level 2: For example, it could happen that the execution of a gesture was too gentle, so that one remained below the threshold and thus slowly changed lanes manually without actually triggering the corresponding gesture. This ‘oversteer’ was possible due to the coupling of human and co-system via the steering wheel, as the driver steering intervention directly affected the vehicle, in contrast to level 3/4. The subjective preferences also show that drivers preferred either manual or highly automated driving. The twist/push gestures combined with level 3/4, were even preferred over to the baseline.
These results are already quite promising, but remaining usability issues became apparent. To mitigate misunderstandings between the driver and the co-automation,
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 some refinements in the design will be necessary. In addition, it should be mentioned that the present study was a test in a static simulator. Therefore, several driving parameters could not be simulated and there was, for example, no feedback regarding the road condition via the steering. Here, a study in real traffic could provide important additional results.
OUTLOOK: TOWARDS A STANDARDIZED CATALOGUE OF MANEUVER GESTURES AND TESTING IN REAL TRAFFIC As SAE level 2 automation is already on the road, and SAE Level 3 and 4 systems are ready to be rolled out - it might look that everything is solved and the R&D community can proceed to other challenges. Nothing is further from the truth: With increasing numbers of automated cars on the road, the variability of solutions for automation behavior, and for human machine interfacing will increase, and without proper standardization, will lead to a high complexity for users, manufacturers and society.
Steering gestures and other concepts can be crucial pieces of this complex automation puzzle. The research described here is a first step on the way towards a real product, with first implementations and other payers started research on this as well, e.g. Honda showed a concept at CES with a similar concept (Motor1, 2021).
The development and testing of steering concepts for automated driving also brings new challenges for the testing of such systems. To test these concepts not only in simulators but also in real traffic, one challenge is to test these without having to wait for the proper implementation of the automation. For this purpose, a test vehicle was designed and built up within Vorreiter. This vehicle can be used as a Wizard-of-Oz (Schieben et al. 2009) system (Flemisch et al. 2020), where the automation is emulated by a human driver hidden behind a half-mirror, or as a theater system, where the glass-‘curtain’ is open so that a test driver can freely discuss design options with the ’automation’.
REFERENCES
Altendorf, E.; Baltzer, M.; Heesen, M.; Kienle, M.; Weißgerber, T.; Flemisch, F.
(2015): H-Mode, a Haptic-Multimodal Interaction Concept for Cooperative Guidance and Control of Partially and Highly Automated Vehicles; Winner et. al. (Eds.): Handbook of Driver Assistance Systems. Springer. Borojeni, S. S., Wallbaum, T., Heuten, W., & Boll, S. (2017). Comparing shapechanging and vibrotactile steering wheels for take-over requests in highly automated driving. In Proceedings of the 9th international conference on
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 automotive user interfaces and interactive vehicular applications (pp. 221- 225). Detjen, H., Geisler, S., & Schneegass, S. (2020). Maneuver-based driving for intervention in autonomous cars. arXiv preprint arXiv:2003.12496. Donges, E. (1995): Supporting Drivers by Chassis Control Systems. Seminar Smart
Vehicles, TNO Delft NL, 13 - 16. In: Pauwelussen, J. P., and Pacejka, H. B. (Ed): Smart Vehicles. Swets & Zeitlinger B. V., Lisse, Netherlands. Flemisch, F. O., Adams, C. A., Conway, S. R., Goodrich, K. H., Palmer, M. T., & Schutte, P. C. (2003). The H-Metaphor as a guideline for vehicle automation and interaction. Flemisch, F. O., Kelsch, J., Schieben, A., & Schindler, J. (2006). Stücke des Puzzles
hoch-automatisiertes Fahren: H-Metapher und H-Mode, Zwischenbericht. Flemisch, F., Semling, C., Heesen, M., Meier, S., Baltzer, M., Krasni, A., & Schieben, A. (2013). Towards a balanced human systems integration beyond time and space: exploroscopes for a structured exploration of human–machine design spaces. In HFM-231 SYMPOSIUM on Beyond Time and Space, Orlando, Florida, USA. Flemisch, F.; Winner, H.; Bruder, R. & Bengler, K. (2016). Cooperative Vehicle
Guidance & Control. In: Winner, H.; Hakuli, S.; Lotz, F. & Singer, C. (Eds.): Handbook of Driver Assistance Systems, Wiesbaden: Springer Nature. Flemisch, F., Abbink, D. A., Itoh, M., Pacaux-Lemoine, M. P., & Weßel, G.
(2019a). Joining the blunt and the pointy end of the spear: towards a common framework of joint action, human–machine cooperation, cooperative guidance and control, shared, traded and supervisory control. Cognition, Technology & Work, 21(4), 555-568. Flemisch, F.; Baltzer, M.; Sadeghian, S.; Meyer, R.; Lopez Hernandez, D.; Baier, R.
(2019b). Making HSI more intelligent: Human Systems Exploration versus Experiment for the Integration of Humans and Artificial Cognitive Systems; 2nd conference on Intelligent Human Systems Integration IHSI San Diego. Flemisch, F., Diederichs, F., Meyer, R., Herzberger, N., Baier, R., Altendorf, E., Spies, J., Usai, M., Kaim, V., Doentgen, B., Bopp-Bertenbreiter, A., Widlroither, H., Ruth-Schumacher, S., Arzt, C., Bozbayir, E., Bischoff, S., Diers, D., Wechner, R., Sommer, A., Aydin, E., Kaschub, V., Kiefer, T., Hottelart, K., Reilhac, P., Wessel, G. & Kaiser, F. (2020): Vorreiter, Manoeuvre-Based Steering Gestures for Partially and Highly Automated Driving; In Meixner, G.: Smart Automotive Mobility; Springer. Gasser, T. M.; Arzt, C.; Ayoubi, M.; Bartels, A.; Bürkle, L.; Eier, J.; Flemisch, F.;
Häcker, D.; Hesse, T.; Huber, W.; Lotz, C.; Maurer, M.; Ruth-Schumacher, S.; Schwarz, J. & Vogt, W. (2012): Rechtsfolgen zunehmender Fahrzeugautomatisierung - Gemeinsamer Schlussbericht der Projektgruppe Bundesanstalt für Straßenwesen (bast), (F 83). Kauer, M., Schreiber, M., & Bruder, R. (2010, June). How to conduct a car? A design example for maneuver based driver-vehicle interaction. In 2010 IEEE Intelligent Vehicles Symposium (pp. 1214-1221). IEEE. Macmillan, N. A. (2002). Signal detection theory. Stevens' handbook of
experimental psychology. Meyer, R., Herzberger, N. & Flemisch, F. (2020). Manövergesten für teil- und hochautomatisiertes Fahren. ATZ extra - Auto(matisierung) 25(4), 10-12.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Mok, B., Johns, M., Yang, S., & Ju, W. (2017). Reinventing the wheel: transforming steering wheel systems for autonomous vehicles. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (pp. 229-241). Motor1 (2021). https://www.motor1.com/news/388962/honda-concept-car-ces/,
retrieved 12.5.2021. SAE (2018). Levels of Driving Automation; SAE International Standard J3016. Saupp, L., Schwarz, B., Schwalm, M., & Eckstein, L. (2019). Evaluation of active drivesticks as alternative controls in a high fidelity driving simulator. Automotive and Engine Technology, 4(1), 37-44. Schartmüller, C., Wintersberger, P., Frison, A. K., & Riener, A. (2019). Type-o-
Steer: Reimagining the Steering Wheel for Productive Non-Driving Related Tasks in Conditionally Automated Vehicles. In 2019 IEEE Intelligent Vehicles Symposium (IV) (pp. 1699-1706). IEEE. Schieben, A.; Heesen, M.; Schindler, J.; Kelsch, J; Flemisch, F. (2009): The theatersystem technique: Agile designing and testing of system behavior and interaction, applied to highly automated vehicles; Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI); Essen. Tippey, K. G., Sivaraj, E., & Ferris, T. K. (2017). Driving while interacting with
Google Glass: Investigating the combined effect of head-up display and hands- free input on driving safety and multitask performance. Human factors, 59(4), 671-688. Usai, M., Meyer, R., Baier, R., Herzberger, N., Lebold, K., & Flemisch, F. (2021).
System Architecture for Gesture Control of Maneuvers in Automated Driving. In Inter-national Conference on Intelligent Human Systems Integration (pp. 65-71). Springer, Cham. Wessel, G. (2017). Exploration of gesture based interaction, IAW-Paravan POC / Exploration Log, 2017. Winner, H., & Hakuli, S. (2006). Conduct-by-wire–following a new paradigm for
driving into the future. In Proceedings of FISITA world automotive congress Vol. 22, p. 27.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-7923-8988-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional