Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Machine Learning Agent to Recommend the Best Modality for Takeover during Conditionally Automated Driving

1 Haute Ecole Arc Ingénierie, HES-SO, 2610 Saint-Imier, Switzerland ↗
2 HumanTech Institute, HES-SO//University of Applied Sciences and Arts, Western Switzerland, 1705 Fribourg, Switzerland ↗
3 Institute for Advanced Systems Engineering, 1700 Fribourg, Switzerland ↗
Publication year
Digital Object Identifier10.54941/ahfe100845

Abstract

>ContextConditionally autonomous vehicles are getting more and more visibility worldwide, but they bring many challenges. In particular, the transition of responsibility for the driving task from the car to the driver can be a dangerous situation, especially if the driver is absorbed in a non-driving-related task. In these cases, the car issues a takeover request (TOR), using different available modalities (such as auditory or visual). Research showed that the choice of modalities affects the takeover quality and should be adapted depending on the situation. >ContributionIn this study, we propose a smart agent recommending to the vehicle the best choice of modality for a TOR. This agent considers the driver physiological state based on the last 90 seconds prior a TOR, the external environment and three possible multimodal TOR. >Methodology>>ExperimentA 50 minutes driving session took place on a fixed-base driving simulator, where 15 drivers’ physiological signals (EDA, ECG and respiration) were recorded. There were 9 takeover situations during the driving session, caused by a fixed obstacle with a time-to-collision of around 7 seconds. The drivers had to perform three different non-driving-related tasks under two different weather conditions.The possible TOR modalities were combinations of haptic (vibrating seat), auditory (short beep sound) and visual (logo on the dashboard). Combinations tested were haptic-visual, auditory-visual, and haptic-auditory-visual. The takeover quality metrics were the reaction time between the TOR and the takeover, and the maximum steering wheel angle attained during the takeover process. >>Machine LearningTakeover quality metrics were aggregated to create a unique label to predict. Features were processed from raw physiological signals, the external environment and feature-selection techniques were applied to keep only the most relevant ones. After outlier suppression and data processing, 80 TOR were kept for the Machine Learning models training. Data Augmentation methods, such as SMOGN and Random Noise were implemented and tested to boost the training dataset.KNeighbors Regressor, Support Vector Regressor, Random Forest Regressor and Neural Networks were trained using a grid search approach and cross validation. >>AgentThe agent predicts the takeover quality using the ML model, with every possible set of modalities. It then chooses the modalities providing the best theoretical takeover quality. The agent was evaluated to avoid the following trivial situations: 1.The agent always chooses the same modalities, meaning that the training was biased toward one set of modalities.2.The agent chooses a random modality because the choice has no impact on predicted performance, meaning that the agent was not able to capture the modalities impact on takeover quality.A novel metric representing modality impact on takeover quality was defined and used to optimize agent behavior. >ResultsThe agent was able to predict takeover quality using a Support Vector Regressor with a MSE of 0.0685, beating the baseline (mean) by 39.65%. It was also able to suggest the 3 different set of modalities in the final phase, with a distribution of (0.625, 0.25 and 0.125). Modality impact on takeover quality had a mean score of 4.95% (std: 2.7%).

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Regression, Smart HMI, Driver State, Physiological Signals

INTRODUCTION

Conditionally autonomous vehicles are getting more and more visibility worldwide, but they bring many challenges. From interacting with the outside world as an actor on the road, to interacting with the driver which shares the control of the vehicle, those challenges must be studied and responded accordingly to reduce the risk of accidents. In particular, the transition of responsibility for the driving task from the vehicle to the driver can be a dangerous situation, especially if the driver is absorbed in a non-driving-related task. In these cases, the vehicle issues a takeover request (TOR). Upon acknowledging the TOR, the driver must take the control back as quickly as possible, but not without understanding correctly the situation and acting accordingly. For example, the correct behavior could be braking, avoiding an obstacle or maintaining the correct trajectory on the road. This whole process is called a takeover, and could lead to accidents if the driver is not able to execute it properly. Moreover, in level 3 automated vehicles, according to the SAE level of automation (SAE, 2021), drivers are not required to monitor the road while they are not driving, meaning they can be out of the driving loop and unaware of their surroundings. Different systems can be used to keep the driver situation awareness as high as possible (Yang, 2018), but ultimately the TOR is the last interaction before a takeover, and as such should be adapted to suit the situation to help improve the chance for a flawless takeover. Study of the literature reveals insights on how TOR can be used to influence takeover quality, as explained in the “Related Work” chapter.

OBJECTIVE In this study, we propose a smart agent recommending to the vehicle the best choice of modality for a TOR, using Machine Learning algorithms. This agent considers the driver physiological state based on the last 90 seconds prior a TOR, the external environment and three possible multimodal TOR.

RELATED WORK

Study of the literature shows that the driver psychophysiological state, the external environment and the TOR modalities have an impact on takeover quality.

Choice of modalities affects the takeover quality and should be adapted depending on the situation: multimodal TOR are considered to convey a stronger sense of urgency and lead to shorter reaction time than unimodal TOR (Zhang, 2019). Regarding the modality, the most commonly used are visual, haptic and auditory. Visual modality includes peripheral lights (Shah, 2020), icon on the vehicle dashboard or a handheld

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 device (Capallera, December 2019) and so on. Haptic modality refers to shape- changing or vibrating material, such as haptic seat (Grah, 2015) or steering wheel (Borojeni, 2017). The auditory modality consists mainly of different audio chimes (Ko, 2019) or messages (Du, 2021).

Driver state induced by non-driving-related tasks was shown to impact takeover performance (Naujoks, 2018). N-back task is widely used as a non-driving-related task, and its impact on mental workload has been extensively researched.

Regarding the external environment, Capallera et al. (Capallera, September 2019) showed that it is a potential cause of takeover. This category included the weather, the shape of the road, the state of the lane markings, and so on. In addition, Li et al. (Li, 2018) demonstrated the impact of adverse weather (such as rain, fog and snow) on the takeover quality. As such, we decided to focus on the weather condition in our study.

Another main point to consider is that the agent relies on takeover quality prediction to make its recommendation. Research shows that the quality of the takeover can be predicted using Machine Learning. Du et al. got a f1-score of 64% predicting takeover performance in two categories: good or bad (Du, 2020). Pakdamanian et al. predicted with a score of 83% a driver subjective measure of takeover quality (Pakdamanian, 2021). These studies considered classes of quality, where our agent uses a regression to predict directly the takeover quality metrics. Takeover quality can be measured by different metrics, such as the reaction time, which is the time difference between the TOR and the moment the driver takes the control of the vehicle, or the maximum steering wheel angle (MaxSWA).

METHODOLOGY

Data Collection A 50 minutes driving session took place on a fixed-base driving simulator, where 15 drivers’ physiological signals were recorded. The signals were EDA, ECG, and respiration of the drivers. There were 9 takeover situations during the driving session, caused by a fixed obstacle with a time-to-collision of around 7 seconds. The drivers had to perform three different tasks under two different weather conditions (sunny weather and adverse weather).

The tasks were:

1. A visual 2-back task, where the driver has a handheld device showing a series of letters one-by-one, and had to press a button each time the current letter appearing was the same as the one presented 2 trials ago (cf. Figure 1).

2. An auditory 2-back task, which was essentially the same task except the

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 letters were presented audibly.

3. A monitoring task, where the driver had to monitor the road at all times.

Figure 1. Illustration of the 2-back task, and expected behavior of the participant If the task score was deemed unsatisfactory, in our case defined as a score equal or below 50%, the current round (until the next takeover) was discarded. This allowed us to discard situations where the participants did not engage properly in the task, which reduces the driver state impact on the takeover quality. The possible TOR modalities were combinations of haptic (vibrating seat), auditory (short audio chime) and visual (icon on the dashboard). Combinations tested were haptic-visual, auditory-visual, and haptic-auditory-visual. The takeover quality metrics were the reaction time between the TOR and the takeover, and the maximum steering wheel angle attained during the takeover process. Machine Learning Models
Original PDF Figure 1.

Takeover quality metrics were normalized and aggregated to create a unique label to predict takeover quality. Physiological features were processed from raw physiological signals using Neurokit (Makowski, 2021), and we applied three feature selection techniques to filter the original high number of features (175) to keep only the most relevant ones. First, we filter the trivial redundant features: constant features, duplicate features and features mostly empty. Then, a Pearson correlation analysis allowed us to refine our selection and drop too correlated features. Finally, a Mutual Information analysis further reduces our number of unnecessary features. This gave a final number of 22 usable physiological features.

After outlier suppression and data processing, 80 TOR were kept for the Machine

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Learning models (64 for training, 16 for testing). Data Augmentation methods, such as SMOGN (Branco, 2017) and Random Noise were implemented and tested to boost the training dataset.

KNeighbors Regressor, Support Vector Regressor, Random Forest Regressor and Neural Networks were trained using a grid search approach and cross validation. Scores were compared to a baseline defined as the constant prediction of the mean of the takeover quality metrics.

The final Machine Learning model and methodology is detailed in (de Salis, 2022).

TOR Modalities Recommendation The agent predicts the takeover quality using the ML model, with every possible set of modalities. It then chooses the modalities providing the best theoretical takeover quality. The agent was evaluated to avoid the following trivial situations:

1. The agent always chooses the same modalities, meaning that the training was biased toward one set of modalities.

2. The agent chooses a random modality because the choice has no impact on predicted performance, meaning that the agent was not able to capture the modalities’ impact on takeover quality.

A novel metric representing modality impact on takeover quality was defined and used to optimize agent behavior: we defined the takeover modality impact as the difference in percent between the highest takeover quality predicted between the three modalities and the lowest one. The formula can be summarized below: (Max(takeover_quality_prediction(modality1, …, modalityn) - Min(takeover_quality_prediction(modality1, …, modalityn)) /

Min(takeover_quality_prediction(modality1, …, modalityn) . We then analyzed the mean and standard deviation of this metric on the test dataset for our models.

RESULTS

The agent was able to predict takeover quality using a Random Forest Regressor with a MSE of 0.0261, beating the baseline (0.0600) by a ratio of 56.5%. See Table 1. for the results of each model.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Table 1: MSE and MAE scores achieved by each model (best one in bold) Model Name MSE
Original PDF Table 1:

CONCLUSIONS

In this paper, we propose an agent able to recommend the best modalities for a TOR in a set of three (haptic-visual, haptic-auditory-visual, and auditory-visual). This agent relies on the prediction of a Random Forest regressor predicting the takeover quality for a given set of modality, the external weather and the driver physiological state in the last 90 seconds prior to the TOR.

The implication of our agent is the possibility for creating a smart HMI designing TOR on the fly, making the transition of control between the car and the driver safer than traditional TOR.

Impacts on using such a HMI on User Experience must be further studied.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

ACKNOWLEDGMENTS

This work is part of the AdVitam project funded by the Hasler Foundation. We would also like to thank our colleagues who helped us during this project.

REFERENCES

Borojeni, S. S., Wallbaum, T., Heuten, W., & Boll, S. (2017, September). Comparing shape-changing and vibrotactile steering wheels for take-over requests in highly automated driving. In Proceedings of the 9th international conference on automotive user interfaces and interactive vehicular applications (pp. 221-225). Branco, P., Torgo, L., & Ribeiro, R. P. (2017, October). SMOGN: a preprocessing

approach for imbalanced regression. In First international workshop on learning with imbalanced domains: Theory and applications (pp. 36-50). PMLR. Capallera, M., Meteier, Q., Salis, E. D., Angelini, L., Carrino, S., Khaled, O. A., & Mugellini, E. (2019, December). Secondary task and situation awareness, a mobile application for semi-autonomous vehicle. Proceedings of the 31st Conference on LInteraction Homme-Machine - IHM 19. doi:10.1145/3366550.3372258 Capallera, M., Meteier, Q., de Salis, E., Angelini, L., Carrino, S., Khaled, O. A., &

Mugellini, E. (2019, September). Owner manuals review and taxonomy of ADAS limitations in partially automated vehicles. In Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 156-164). de Salis, E., Meteier, Q., Capallera, M., Pelletier, C., Angelini, L., Khaled, O. A., Mugellini, E., Widmer, M., & Carrino, S. (In Press). Predicting Takeover Quality in Conditionally Autonomous Vehicles based on Takeover Request Modalities, Driver Physiological State and the Environment. International Conference on Intelligent Human Systems Integration. Du, N., Zhou, F., Tilbury, D., Robert, L. P., & Yang, X. J. (2021, September).

Designing Alert Systems in Takeover Transitions: The Effects of Display Information and Modality. In 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 173-180). Du, N., Zhou, F., Pulver, E. M., Tilbury, D. M., Robert, L. P., Pradhan, A. K., & Yang, X. J. (2020). Predicting driver takeover performance in conditionally automated driving. Accident Analysis & Prevention, 148, 105748. Grah, T., Epp, F., Wuchse, M., Meschtscherjakov, A., Gabler, F., Steinmetz, A., &

Tscheligi, M. (2015, September). Dorsal haptic display: a shape-changing car seat for sensory augmentation of rear obstacles. In Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 305-312). Ko, S., Zhang, Y., & Jeon, M. (2019, September). Modeling the effects of auditory display takeover requests on drivers' behavior in autonomous vehicles. In Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings (pp. 392-398).

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Li, S., Blythe, P., Guo, W., & Namdeo, A. (2018). Investigation of older driver's takeover performance in highly automated vehicles in adverse weather conditions. IET Intelligent Transport Systems, 12(9), 1157-1165. Makowski, D., Pham, T., Lau, Z. J., Brammer, J. C., Lespinasse, F., Pham, H.,

Schölzel, C. & Chen, S. A. (2021). NeuroKit2: A Python toolbox for neurophysiological signal processing. Behavior Research Methods, 1-8. Naujoks, F., Höfling, S., Purucker, C., & Zeeb, K. (2018). From partial and high automation to manual driving: Relationship between non-driving related tasks, drowsiness and take-over performance. Accident Analysis & Prevention, 121, 28-42. Pakdamanian, E., Sheng, S., Baee, S., Heo, S., Kraus, S., & Feng, L. (2021, May).

Deeptake: Prediction of driver takeover behavior using multimodal data. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-14). SAE (2021) J3016C: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles - SAE International. Retrieved from https://www.sae.org/standards/content/j3016_202104/ Shah, A. H., & Lin, Y. (2020, September). In-Vehicle Lighting and its Application in

Automated Vehicles: A Survey. In 2020 Fifth Junior Conference on Lighting (Lighting) (pp. 1-4). IEEE. Yang, Y., Karakaya, B., Dominioni, G. C., Kawabe, K., & Bengler, K. (2018, November). An hmi concept to improve driver's visual behavior and situation awareness in automated vehicle. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC) (pp. 650-655). IEEE. Zhang, B., de Winter, J., Varotto, S., Happee, R., & Martens, M. (2019).

Determinants of take-over time from automated driving: A meta-analysis of 129 studies. Transportation research part F: traffic psychology and behaviour, 64, 285-307.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-7923-8989-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →