Emmanuel De Salis, Quentin Meteier, Marine Capallera, Colin Pelletier, Leonardo Angelini, Omar Abou Khaled, Elena Mugellini, Marino Widmer, Stefano Carrino (2022). Predicting Takeover Quality in Conditionally Autonomous Vehicles based on Takeover Request Modalities, Driver Physiological State and the Environment. In: Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems. AHFE Open Access, vol. 22. https://doi.org/10.54941/ahfe100989
ContextConditionally autonomous vehicles are studied for numerous reasons, due to the dual responsibility in the driving task, switching from the car to the driver depending on the situation. This transition of control, namely the takeover, is usually not a concern when initiated by the driver. However, when the car issues a takeover request (TOR), it does it without consideration for the driver state, or anything else. In such a situation, the driver can be out of the driving loop, unaware of the current environment or focusing on a non-driving-related task (NDRT), impacting the takeover quality.>ObjectiveIn this study, the goal is to find out if a Machine Learning can take advantages of many different factors and how they interact together, to create a model able to predict the takeover quality. Factors highlighted by the literature were considered: the driver physiological state, the external environment, the driver current activity and the modality of the TOR. Some studies tried to predict takeover quality, but they usually study the TOR modality impact, the NDRT or the driver state. In this paper, we attempt to study the bigger picture and grasps the impact of the interaction between those multiple factors. >Methodology>>Experiment15 drivers’ physiological signals (EDA, ECG and respiration) were recorded during a 50 minutes driving session on a fixed-base driving simulator. They encountered 9 takeover situations each, caused by a stationary obstacle appearing on a road with a time-to-collision of around 7 seconds. Physiological data were processed on the last 90 seconds before each TOR. The possible TOR modalities were combinations of visual (icon on the dashboard), auditory (short beep sound) and haptic (vibrating seat). Combinations tested were visual-haptic, visual-auditory and visual-auditory-haptic. The drivers had 3 task for each third of the driving session: Visual n-back task, auditory n-back task or monitoring the car (no task).Half the participants (8) had an adverse weather during the driving session, with low luminosity and heavy rain, while the other half (7) had a sunny weather. Reaction time between the TOR and the takeover, and the maximum steering wheel angle attained during the takeover process were recorded as takeover quality metrics. >>Machine LearningTakeover quality metrics were combined to create a unique label to predict. State-of-the-art Feature Selection techniques were applied to keep only to more relevant features, and after outliers suppressions and data processing, 80 takeovers were kept for the Machine Learning models training. Data Augmentation methods, such as SMOGN and Random Noise were implemented and tested to boost the training dataset. KNeighbors Regressor, Support Vector Regressor (SVR), Random Forest Regressors (RFR) and Neural Networks (NN) were trained using a grid search approach and cross validation. Evaluation was done using MSE, MAE and R2.>ResultsThe NN gave the best results, with a MSE of 0.0538, a R2 of 0.1040 and a MAE of 0.1614. The results are similar to the most recent literature. Most important physiological features included indicators of the Respiratory Sinus Arrhythmia, Heart Rate Variability and Skin Conductance Response
Conditionally autonomous vehicles are studied for numerous reasons, from their pedestrian detection systems (Boukerche, 2021) to their interaction with the driver. One common point from all these studies is to increase the safety of the use of such autonomous systems. There are numerous critical situations in driving scenarios and on the road in general, but one that is specific to conditionally autonomous vehicles is the transition of control of the driving task, between the car and the driver. This transition of control, namely the takeover, is usually not a concern when initiated by the driver. How-ever, when the car issues a takeover request (TOR), it does it without consideration for the driver state, or anything else. In such a situation, the driver can be out of the driving loop, unaware of the current environment or focusing on a non-driving-related task (NDRT), impacting the takeover quality.
Figure 1 shows the full takeover process:
1. Up until t0 the car is in autonomous mode.
2. At t0, the car detects that it cannot continue to be in autonomous mode, and emits a TOR.
3. The driver takes time to assess the situation, and takes the control back at t1.
4. From t1 to t2, the driver is in control of the car, taking care of the problematic situation that caused the TOR with the appropriate behavior.
5. At t2 the takeover is over and autonomous mode is resumed.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Figure 1. The takeover process
OBJECTIVE In this study, the goal is to find out if Machine Learning algorithms can take advantage of many different factors and how they interact together, to create a model able to predict the takeover quality. Factors highlighted by the literature were considered: the driver physiological state, the external environment, the driver current activity and the modality of the TOR.
Some studies tried to predict takeover quality, but they usually study the TOR modality impacts, the NDRT or the driver state. In this paper, we attempt to study the bigger picture and grasp the impact of the interaction between the multiple factors.
RELATED WORK
Factors Influencing Takeover Quality TOR are researched extensively due to their high criticality from a safety point of view, and the way they are conveyed to the driver, meaning their modalities, was shown to impact the quality of the takeover: usual modalities include the haptic modality (vibrating seat (Grah, 2015), shape-changing steering wheel (Borojeni, 2017), etc.), the visual modality (ambient lights (Shah, 2020), icon on a handheld device (Capallera, 2019, December), etc.) and the auditory modality (short or longer chime (Ko, 2019), voice message asking to take-over (Du, 2021), etc.). Those modality can be used individually or conjointly, and the multimodality of a TOR was shown to indicate urgency and lead to shorter reaction time, compared to a unimodal TOR (Zhang, 2019). Overall, review of the literature showed the potential for advanced UI in conditionally autonomous vehicles (Kim, 2021).
Another factor is the driver's psychophysiological state. Driver’s state was shown to affect takeover quality (Morales-Alvarez, 2020), and as such was considered as an important factor to be recorded in this study.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Also, the external environment (which can be the traffic, the type of road or the weather, for example) is a probable cause of takeover, as demonstrated by (Capallera, 2019, September). Moreover, Li et al. (Li, 2018) showed that adverse weather had an impact on takeover quality, especially snow and fog, but also rain.
Using ML to Predict Takeover Quality A review of the literature showed that there was very few research trying to predict takeover quality. Most notably, Du et al. (Du, 2020) tried to predict the drivers' subjective ratings of their takeover performance, with a f1-score of 70.1%, with a random forest model.
A previous study also tried to predict takeover quality, but considered way less fac-tors. It achieved a Mean Squared Error (MSE) of 1.69 for the reaction time, and 161.93 for the MaxSWA (de Salis, 2021, February).
Nonetheless, some research used ML for similar purposes: distraction detection using deep learning (Kouchak, 2019), driver drowsiness detection (Chirra, 2019), or clustering the different types of driver performance in a takeover scenario (de Salis, 2021, August).
METHODOLOGY
Data Collection Physiological signals of 15 participants were recorded during a 50 minutes rural driving session on a fixed-base driving simulator. Signals considered were Electrodermal Activity (EDA), Electrocardiogram (ECG) and respiration. The participants encountered 9 takeover situations each, caused by a fixed obstacle appearing on a road with a time-to-collision of around 7 seconds. Physiological data were processed in the last 90 seconds before each TOR, using the Neurokit library (Makowski, 2021).
The possible TOR modalities were combinations of visual (red icon on the dashboard), auditory (short chime) and haptic (vibrating seat). Combinations tested were visual-haptic, visual-auditory, and visual-auditory-haptic. The drivers had a different NDRT for each three consecutives takeover: Visual 2-back task, auditory 2-back task or monitoring the car (no task). They performed the task on a handheld device. Half the participants (8) encountered adverse weather during the driving session, with low luminosity and heavy rain, while the other half (7) were experiencing sunny weather.
Reaction time between the TOR and the takeover, and the maximum steering wheel angle attained during the takeover process are recorded as takeover quality metrics.
Machine Learning Takeover quality metrics were normalized and aggregated to create a unique label to predict. State-of-the-art Feature Selection techniques were applied to keep only to more relevant
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 features, and after outliers suppressions and data processing, 80 TOR were kept for the Machine Learning models training. Data Augmentation methods, such as SMOGN and Random Noise were implemented and tested to boost the training dataset. Random Noise gave the best results, making the model less sensible to overfitting and improving the final results more than SMOGN.
KNeighbors Regressor, Support Vector Regressor, Random Forest Regressors and Neural Networks from Scikit-Learn (Pedregosa, 2011) were trained using a grid search approach and cross validation. Evaluation was done using MSE and Mean Absolute Error (MAE).
RESULTS AND DISCUSSION
To compare our results to something similar, we calculated the score that the mean of the takeover quality would get if used as constant prediction. We call this “baseline” in the comparison of all models (see Table 1).
Table 1: MSE and MAE scores achieved by each model (best one in bold) Model Name MSE MAE
As we can see in Table 1, a MSE of 0.0261 is an improvement of 56.5% over the baseline, respectively 38.64% for the MAE, meaning our Random Forest is able to predict the takeover quality better than the constant prediction of the takeover quality mean.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Regarding the feature selection, there were 20 physiological features retained by the feature selection process, which are listed here (cf. the Neurokit documentation (Neurokit) for more information about the features details and calculations): EDA features:
phasic_EDA_freq_NS_SCRs
EDA_filtered_std
EDA_tonic_mean
EDA_filtered_std
phasic_EDA_freq_NS_SCRs
SCR_Peaks_N ECG and Heart rate variability (HRV) features:
ECG_Rate_Mean
ECG_Rate_Mean
HRV_CSI_Modified
HRV_C1a
HRV_Ca
HRV_CorrDim
HRV_VHF
HRV_CSI
HRV_PAS Respiration features:
RRV_DFA_2
RRV_MedianBB
RRV_MCVBB
RRV_MCVBB
RSP_Phase_Duration_Expiration Features regarding the weather condition and the TOR modalities were also kept during the feature selection process.
CONCLUSIONS
In this paper, we proposed a new model able to predict the takeover quality, which is a novel metric composed of the driver reaction time and MaxSWA. Results suggest that this task is possible from a Machine Learning point of view, allowing the use of this information in future HMI. More tests on the implication of this information should be conducted. The short time window prior to the TOR used to predict takeo-ver quality (90 seconds) seems to indicate that this model could be used in a real time scenario, even for a short period of autonomous driving. In this case, a real-time ver-sion of this model should be implemented and evaluated independently to pinpoint the consequences of using takeover quality as an input to selecting TOR modalities on the fly.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
ACKNOWLEDGMENTS
This work is part of the AdVitam project funded by the Hasler Foundation. We would also like to thank our colleagues who helped us during this project.
REFERENCES
Boukerche, A., & Sha, M. (2021). Design Guidelines on Deep Learning–based Pedestrian Detection Methods for Supporting Autonomous Vehicles. ACM Computing Surveys (CSUR), 54(6), 1-36. Borojeni, S. S., Wallbaum, T., Heuten, W., & Boll, S. (2017, September). Comparing
shape-changing and vibro-tactile steering wheels for take-over requests in highly automated driving. In Proceedings of the 9th international conference on automotive user interfaces and interactive vehicular applications (pp. 221-225). Capallera, M., Meteier, Q., de Salis, E., Angelini, L., Carrino, S., Khaled, O. A., & Mugellini, E. (2019, September). Owner manuals review and taxonomy of ADAS limitations in partially automated vehicles. In Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 156-164). Capallera, M., Meteier, Q., Salis, E. D., Angelini, L., Carrino, S., Khaled, O. A., &
Mugellini, E. (2019, December). Secondary task and situation awareness, a mobile application for semi-autonomous vehicle. Proceedings of the 31st Conference on LInteraction Homme-Machine - IHM 19. doi:10.1145/3366550.3372258. Chirra, V. R. R., ReddyUyyala, S., & Kolli, V. K. K. (2019). Deep CNN: A Machine Learning Approach for Driver Drowsiness Detection Based on Eye State. Rev. d'Intelligence Artif., 33(6), 461-466. de Salis, E., Meteier, Q., Capallera, M., Angelini, L., Sonderegger, A., Abou Khaled, O.,
Mugellini, E., Widmer, M., & Carrino, S. (2021, February). Predicting Takeover Quality in Conditionally Automated Vehicles Using Machine Learning and Genetic Algorithms. In Inter-national Conference on Intelligent Human Systems Integration (pp. 84-89). Springer, Cham. de Salis, E., Meteier, Q., Pelletier, C., Capallera, M., Angelini, L., Sonderegger, A., Mugellini, E., Widmer, M., & Carrino, S. (2021, August). Clustering of Drivers’ State Before Takeover Situations Based on Physiological Features Using Unsupervised Machine Learning. In Inter-national Conference on Human Interaction and Emerging Technologies (pp. 550-555). Spring-er, Cham. Du, N., Zhou, F., Pulver, E., Tilbury, D., Robert, L. P., Pradhan, A. K., & Yang, X. J.
(2020, April). Predicting takeover performance in conditionally automated driving. In Extended abstracts of the 2020 chi conference on human factors in computing systems (pp. 1-8). Du, N., Zhou, F., Tilbury, D., Robert, L. P., & Yang, X. J. (2021, September). Designing Alert Systems in Takeover Transitions: The Effects of Display Information and Modality. In 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 173-180). Grah, T., Epp, F., Wuchse, M., Meschtscherjakov, A., Gabler, F., Steinmetz, A., &
Tscheligi, M. (2015, September). Dorsal haptic display: a shape-changing car seat for sensory augmentation of rear obstacles. In Proceedings of the 7th International
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 305-312). Kim, S., van Egmond, R., & Happee, R. (2021). Effects of User Interfaces on Take-Over Performance: A Review of the Empirical Evidence. Information, 12(4), 162. doi:10.3390/info12040162 Ko, S., Zhang, Y., & Jeon, M. (2019, September). Modeling the effects of auditory display
takeover requests on drivers' behavior in autonomous vehicles. In Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings (pp. 392-398). Kouchak, S. M., & Gaffar, A. (2019, September). Driver distraction detection using deep neural network. In International Conference on Machine Learning, Optimization, and Data Science (pp. 13-23). Springer, Cham. Li, S., Blythe, P., Guo, W., & Namdeo, A. (2018). Investigation of older driver's takeover
performance in highly automated vehicles in adverse weather conditions. IET Intelligent Transport Systems, 12(9), 1157-1165. Makowski, D., Pham, T., Lau, Z. J., Brammer, J. C., Lespinasse, F., Pham, H., Schölzel, C., & Chen, S. A. (2021). NeuroKit2: A Python toolbox for neurophysiological signal processing. Behavior Research Methods, 53(4), 1689–1696. https://doi.org/10.3758/s13428-020-01516-y Morales-Alvarez, W., Sipele, O., Léberon, R., Tadjine, H. H., & Olaverri-Monreal, C.
(2020). Automated Driving: A Literature Review of the Take over Request in Conditional Automation. Electronics, 9(12), 2087. doi:10.3390/electronics9122087 Neurokit Documentation¶. (n.d.). Retrieved from https://neurokit.readthedocs.io/en/latest/documentation.html Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., ... & Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12, 2825-2830. Shah, A. H., & Lin, Y. (2020, September). In-Vehicle Lighting and its Application in
Automated Vehicles: A Survey. In 2020 Fifth Junior Conference on Lighting (Lighting) (pp. 1-4). IEEE. Zhang, B., de Winter, J., Varotto, S., Happee, R., & Martens, M. (2019). Determinants of take-over time from automated driving: A meta-analysis of 129 studies. Transportation re-search part F: traffic psychology and behaviour, 64, 285-307.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-7923-8988-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional