Nicolás Palomares, Juan Manuel Belda-Lois, Begoña Mateo, Sofía Iranzo, Javier Silva, Víctor De Nalda-Tárrega, Jose Laparra, Jose Solaz (2022). Development of empathic autonomous vehicles through understanding the passenger’s emotional state. In: Human Factors in Transportation. AHFE Open Access, vol. 60, pp. 78–86. https://doi.org/10.54941/ahfe1002436
Emotion recognition is crucial to increase user acceptance in autonomous driving. SUaaVE project aims to formulate ALFRED, defined as the human-centered artificial intelligence to humanize the vehicle actions by estimating the emotions felt by the passengers and managing preventive or corrective actions, providing tailored support. This paper presents the development of an emotional model able to estimate the values of valence (how negative or positive a stimulus is) and arousal (the level of excitement) from the analysis of physiological signals. The model has been validated with an experimental test simulating different driving scenarios of autonomous vehicles. The results found that driving mode can influence the emotional state felt by the passengers. Further exploration of this emotional model is therefore advised to detect on board experiences and to lead to new applications in the framework of empathic vehicles.
Improving user experience in highly autonomous vehicles is essential to increase their acceptance (Oehl et al., 2019). The goal is to create a direct connection to the human experience (Izquierdo-Reyes et al., 2018). One possibility to realize this is the design of empathic cars that are capable of assessing the emotional state of vehicle occupants and react to it accordingly by providing tailored support (Oehl et al., 2019). Confused or scared drivers could be supported if emotions are recognized on time (Bosch et al., 2018). In this regard, understanding the driver style in an automated vehicle can adjust the vehicle operation to the driver’s needs (Barosan, 2020).
Given that acceptance is tightly related to emotional processes and trust in a new technology, H2020 SUaaVE project “SUpporting acceptance of auto- mated VEhicle” aims to understand the passengers’ state during the trip (in real-time) and to adapt the features of automated vehicles to enhance the in-vehicle user experience, and therefore acceptance.
Development of Empathic Autonomous Vehicles Through Understanding 79
This paper presents the validation of the emotional model in a test with participants experiencing two vehicle’s dynamic behavior (Comfort and Sport modes) in an autonomous vehicle through a driving simulator. It is studied how driving style can affect the emotional state of the passengers in autono- mous vehicles. The remainder of this article is organized as follows: Section 2 describes the theoretical framework and previous researches to identify the emotional state. Section 3 presents the methodology to conduct the test, the physiological data gathered and the method followed to build the emotional model. The results are then presented and discussed in Section 4 analyzing the emotion obtained by the emotional model and the emotions reported by the participants. Finally, Section 5 provides conclusions and future research directions.
RELATED RESEARCH Emotion can be defined as psychological states comprised of multiple inter- related processes such as cognitive appraisals, physiological processes, beh- avioral action tendencies, and the phenomenological experience of feelings (Chang and Smith, 2015). There are two main approaches to classify emo- tions, categorical and dimensional. The categorical theory proposes the existence of basic, discrete and universal emotions, with clear boundaries separating emotional states. According to this approach, there are a num- ber of basic emotions such as anger, disgust, happiness, sadness, fear, and surprise (Ekman, 1992). Other emotions can be arranged as subcategories of these emotions (Suk, 2006). However, discrete emotions can be unable to cover a complete space of emotions in all cases since human emotions corre- spond to subtle variation within each category (Novikova and Watts, 2014). The dimensional theory aims to classify the emotions in the gap in the discrete approach and proposes the existence of the two fundamental dimensions of valence (the positive or negative connotation of the emotion) and arousal how calming or exciting the emotion is. The dimensional Russell’s circumplex model states that each emotional state may be continuously represented as a linear combination of values in these two dimensions as shown in Figure 1 (Russell, 1980).
Different studies have revealed that the emotional state can be estimated from the values of valence and arousal from different methods on emotion recognition research. One of the most widely used method is the analysis of physiological signals, such as electrocardiogram (ECG), Galvanic Skin Response (GSR) or electromyogram (EMG). The main advantage of conside- ring physiological signals is that they are not under the voluntary control of humans (Shu et al., 2018), other techniques (such as face expressions or tone voice), which not always represent feelings (Yıldırım and Varol, 2017).
ECG signal reflects the electrical activity of the heart and is proven its rela- tionship with the emotional states. Heart-related biological signals, such as heart rate variability (HRV), provide useful information about human emo- tions (Yee Chung et al., 2021). GSR (derived from the sweat reaction) is also modulated autonomously by sympathetic activity which drives aspects of human behavior, as well as cognitive and emotional states (Critchley, 2002).
Figure 1: Russell’s circumplex model positions emotions on a two-dimensional space (valence and arousal).
EMG detects the muscle activation by the brain depending on emotional status (Mithbavkar and Shah, 2021) and it is broadly used in research for emotion recognition. Besides, although most studies are focused on a single physiological variable, researchers are investigating methods that use several biosignals in conjunction because of the difficulty to recognize with higher degree of accuracy the emotions based on a single signal (Kim et al., 2020).
The importance of emotions has motivated the researchers in several sci- ence disciplines to develop methods to recognize emotions. In the case of automotive field, the state of the drivers have been widely employed in human factors research with the purpose of studying, among others, mental wor- kload, stress, fatigue, distraction or drowsiness. With recent technological innovations (such as wearable technologies or cameras for facial emotion recognition), the researchers are also exploring the effects on emotional sta- tes of driving (e.g. anger can involve a negative impact while driving) (Zepf et al., 2020). In recent years, different studies have indicated the relevance of addressing the effects of the emotions on autonomous vehicles, particularly with high levels of automatization where the driver becomes a passenger and is no longer required to drive. In these cases, lite is known about the occu- pant’s emotion considering that they will no longer rely on the driver but rather on the car to behave in an acceptable and trustworthy way. Establi- shing and maintaining trust between occupants and the machine intelligence controlling the vehicle is reportedly one of the biggest challenges faced by the industry (Miller et al., 2016). In this regard, future studies are needed to inve- stigate the impact of changes on drivers’ behavior and acceptance (Melman et al., 2021).
METHODOLOGY
Experimental Setup To provide participants with tangible experiences of automated driving featu- res, they drove the Human Autonomous Vehicle (HAV) at IBV, an advanced dynamic driving simulator (six degrees of freedom) that allows to emulate the behavior of a vehicle with different degrees of autonomy enabling fully immersive driving experience. It includes steering wheel and pedals for
Development of Empathic Autonomous Vehicles Through Understanding 81
Figure 2: HAV simulating the behavior of an autonomous vehicle in a highway.
simulating different levels of automation, together with simulated audio and virtual panels instrument providing time to destination, vehicle status and another trip information (Figure 2). The simulator runs using CARLA software able to simulate different road and environment conditions.
Participants A total of 40 volunteers integrated the subject sample in the test. This sample is composed of car drivers (males and females) between 25 and 55 years old, aged and BMI balanced. Inclusion criteria included holding a current dri- ving license, not to suffer visual & hearing impairment (wearing glasses was allowed), regular driving (at least twice a week on average), and not being a professional driver. After a detailed explanation of the experimental pro- cedure, all participants provided informed consent. Our study was approved by the Ethical Committee of the Polytechnic University of Valencia (UPV).
Experimental Procedure Participants got familiar with the driving simulation by first carrying out a 5 min manual driving. This practice consisted in driving a car in a residential area with low traffic and without pedestrian interaction following a specific route indicated by the technician in charge of the test. After it, the participants were informed about simulating several scenarios in autonomous mode. This paper focused on the results of Comfort scenario and Sport scenario. Both scenarios complete the same route (urban area and highway) and the same external conditions (traffic, weather). The only difference is the automated driving mode simulated:
Comfort scenario (duration: 6’20”): Smooth and slow driving mode.
Sport scenario (duration: 3’25”): Aggressive and fast driving mode.
Figure 3: SAM questionnaire.
The scenarios were presented in a pseudo-randomized order so that 50% participants started with the Sport scenario and 50% started with Comfort scenario. After each scenario, all participants were asked to report the emo- tions felt regarding specific events through the scales of valence and arousal from the Self-Assessment Manikin (SAM) (Bradley and Lang, 1994). The value of valence in a scale from 1 to 9 refers to the negative or positiveness of the emotion felt. In the same way, arousal refers to the intensity of the emotion in terms of calmness or excitation (Figure 3).
Data Analysis. Emotional Model The emotion model is a Bayesian Classifier. The model has been trained through the Continuously Annotated Signals of Emotion (CASE) dataset, which is composed by synchronized physiological recordings (ECG, BVP, EMG (3x), GSR) of 30 subjects watching several emotion-inducing videos together with the instantaneous emotional state in a valence-arousal space. The model has been built around 5 groups of valence and arousal obtained by k-means cluster analysis of the variables.
Per each group we have recorded the center in the emotion plane (Ai, Vi) and the probability distribution function (pdf) of each of the variables incor- porated in the model. Therefore, the probability of being in each of the valence and arousal groups is given by:
P((Vi, Ai)|v1, ...vn) = L(v1|(Vi, Ai)) · ...L(vn|(Vi, Ai))cdotP(Vi, Ai) When all the probabilities are known, the average is calculated as the current emotional state.
X (V, A) = ((Vi, Ai) · ((Vi, Ai)|v1, ...vn)
For this model, the best physiological signals that predict the emotional state were GSR and ECG. The physical modelling of the GSR has been ach- ieved by a Kalman Filtering of the GSR signal, decomposing this way the signal into the Tonic component, the Phasic component and the Neural Drive. Regarding ECG, HR have been decomposed in three frequency bands (Very
Development of Empathic Autonomous Vehicles Through Understanding 83
Figure 4: Mean values of valence and arousal estimated by the emotional model for Comfort and Sport scenarios. Error bars depict 95% confidence intervals.
Low Frequency Band, Low Frequency Band and High Frequency Band) by implementing a follower of frequencies in the HR signal centered in each fre- quency band as a Kalman Filter in which the matrix of transitions of states and the output matrix are calculated to each new sample. This approach in the model building allows the estimation of the emotional state in real time in a valence-arousal space in a scale from 1 to 9.
RESULTS
The emotional model was applied to calculate the emotional state of the participants (valence and arousal values), in the main events of the driving scenarios analyzed. A comparison between Comfort and Sport scenarios was performed through a one-way ANOVA (factor: scenario) for model varia- bles valence and arousal (Figure 4). The results indicated that that there were significant differences in the values of valence (p < 0.05). Comfort scenario obtains higher values of valence comparing with Sport Scenario. Regarding the value of arousal, it was also found significant differences between the sce- narios (p < 0.05). The arousal in Comfort scenario is lower than the obtained in Sport scenario.
Differences between Comfort and Sport scenario can be also seen with the individual values of valence and arousal estimated by each event that chara- cterize them (6 events for Comfort scenario and 10 events for Sport scenario). As can be observed in Figure 5, comfort events are concentrated in terms of valence and arousal, whereas the events in sport scenario obtain higher variability. Certain events of Sport scenario elicits lower values of valence and higher values of arousal whereas some events elicit similar levels than Comfort scenario events.
For these scenarios, self-reported emotional state from the Self-Assessment Manikin (SAM) were also compared. In this case, a non-parametric Friedman test (factor: scenario) has been considered to analyze the effect of the driving mode in the emotional response declared by participants (Figure 5). In case of valence, no significant differences were found due to this factor. On the other hand, results show an effect of the scenario in the values of arousal (p <
Figure 5: Mean values of valence and arousal estimated by each event that characterize Comfort and Sport scenarios.Figure 6: Mean values of valence and arousal reported by participants for Comfort and Sport Scenarios. Error bars depict 95% confidence intervals.
0.05). Comfort scenario obtains lower values of arousal reported comparing with Sport scenario.
CONCLUSION
This research has shown that it is possible to understand the emotional state of the passenger in the context of autonomous vehicles from the emotional model developed based on physiological signals. The results found that the driving mode of the vehicle can influence the emotional state. Sport scena- rio obtains lower values of valence and higher values of valence comparing with Comfort scenario. In case of arousal, these results are in line with the subjective feeling reported by participants. Therefore, we can suppose that depending on the speed and maneuvering of the vehicle, passengers can expe- rience different emotions felt so that it can influence the acceptance of this technology.
The emotional model has positive implications for the future mobility enabling a better integration of human factor in the deployment of autono- mous vehicles and leading to new applications in the framework of empathic vehicles, aimed to understand the state of the passengers. High levels of automation (Level 4 and Level 5) will involve a disruptive change in the
Development of Empathic Autonomous Vehicles Through Understanding 85
user experience in the vehicle. Passengers will have to rely on the vehi- cle’s behavior and the vehicle will have to avoid negative experiences. The empathic vehicles will open the possibility of managing the driving mode in real time considering the current emotional state of the passenger, which can be influenced, among others, by the external conditions, purpose of the trip and time to get to destination, activities to be performed during the trip (work, leisure, social) or own driving preferences. The applica- tion of the emotional model proposed is based on predicting the state of the passengers through monitoring systems. Although the technology of monitoring is still in progress, there have been recently great improvements resulting from the shrink costs and growing accuracy of biometric sen- sors, such as smart and wearable devices and non-intrusive devices, based on cameras to record breathing and to monitor heart rate. As technology progresses, these technological developments will be more accurate and fea- sible, allowing to actively monitor the passengers in future autonomous vehicles.
As future work, the emotional model will be applied for the rest of scena- rios tested to explore the influence of different contextual events as emotional triggers and to further analyze the model capability and its operating boun- dary conditions in the estimation of the emotional state. In the next stages of the project, the research will be also focused on deepening the study of mitigation strategies to avoid negative experiences (such as stress, fatigue or boredom) and to improve passengers state, thereby building trust with the vehicle.
ACKNOWLEDGMENT
This work was funded by the European Union’s Horizon 2020 Research and Innovation Program SUaaVE project: “SUpporting acceptance of automated Vehicles”; under Grant Agreement No. 814999.
REFERENCES
Barosan, I.I., 2020. CarESP: an emotion vehicle with stress, personality and embodiment of emotions. Bosch, E., Oehl, M., Jeon, M., Alvarez, I., Healey, J., Ju, W., Jallais, C., 2018. Emo-
tional GaRage: A workshop on in-car emotion recognition and regulation, in: Adjunct Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. pp. 44–49. Bradley, M.M., Lang, P.J., 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 49–59. Chang, L.J., Smith, A., 2015. Social emotions and psychological games. Current
Opinion in Behavioral Sciences 5, 133–140. Critchley, H.D., 2002. Review: Electrodermal Responses: What Hap- pens in the Brain. Neuroscientist 8, 132–142. https://doi.org/10.1177/ 107385840200800209 Ekman, P., 1992. Are there basic emotions?
Izquierdo-Reyes, J., Ramirez-Mendoza, R.A., Bustamante-Bello, M.R., Pons-Rovira, J.L., Gonzalez-Vargas, J.E., 2018. Emotion recognition for semi-autonomous veh- icles framework. International Journal on Interactive Design and Manufacturing (IJIDeM) 12, 1447–1454. Kim, T.-Y., Ko, H., Kim, S.-H., 2020. Data Analysis for Emotion Classification Based
on Bio-Information in Self-Driving Vehicles. Journal of Advanced Transportation 2020, e8167295. https://doi.org/10.1155/2020/8167295 Melman, T., de Winter, J., Mouton, X., Tapus, A., Abbink, D., 2021. How do driving modes affect the vehicle’s dynamic behaviour? Comparing Renault’s Multi-Sense sport and comfort modes during on-road naturalistic driving. Vehicle system dynamics 59, 485–503. Miller, D., Johns, M., Mok, B., Gowda, N., Sirkin, D., Lee, K., Ju, W., 2016. Beh-
avioral measurement of trust in automation: the trust fall, in: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. SAGE Publications Sage CA: Los Angeles, CA, pp. 1849–1853. Mithbavkar, S.A., Shah, M.S., 2021. Analysis of EMG Based Emotion Recognition for Multiple People and Emotions, in: 2021 IEEE 3rd Eurasia Conference on Bio- medical Engineering, Healthcare and Sustainability (ECBIOS). Presented at the 2021 IEEE 3rd Eurasia Conference on Biomedical Engineering, Healthcare and Sustainability (ECBIOS), pp. 1–4. https://doi.org/10.1109/ECBIOS51820.2021. 9510858 Novikova, J., Watts, L., 2014. A design model of emotional body expressions in
non-humanoid robots, in: Proceedings of the Second International Conference on Human-Agent Interaction, HAI ‘14. Association for Computing Machinery, New York, NY, USA, pp. 353–360. https://doi.org/10.1145/2658861.2658892 Oehl, M., Pape, A.-A., Ihme, K., Vukeli´c, M., Bosch, E., Braun, M., 2019. Emotions in the Age of Automated Driving. Russell, J.A., 1980. A circumplex model of affect. Journal of personality and social
psychology 39, 1161. Shu, L., Xie, J., Yang, M., Li, Ziyi, Li, Zhenqi, Liao, D., Xu, X., Yang, X., 2018. A review of emotion recognition using physiological signals. Sensors 18, 2074. Suk, H.-J., 2006. Color and Emotion- a study on the affective judgment across media
and in relation to visual stimuli [WWW Document]. None. URL http://ub-madoc .bib.uni-mannheim.de/1336 (accessed 2.18.22). Yee Chung, J.W., Fuk So, H.C., Tak Choi, M.M., Man Yan, V.C., Shing Wong, T.K.,
2021. Artificial Intelligence in education: Using heart rate variability (HRV) as a biomarker to assess emotions objectively. Computers and Education: Artificial Intelligence 2, 100011. https://doi.org/10.1016/j.caeai.2021.100011 Yıldırım, N., Varol, A., 2017. A research on estimation of emotion using EEG signals and brain computer interfaces, in: 2017 International Conference on Computer Science and Engineering (UBMK). Presented at the 2017 International Conference on Computer Science and Engineering (UBMK), pp. 1132–1136. https://doi.org/ 10.1109/UBMK.2017.8093523 Zepf, S., Hernandez, J., Schmitt, A., Minker, W., Picard, R.W., 2020. Driver emotion
recognition for intelligent vehicles: A survey. ACM Computing Surveys (CSUR) 53, 1–30.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Transportation
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-36-0
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional