Ramona Schmid, Linn Braunmiller, Lena Hansen, Christopher Schonert, Knut Möller, Verena Wagner-Hartl (2022). Emotion recognition – Validation of a measurement environment based on psychophysiological parameters. In: Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems. AHFE Open Access, vol. 22. https://doi.org/10.54941/ahfe1001065
Emotions are an essential part of our daily social interaction. They are highly complex processes and can cause different physiological, behavioral, and cognitive changes in individuals. A person for whom it is difficult or impossible to interpret emotions may face major problems in everyday life, e.g., patients with autism spectrum disorders. Understanding emotions is not only of great importance in private social interactions but also in the working environment, e.g., for managers or collaborative work. Initial research aims to positively influence behavior and outcomes of social interactions at work by emotion training. Additionally, the COVID-19 pandemic limited social interactions drastically. Due to isolation and contact restrictions, it is important to understand the impact on our ability to recognize emotions. Hence, there is a great interest in emotion research, including how emotions can be measured. For this purpose, a measuring environment was implemented that allows to detect a person’s emotional state. The aim of the presented study was to validate the measurement environment by evoking different emotions. As stimulus, affective visual material (EmoPicS) was used to address different emotional poles. To make emotions measurable, a multidimensional approach combining subjective and objective measures according to Boucsein and colleagues was chosen. Therefore, participants rated their emotional state using the Self-Assessment Manikin (SAM) on the three different affective dimensions: valence, arousal, and dominance. Also, facial expression was recorded as subjective measure. The objective psychophysiological measures were electrocardiogram (ECG), electrodermal activity (EDA) and electromyogram on the forearm and neck (EMG). Results prove a successful validation of the measurement environment. Furthermore, first findings show a significant difference regarding the subjective assessment of the affective visual material for the dimension valence. Nevertheless, no significant differences were found in the psychophysiological data for the stimulus material used. Future studies should use stronger stimuli to evoke emotions, e.g., by combining auditory and visual stimuli or using videos.
Emotions are defined as internal, psychological processes and are an essential part of our daily social interaction (Frenzel et al. 2009). They can be described as “(…) multidimensional constructs with affective, expressive, physiological, and motivational components (…).” [ibid., p. 207; direct translation from German]. Therefore, emotions are highly complex processes and can cause different physiological, behavioral, and cognitive changes in individuals. Attempts to identify basic dimensions of an emotion were made early in emotion research (Russell 1980; Shaver et al. 1987; Wundt 1896). Feldman (1995) defined 16 mood terms based on the dimension’s valence (positive or negative) and arousal (level of activation, from low to high). According to the three-factor theory of Russel and Mehrabian (1977), each emotional feeling can also be valued in a third dimension of dominance, in addition to valence and arousal.
However, not all people are able to interpret emotions adequately in all situations. For patients with autism spectrum disorders (ASD) emotion recognition may be difficult or impossible to achieve (Hartl 2010). Therefore, affected persons have qualitative impairments in social interactions, communication and exhibit limited, stereotypical patterns of behavior, interests, and activities. Therapeutic interventions such as emotion trainings are intended to help them to cope better in social interactions and everyday life (Golan et al. 2010; Kandalaft et al. 2013; Yuan and Ip 2018). Nevertheless, recognizing emotions is not only a central issue for patients with ASD. Due to isolation caused by the current COVID-19 pandemic, social contacts have been drastically reduced. Meléndez et al. (2020) showed that a confinement situation significantly alters and reduces our social interactions, which can affect our mood as well as our emotional facial recognition. Hence, the ability to recognize emotions is of great importance in general. Emotion recognition can be essential in our everyday life but also in the working environment, e.g., for managers or for collaborative work. Studies already showed that emotion training for adults improves their ability to recognize emotions (Schlegel et al. 2017). In the working environment, emotion training has also been shown to improve interpersonal effectiveness (Schlegel and Hall 2021) and negotiation outcomes (Elfenbein et al. 2007). Emotion recognition is, therefore, of great interest in research. In order to study the effects of emotion training, it is important to make the emotional state of a person measurable.
One of the greatest challenges in emotion research is the detection or measurement of emotions (Frenzel et al. 2009). Emotions are characterized by subjective experience. One
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 way to measure them is to interview people directly or let them assess their emotions using questionnaires. For this purpose, a variety of assessment systems has been developed in research. There are various survey methods that function as self-description instruments based on language (e.g., Watson et al. 1988; Spielberger 2010). In addition, the Self- Assessment Manikin (Bradley and Lang 1994) was established according to the three-factor theory of emotions (Russell and Mehrabian 1977). The scales for measuring emotions are represented by pictograms and offer the advantage of a non-verbal assessment. Still, a number of methods for emotion recognition has also been developed that do not rely on the subjective experience of the participants. Ekman and Rosenberg (1997) developed the Facial Action Coding System (FACS), a technique to recognize emotions based on facial expressions. Another possibility to objectively measure emotional experience are psychophysiological measures (e.g., electrodermal and cardiovascular activity, electromyogram, etc.). They are widely used for example in product development or when performing different tasks (Boucsein 2006; Wagner 2014; Wagner and Kallus 2014; 2015; Mandryk and Atkins 2007; Yagi 2000). Advantages of the use of psychophysiological responses are that, under normal circumstances, they are not volitional controlled by individuals (Boucsein and Backs 2009) and that they can be measured using noninvasive techniques (Boucsein 2006).
Birkle, Weber, Möller and Wagner-Hartl (2022) have already concepted a measurement environment based on psychophysiological parameters that allows to detect a person’s emotional state. However, the suitability of the measurement parameters for emotion recognition still has to be determined. Therefore, the aim of this study is (a) to validate the measurement environment and (b) to investigate possible differences in participants' subjective and psychophysiological responses to different induced emotions.
MATERIALS AND METHOD A laboratory study was conducted using a within subject design. The study contained two main parts – these were an emotion and a stress measurement. In this paper, only the emotion measurement is described.
Participants A total of eight participants participated in the study. 50% of them were female and 50% male. The age of the participants ranges between 20 and 27 years (M = 23.38, SD = 2.45). All of them were undergraduate students at Furtwangen University (Campus Tuttlingen, Germany). They took part in the study voluntarily and received no compensation. All participants provided their informed consent at the beginning of the study.
Materials
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
To evoke different emotions of the participants the database Emotional Picture Set (EmoPicS) (Wessa et al. 2010) was used. The EmoPicS is a standardized set of emotionally salient pictures, which are already rated in their valence and arousal. The images selected for this study are intended to induce positive, neutral, and negative emotions. Therefore, the following five picture categories were chosen: joy 1 (animals), joy 2 (humans), serenity (plants), sadness (humans), and fear (humans and animals). The categories were presented permuted. Five pictures per category were shown to the participants on a 23-inch hp screen at a distance of 50 cm.
In addition, the Recovery-Stress Questionnaire (RestQ) - short version A (Kallus and Bognar 2018; Kallus and Kellmann 2016) was used. The RestQ evaluates the current state (last three days) of the participants' recovery and stress (Kallus and Bognar 2018). Also, the Big Five Structure Inventory (BFSI) - screening form S2 (Arendasy 2018) was used to determine a person’s big five dimensions emotional stability, extraversion, openness, conscientiousness, and agreeableness. Both were conducted using the Vienna Test System by SCHUHFRIED GmbH.
In order to measure the emotional responses, a multidimensional approach according to Boucsein and colleagues (Boucsein et al. 2002; Boucsein and Schaefer 2008) was chosen. This combines subjective assessments of the participants as well as psychophysiological measurements and technical parameters of different products. In the presented study only two of the three measures, subjective assessments and psychophysiological parameters were included.
Subjective measurement The Self-Assessment Manikin (SAM) (Bradley and Lang 1994) was used for the subjective assessment after each picture category (5-point rating scale based on pictograms). The participants were asked to rate their emotional state on the three scales: valence, arousal, and dominance. In addition, facial expressions of the participants were recorded via a Logitech C925e webcam during the whole study. After each picture category, they were asked to represent their emotional state by two facial expressions. With the first one, the participants should show their actual feelings immediately after a specific picture category was shown. With the second facial expression they should imitate how they think they looked while looking at the pictures.
Psychophysiological measurement Cardiovascular activity (ECG), electrodermal activity (EDA), and electromyography (EMG) were used as objective parameters. ECG and EDA data were collected using systems of Movisens (Movisens GmbH 2021), the EMG data using a system of TEA Ergo (TEA Ergo Inc. 2021). The sensor for ECG was attached to a chest strap. For the EDA, the skin conductance of the inner palm (thenar and hypothenar) of the participants' non-dominant hand was recorded. EMG data was derived on the upper trapezius (musculus trapezius pars
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 descendens) and forearm muscle (musculus flexor carpi radialis) using adhesive electrodes. For statistical analysis the measures used for ECG were heart rate (HR) in beats per minute and heart rate variability (HRV RMSSD). The following parameters were collected for the EDA: skin conductance level (SCL), amplitude of non-specific electrodermal responses (NS.SCR amp), frequency of non-specific electrodermal responses (NS.SCR freq) and mean sum amplitude (NS.SCR amp/NS.SCR freq). For EMG the root mean square amplitude (RMS) was used.
Procedure At the beginning of the study, the socio-demographic data of the participants were collected through a short questionnaire. Afterwards, the electrodes were positioned and fixed for the psychophysiological measurement. After checking the functionality of the electrodes, the participants were briefed. The measurement started with a five-minute baseline measurement. This was followed by the presentation of the picture categories. In each of the five categories, five single pictures were shown one after the other, each lasting 12 seconds (see also materials section). Each picture category was presented for a total of 60 seconds. After each category the participants had to rate their emotional state with the SAM (Bradley and Lang 1994) and do the facial expressions subsequently. In addition, a subsequent 2.5- minute rest measurement was taken after each picture category. At the end of the study, the Recovery-Stress Questionnaire (RestQ) (Kallus and Bognar 2018; Kallus and Kellmann 2016) and the Big Five Structure Inventory (BFSI) (Arendasy 2018) were conducted.
Statistical Analysis For the statistical analysis of the data the software IBM SPSS Statistics was used. The psychophysiological measures were baseline corrected. Analyses of variance with repeated measures were used as statistical procedure. The evaluation was based on a significance level of 5%.
RESULTS
Measurement environment Results prove a successful validation of the measurement environment (Birkle et al. 2022). All parameters were recorded without any technical problems. Furthermore, the data check showed the combined measurement methods were robust against artifacts and did not interfere with each other.
Responses to the emotional stimuli
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
First, the results of the subjective assessment of the visual material on the dimensions of valence, arousal, and dominance of the SAM (Bradley and Lang 1994) are reported. The different picture categories were assessed significantly different in the dimension valence by the participants, F(4, 4) = 84.42, p ≤ .0001, η²part = .988 (see Figure 1). The pictures of the category joy 1 (M = 1.75, SD = .46) and joy 2 (M = 1.50, SD = .54) were experienced as significantly more positive than the pictures of the category’s sadness (M = -.25, SD = 1.04; joy 1: p = .011; joy 2: p = .008) and fear (M = -.63, SD = .74; joy 1: p ≤ .0001; joy 2: p = .011). Moreover, the category joy 1 was experienced significantly more positive than serenity (M = 1.00, SD = .54; p = .025) and serenity significantly more positive than fear (p = .015).
Note. * … p ≤ .05; I … standard error of mean
Figure 1. Subjective assessment of the valence of the visual material No significant differences can be shown for the assessment of the dimension arousal for the picture categories, FHF(2.41, 16.88) = 1.64, p = .222, η²part = .190. There are no significant differences for the dimension dominance either, F(4, 4) = .58, p = .693, η²part = .369. The results of the statistical analyses of the psychophysiological parameters of the cardiovascular (ECG: HR and HRV), electrodermal (EDA: SCL, NS.SCR amp, NS.SCR freq, and mean sum amplitude) and muscular (EMG: RMS) activity showed no significant differences for the stimulus material used.
DISCUSSION
The aim of this study was (a) to validate a measurement environment for recognizing
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 emotions based on psychological parameters (Birkle et al. 2022) and (b) to investigate possible differences in participants' subjective and psychophysiological responses to different induced emotions. The results of the laboratory study show the suitability of the measurement environment, combining different psychophysiological measurements. All parameters were recorded without technical problems and the conducted measures did not interfere with each other. If they interfered with each other, it could have resulted in artefacts in the recorded data, which did not occur in the presented study. Therefore, a successful validation of the measurement environment can be determined.
To investigate possible differences in participants' subjective and psychophysiological responses to different induced emotions five different picture sets were assessed. Through this combination of psychophysiological and subjective measures the emotional state of a person should be adequately represented. Results of the subjective assessment show that emotions were evoked successfully according to the intention of the visual material. Nevertheless, no significant differences were found in the psychophysiological data for the stimulus material used. The used stimulus material may not have been able to evoke emotions that were strong enough to measure them via the used psychophysiological measures. One reason could be that the stimuli were only presented on one sensory channel, the visual system. On the other hand, when selecting the stimuli, care was taken to show acceptable pictures that were not too disturbing or frightening. The selection criteria used may have been too strict, which could have kept the effects small. Future studies should use stronger stimuli to evoke emotions, e.g., by combining auditory and visual stimuli or using videos.
One limitation of the presented study is the relatively small sample size as well as the fact that only young participants participated in the study. The explanation for this lies in the current COVID-19 pandemic situation. Due to the existing restrictions during the time when the study was developed (June 2021) like the prohibition of access for external persons, it was not possible to develop the whole planned sample. Thus, the sample was much smaller and homogeneous regarding age than originally planned. Consequently, further studies with a greater sample size are planned for the future.
In summary, the results prove a successful validation of the measurement environment. Regarding the subjective responses to the used visual stimulus material, the results show significant differences regarding the subjective assessments for the dimension valence but not for the dimensions arousal and dominance. In addition, no significant differences can be shown for the psychophysiological responses to the used stimulus material. In the future, stimulus material that addresses more senses (e.g., videos) should be developed. Furthermore, it is planned to use the measurement environment for different tasks (e.g., product development or to develop and evaluate trainings).
ACKNOWLEDGMENTS
The authors would like to thank Katharina Gleichauf and Jonas Birkle for their support, and all participants who participated. Partial support by a grant from the Institute of Applied Research (EmTRA-Tandem) is gratefully acknowledged.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
AUTHOR’S STATEMENT The authors state no conflict of interest. Informed consent has been provided from all participants of the study. The study was examined and approved by the ethics committee of the Furtwangen University.
REFERENCES
Arendasy, M. (2018). Manual Big-Five Struktur Inventar. Version 54-Revision 1. Mödling: SCHUHFRIED GmbH. Birkle, J., Weber, R., Möller, K. and Wagner-Hartl, V. (2022). Psychophysiological
Parameters for Emotion recognition – Conception and first evaluation of a measurement environment. (under review). In: Proceedings of the 5th International Conference on Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems. Taylor & Francis Group. Boucsein, W. (2006). Psychophysiologische Methoden in der Ingenieurpsychologie. In: Zimolong, B. and Konradt, U., eds. Sonderdruck aus Enzyklopädie der Psychologie: Themenbereich D Praxisgebiete: Serie III Wirtschafts-, Organisations- und Arbeitspsychologie. Band 2: Ingenieurspsychologie. Göttingen: Hogrefe, pp.317–358. Boucsein, W. and Backs, R.W. (2009). The Psychophysiology of Emotion, Arousal, and
Personality: Methods and Models. Duffy, V. G. (Ed.) Handbook of Digital Human Modeling. Research for Applied Ergonomics and Human Factors Engineering, pp.35- 1 – 35-18. Boucsein, W. and Schaefer, F. (2008). Objective emotional assessment of industrial products. In: Probing Experience. Springer, pp.69–76. Boucsein, W., Schaefer, F., Kefel, M., Busch, P. and Eisfeld, W. (2002). Objective emotional
assessment of tactile hair properties and their modulation by different product worlds. International Journal of Cosmetic Science, 24(3), pp.135–150. Bradley, M.M. and Lang, P.J. (1994). Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1), pp.49–59. Ekman, P. and Rosenberg, E.L. (1997). What the face reveals: Basic and applied studies of
spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA. Elfenbein, H.A., Foo, M.D., White, J., Tan, H.H. and Aik, V.C. (2007). Reading your Counterpart: The Benefit of Emotion Recognition Accuracy for Effectiveness in Negotiation. Journal of Nonverbal Behavior, 31(4), pp.205–223. Feldman, L.A. (1995). Valence focus and arousal focus: individual differences in the
structure of affective experience. Journal of personality and social psychology, 69(1), pp.153-166. Frenzel, A.C., Götz, T. and Pekrun, R. (2009). Emotionen. In: Wild, E. and Möller, J., eds.
Pädagogische Psychologie. Springer, Berlin, pp.205–231. Golan, O., Ashwin, E., Granader, Y., McClintock, S., Day, K., Leggett, V. and Baron-Cohen, S. (2010). Enhancing emotion recognition in children with autism spectrum conditions: An intervention using animated vehicles with real emotional faces. Journal of autism and developmental disorders, 40(3), pp.269–279.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
Hartl, M. (2010). Emotionen und affektives Erleben bei Menschen mit Autismus. Springer, Wiesbaden. Kallus, K.W. and Bognar, B. (2018). Manual Erholungs-Belastungs-Fragebogen. Version
23-Revision 5. Mödling: SCHUHFRIED GmbH. Kallus, K.W. and Kellmann, M. (2016). The recovery-stress questionnaires: user manual.
Pearson London, UK Kandalaft, M.R., Didehbani, N., Krawczyk, D.C., Allen, T.T. and Chapman, S.B. (2013).
Virtual reality social cognition training for young adults with high-functioning autism. Journal of autism and developmental disorders, 43(1), pp.34–44. Mandryk, R.L. and Atkins, M.S. (2007). A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies, 65(4), pp.329–347. Meléndez, J.C., Satorres, E., Reyes-Olmedo, M., Delhom, I., Real, E. and Lora, Y. (2020).
Emotion recognition changes in a confinement situation due to COVID-19. Journal of Environmental Psychology, 72(101518). Movisens GmbH. (2021). Available from: https://www.movisens.com/de/ [accessed 16 November 2021]. Russell, J.A. (1980). A circumplex model of affect. Journal of personality and social
psychology, 39(6), pp.1161–1178. Russell, J.A. and Mehrabian, A. (1977). Evidence for a three-factor theory of emotions.
Journal of research in Personality, 11(3), pp.273–294. Schlegel, K., Vicaria, I.M., Isaacowitz, D.M. and Hall, J.A. (2017). Effectiveness of a short audiovisual emotion recognition training program in adults. Motivation and Emotion, 41(5), pp.646–660. Schlegel, K. and Hall, J.A. (2021). The effects of emotion recognition training on
interpersonal effectiveness. Basic and Applied Social Psychology, 43(2), pp.141–153. Shaver, P., Schwartz, J., Kirson, D. and O’Connor, C. (1987). Emotion knowledge: further exploration of a prototype approach. Journal of personality and social psychology, 52(6), pp.1061–1086. Spielberger, C.D. (2010). State-Trait anxiety inventory. The Corsini encyclopedia of
psychology, 2010, pp.1–1. TEA Ergo Inc. (2021). Available from: https://www.teaergo.com/ [accessed 2 November 2021]. Wagner, V. (2014). Hochwertigkeit von Geräuschen im Fahrzeuginnenraum. In: Sachse, P.
and Ulich, E., eds. Beiträge zur Arbeitspsychologie. Lengerich: Pabst Science Publishers. Wagner, V. and Kallus, K.W. (2014). Use of a multidimensional approach in the automotive product de-velopment: Quality of turn indicator sounds. In: de Waard, D. et al. (Eds.), Proceedings of the Human Factors and Ergonomics Society Europe Chapter 2013 Annual Conference, pp. 169-181. Wagner, V. and Kallus, K.W. (2015). Sound quality of turn indicator sounds–use of a
multidimensional approach in the automotive product development. Journal of Traffic and Transportation Engineering, 3, pp.158–165. Watson, D., Clark, L.A. and Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of Personality and Social Psychology, 54(6), pp.1063–1070. Wessa, M., Kanske, P., Neumeister, P., Bode, K., Heissler, J. and Schönfelder, S. (2010).
EmoPics: Subjektive und psychophysiologische Evaluation neuen Bildmaterials für die
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 klinisch-biopsychologische Forschung. Zeitschrift für Klinische Psychologie und Psychotherapie, 39(Suppl. 1/11), p.77. Wundt, W. (1896). Grundriss der Psychologie. Leipzig: Verlag von Wilhelm Engelmann. Yagi, A. (2000). Engineering psychophysiology in Japan. In: Backs, R. W. and Boucsein,
W., eds. Engineering psychophysiology: Issues and applications. Mahwah, New Jersey: Lawrence Erlbaum Associates, pp.361–368. Yuan, S.N.V. and Ip, H.H.S. (2018). Using virtual reality to train emotional and social skills in children with autism spectrum disorder. London journal of primary care, 10(4), pp.110–112.
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-7923-8988-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional