Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Development of a non-contact method for the examination of emotional stability

Yugra state University, Khanty-Mansiysk 628012, Russia ↗
Publication year
Digital Object Identifier10.54941/ahfe100928

Abstract

Recently, the issue of extracting information about the emotions of people, including people who are under stress, with the help of video analytics, has become relevant. It is known that the change in GSR can reliably judge the presence of depression or a stress state [1], in which a person can pose a threat to himself or society. Since with an increase in skin conductivity as a result of stress, the sympathetic nervous system is involved, one of the signs indicating changes in the emotional state is mydriasis. The purpose of our work: the development of a non-contact method for the examination of emotional stability. To study the emotional stability of a person under the influence of external stress factors, a pupillographic module for registering changes in pupil size without infrared illumination was used; hardware-software complex “Ats-6”. During the experiment, the stimulus material was shown from the monitor screen. The stimulus material contained graphic information - images of the main stress factor of office workers - tedious routine paper work. Images that evoke pity were used as an additional stress factor. The experiments involved 60 volunteers (including the control group). Of these, 28 are female (46.7%), 32 are male (53.3%). The GSR and pupillograms of two groups of people were compared - students during the session and active office employees of the Ministry of Emergency Situations. An algorithm for processing and analyzing the received video files is presented. The SNR(dB) value for different types of eyes before and after processing the received video files changed as follows: for light eyes, the value increased from 5.4 dB to 13.9 dB; for brown eyes - from 3.4 dB to 9.4 dB; for "transparent" eyes - from 1.9 dB to 2.5 dB. Despite the small SNR, ImageJ allows for high-precision pupil contouring. Clustering of the pupillary response data of the participants made it possible to determine the most probable values ​​of µ, µ2 centroids. Checking the equality of the means µ, µ2 by the Friedman ANOVA method showed that at the level of 0.05 these two distributions differ significantly. Through correlation analysis, it was found that there is a strong relationship between GSR and the duration of pupil size increase in the control group (p=0.87). At the same time, for the group of students in which the pupillary reaction is stronger, the relationship with GSR is less (p=0.72). This can be explained by the prolonged stressful state of the participants in the control group (office workers). The results suggest that the participants in the control group have cognitive processes that allow them to control their emotional state. The results obtained can be useful for non-contact detection of a person's stress state through video analytics.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

INTRODUCTION

Emotions are one of the main mechanisms for assessing the psychological characteristics of human behavior (Bekhtereva 2010, Harlé et al. 2013, Karpov 2014). The system of emotional regulation of behavior, the reaction of which is reflected in psychophysiological states, can be represented as a multicomponent and multilevel system, including the mental level, physiological and behavioral levels (Dmitriev 2018). New paradigms have given rise to research on the role of emotions and cognitive processes in decision making. JR Gray, TS Braver, ME Raichle found out that emotions and higher cognition can be truly integrated, i.e. both emotions and cognition jointly and equally affect thinking and behavior (Gray et al. 2002). Strong emotions reduce or alter the cognitive processes that are part of decision making. Therefore, in the light of modern scientific knowledge, when analyzing decision- making processes, it is necessary to take into account the complexity and importance of emotional processes (Sołtys et al. 2013). It was established (Yagoda et al. 2011) that a significant decrease in the level of skin electrical conductivity, reflecting the processes of psychophysiological inhibition and a decrease in the activity of the sympathetic nervous system, is characteristic of an acute reaction to stress (35-40 units), panic disorder (25-33 units), organic disorder personality (17-22 conventional units in head leads), as well as productive forms of schizophrenia (by 4 - 33 conventional units) and depression (5-13 conventional units). An increase in skin electrical conductivity in the head leads (by 9-22 units), indicating a pronounced sympathetic hyperactivity and a high level of psycho-emotional arousal, is recorded in post-traumatic stress disorder. Since with an increase in skin conductivity as a result of stress, the sympathetic nervous system is involved), one of the signs indicating changes in the emotional state is mydriasis. In our age, emotional stability is of great importance. For example, low emotional stability of an employee in the performance of professional duties can sometimes lead to irreparable consequences (Kowalczuk et al. 2020). Our goal: to develop an unobtrusive method of non-contact examination of emotional stability.

It is known that the physical and psychological health of employees of the Ministry of Emergency Situations is strictly controlled, so they made up the control group - emotionally stable.

METHODOLOGY AND TECHNIQUE OF THE EXPERIMENT Experimental setup

The experimental setup is presented in the form of a pupillographic module, which is a symbiosis of a special helmet and a ZWO ASI120MC digital video camera (Boronenko et al. 2021). The helmet is based on a boxing helmet and is an aluminum

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 tube frame. The helmet must be worn so that it lifts the skin on the forehead, which will eliminate the overhang of the eyelid on the pupil. At the same time, it is important to create conditions under which there will be no shadow from the helmet, reflections of various objects and glare on the pupil. This can be achieved by selecting video parameters. This will further facilitate the processing of the received data. With the help of the described module, the process of obtaining pupillograms is carried out.

Measurement of changes in the psycho-emotional state of the subjects by measuring the galvanic skin response (GSR) was carried out using Ats-6. In this case, it is necessary to create the most identical measurement conditions for all subjects. The general picture of the experiment is shown in Fig. 1. The subject needs to put his hands on the metal plates and press them. A mirror was attached to the viewing area of the video camera. The displayed test object and the readings of the device were synchronously reflected in the mirror to link GSR changes to the stimulus material.

Figure 1. General view of the measuring complex As a stimulus material, we use test objects that evoke any emotions. The selection of test objects was carried out using an anonymous survey in a Google form. During the experiment, a presentation was shown on the screen, which included calibration slides and slides with test objects.
Original PDF Figure 1.

Dissection and analysis of video files The input data are video files (sequence of images of pupils), the typical size of which is 1024×768. The process of obtaining video files is quite simple and takes no more than 10 minutes.

The selection of the boundaries of the pupil (outlining) may have an error due to the following reasons: poor lighting; eyelashes or eyelids covering the pupil; the pupil

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 and iris are poorly distinguishable (due to the dark color of the iris or because of the “transparent” pupil); glare on the pupil.

Start processing any video files in ImageJ (Isaeva et al. 2020). First you need to cut the files, namely, select the areas containing the pupil. Because the files are too large and the computer will not be able to process them entirely. Then adjust the brightness and contrast of the active image. The next stage of the processing algorithm will vary depending on the color of the iris and the illumination. For bright eyes with a well-defined pupil outline, you can split the RGB image obtained by subtractive blending into separate channels corresponding to three specific colors. For brown eyes with a well-defined pupil contour and for eyes without a visible pupil contour, before applying the above step, you need to remove the smooth continuous background from the images. The next stages of the processing algorithm are the same for all cases. We filter the image given its integral image, then we smooth the image by replacing each pixel with the average value of neighboring pixels. Next, we replace the pixel with the median of pixels in the environment if it deviates from the median by more than a certain value (thresh-old). After that, we adjust the upper and lower threshold values for segmenting the region of interest and the background of the image.

Then we take measurements. In order to visualize the data, it is necessary to select the parameters to be measured: Area, Center of mass, Stack position. It is also necessary to check the accuracy of the contouring. If the accuracy is insufficient, then it is necessary to carry out the contouring process again. Processing algorithms for different cases are presented separately in Fig. 2 a, b, c.

Figure 2. a) Processing algorithm for light eyes with a well-defined pupil contour; b) Processing algorithm for brown eyes with a well-defined pupil contour; c) Processing algorithm for eyes without visible pupil outline
Original PDF Figure 2.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

The cases described above were chosen as examples, since the selection of the boundaries of the pupil in these cases is a difficult task. Let us find the signal-to-noise ratio for the described cases (Table 1), where the signal is a light tone, the noise is a dark tone (pupil), according to the formula (1): ) (1)

Mathematical equation reproduced from the original PDF
Equation reproduced from the original PDF.
Table 1: Signal-to-noise ratio before and after processing Signal-to-noise ratio for the original video Signal-to-noise ratio for video file after file changing brightness/contrast For bright eyes
Original PDF Table 1:

METHODOLOGY FOR THE ANALYSIS OF EMOTIONAL STABILITY Emotional stability is a personality trait that manifests itself in varying degrees of sensitivity to emotional stimuli (Sosnovsky 2008). The study of the qualitative diversity of emotional stability, the psychological understanding of its essence as a mental phenomenon consists in finding an answer to the question: how emotions are given to the subject in his experiences, how they are perceived and realized and by what.

It has been established that the most important characteristic of the manifestation of stress tension are GSR fluctuations with an amplitude of 4 c.u. and higher, a significant correlation was shown between the number and the sum of the amplitudes of these oscillations (Yagoda et al. 2011).

From the point of view of the method of influences leading to a change in the size of the pupils, the existing factors can be conditionally divided into informational (direct impact of information on the brain) and non-informational (indirect impact on the brain). The selection of test objects was based on the following theory. For each person, there are topics that cause ultra-high emotional responses. The information that concerns them is significant and important for a person, and causes involuntary attention if it appears in the field of perception of an in-dividual. If we assume that such information may contain a test object, the reaction of the pupils to such a test

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 object is proportional to the intensity of the volt-age experienced in this case. The fulfillment of the condition Ψ< Ψactivation corresponds to afferent synthesis that does not lead to active actions. In a person for whom information is of no value, the emotional component of the mental state does not change, the size of the pupils is normal, fluctuations in the area of the pupil correspond to the norm. An excessively strong emotional reaction to such images may indicate that a person is subject to stress, is in an unstable emotional state.

When measuring situational tension, a total of 60 volunteers took part (including the control group). Of these, 28 are female (46.7%), 32 are male (53.3%). By age, the subjects were distributed as follows: 18 years old - 9 people (15%), 19 years old - 11 people (18.3%), 20 years old - 13 people (21.7%), 21 years old - 9 people (15%), 22 years old - 8 people (13.3%), 23 years old - 1 person (1.7%), over 23 years old - 9 people (15%). At the same time, subjects with Slavic (90%) and Asian appearance (10%) were present. During the experiment, all test objects (stimulus material) shown from the monitor screen contained images of the main stress factor for office workers - tedious routine paper work. As an additional stress factor, images that evoke pity were used: a wet and frightened kitten clinging to a person’s leg; a child with multiple bruises in a hospital bed.

Obtaining information about the emotional state from synchronized video files of pupillary reaction and GSR occurred according to the following algorithm:

1. Obtaining a pupillary reaction with the help of a pupillographic module and a galvanic skin reaction with the help of “Ats-6”.

2. Image preparation, processing and contouring of pupils using the ImageJ program based on the algorithm described above.

3. Classification of pupillograms based on cluster analysis. The composition of the mixture includes the following parameters:

- The average value of μ, which determines its center; - Covariance Σ, which determines its width; - The mixing probability π, which determines how large or small the Gaussian function will be.

The result is a data set: µ, p, Δt. For correlation analysis of the parameters of pupillograms and GSR, as well as testing hypotheses, we eliminate the problem of different amounts of data (interpolation of 1000 points).

METHODOLOGY FOR THE ANALYSIS OF EMOTIONAL STABILITY This section compares the GSR and pupillograms of two groups of human students during the session and active employees of the Ministry of Emergency Situations. Both groups of people are under stressful conditions. Consistent demonstration of stress factors leads to the accumulation of emotional stress. The data in Fig. 3 is grouped as follows. C1, C2 - pupillary reaction to TO1 in the group of students during the session, Ch1, Ch2 - pupillary reaction to TO2 in the group of employees of the

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Ministry of Emergency Situations (control group). All data С1, С2, Сh1, Ch2 are grouped into three classes by the Gaussian Mixture method.

Figure 3. Grouped Box Charts: C1, C2 - pupillary reaction to TO1 in the group of students during the session, Ch1, Ch2 - pupillary reaction to TO2 in the group of employees of the Ministry of Emergency Situations (control group) Comparison of µ, µ2 cluster centroids by the Friedman ANOVA test showed their difference - at the level of 0.05, these two distributions differ significantly (Table 2). The results suggest that the participants in the first group (students) are less emotionally stable than those in the control group. Subjects from the control group can control themselves, so their indicators of stress increased slightly.
Original PDF Figure 3.
Table 2: Friedman ANOVA
Original PDF Table 2:

Descriptive Statistics N Min Q1 Median Q3 Max "μ" 31 0.69 0.97 1.03 1.12 1.32 "μ2" 60 0.69 0.89 0.985 1.0775 1.62 Ranks N Mean Rank Sum Rank "μ" 31 53.67742 1664 "μ2" 60 42.03333 2522 Test Statistics U Z Exact Prob>|U| Asymp. Prob>|U| 1168 1.98976 0.04598 0.04662 Null Hypothesis: Median1 = Median2 Alternative Hypothesis: Median1 <> Median2 At the 0.05 level, the two distributions are significantly different.

Next, a correlation analysis of GSR, △t and µ (cluster centers S/Smed) was carried

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 out, the results are presented in Fig.4.

Figure 4. Correlation matrix GSR, △t and µ a) group of students b) control group Correlation analysis showed a strong relationship (p=0.87) GSR and the duration of the increase in pupil size in the control group. At the same time, for the group of students in which the pupillary reaction is stronger, the connection with GSR is less. This can be explained not only by the prolonged stressful state of the participants in the control group, but also by the participation of cognitive processes that allow controlling the emotional state.
Original PDF Figure 4.

CONCLUSIONS

Experiments were carried out to study the emotional stability of a person under the influence of external stress factors using pupillometry and GSR. The GSR and pupillograms of two groups of people were compared - students during the session and active employees of the Ministry of Emergency Situations.

- An algorithm for processing and analyzing the received video files is presented. The SNR(dB) value for different types of eyes before and after processing the received video files changed as follows: for light eyes, the value increased from 5.4 dB to 13.9 dB; for brown eyes - from 3.4 dB to 9.4 dB; for "transparent" eyes - from 1.9 dB to 2.5 dB. Despite the small SNR, ImageJ allows for high-precision pupil contouring.

- Clustering of the pupillary response data of the participants made it possible to determine the most probable values of µ, µ2 centroids. Checking the equality of the means µ, µ2 by the Friedman ANOVA method showed that at the level of 0.05 these two distributions differ significantly.

- Through correlation analysis, it was found that there is a strong relationship between the GSR and the duration of the increase in pupil size in the control group (p = 0.87). At the same time, for the group of students in which the pupillary reaction is stronger, the relationship with GSR is less (p=0.72). This can be explained by the prolonged stressful state of the participants in the control group (office workers). The results suggest that the participants in the control group have cognitive processes that allow them to control their emotional state.

The results obtained can be useful for non-contact detection of a person's stress

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 state through video analytics.

REFERENCES

Bekhtereva N.P. (2010), “The magic of the brain and the labyrinths of life”. Moscow: AST; St. Petersburg: Owl. Boronenko M., Isaeva O., Boronenko Y., Zelensky V., Gulyaev P. (2021),

“Recognition of Changes in the Psychoemotional State of a Person by the Video Image of the Pupils.” In: Wojtkiewicz K., Treur J., Pimenidis E., Maleszka M. (eds) Advances in Computational Collective Intelligence. ICCCI 2021. Communications in Computer and Information Science, 1463, 123–133. https://doi.org/10.1007/978-3-030-88113-9_10 Dmitriev A.A. (2018), “Emotional response: phenomenology, structural and component composi-tion of the problem and research prospects.” Siberian Pedagogical Journal, 3, 90–98. Isaeva O., Boronenko M. (2020), “Application of ImageJ program for the Analysis

of Pupil Reaction in Security Systems.” Journal of Physics: Conference Series, 1519, 1–8. http://doi:10.1088/1742-6596/1519/1/012022 Gray J. R., Braver T. S., Raichle M. E. (2002), “Integration of emotion and cognition in the lateral prefrontal cortex.” Proceedings of the National Academy of Sciences, 99(6), 4115–4120. Harlé K. M., Shenoy P., Paulus M. P. (2013), “The influence of emotions on cognitive

control: feelings and beliefs—where do they meet?” Frontiers in human neuroscience, 7, 508. Karpov V. E. (2014), “Emotions of robots.” Proceedings of the XII National Conference on Artifi-cial Intelligence with International Participation KII- 2010, 20–24. Kowalczuk Z., Czubenko M., Merta T. (2020), “Interpretation and modeling of

emotions in the management of autonomous robots using a control paradigm based on a scheduling vari-able.” Engineering Applications of Artificial Intelligence, 91, 103562. https://doi.org/10.1016/j.engappai.2020.103562 Sołtys A. et al. (2013), “Emotions in Decision Making.” Neuroeconomic and Behavioral Aspects of Decision Making, 3–47. Sosnovsky B.A. (2008), “Sustainability - emotional non-susceptibility of emotional

states and processes to the destructive influences of internal and external conditions.” Psychology. Yagoda S.A. et al. (2011), “Diagnosis of mental illness based on the results of electrophysiological examination.” Engineering Bulletin of the Don, 18(4), 10– 12.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-7923-8989-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →