Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Probabilistic Approach to Modeling Human Behavior Based on Pupillary Response Control

Yugra state University, Khanty-Mansiysk 628012, Russia ↗
Publication year
Digital Object Identifier10.54941/ahfe1001087

Abstract

Currently, researchers are actively working on methods for making decisions based on the results of automated analysis of the sequence of images received from video cameras. The purpose of our work is to develop a probabilistic approach to predicting human behavior based on modeling the pupillary response. The novelty of the research lies in the use of the parameters of the pupillogram obtained in the process of exposure of a person to a stimulus material as a prognostic tool. The likelihood was assessed by empirical frequencies (stimulus material puts a person in a tense state). In total, 40 people took part in the experiments to identify the stress state, including 20 males and 20 females. The average age of the participants was 20 years. All participants in the experiment were volunteers. During the experiment, we measured the change in galvanic skin response (GSR) and pupillary response. The GSR was measured using the "Aktivatiometer 6" hardware and software complex. Pupilary response was monitored using a pupillographic module for registering changes in pupil size. Pupillographic module consists of a special helmet and a digital video camera ZWO ASI120MC. The camcorder has the following main technical characteristics: frame rate - 30 fps, lens with optical zoom 1X-100X, resolution - not less than 600 TVL, signal-to-noise ratio not worse than SNR=40 dB, focusing range - from 20 mm to infinity. The captured video files were dissected and analyzed using the ImageJ software. Preliminary calibration is individual in nature, allows you to take into account the initial psychophysical state of a person. The assessment of the most probable S \ Smed value for calibration stimuli that does not induce a stress state falls within the interval (1± 0,2). A strong correlation was found between GSR and the duration of the increase in pupil size (p=0,7). On the basis of pupillary response modeling, a decision-making algorithm for security systems is proposed.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

INTRODUCTION

Currently, researchers are actively working on methods for making decisions based on the results of automated analysis of the sequence of images received from video cameras. Also, the development of tools is underway to optimize the interaction between humans and technology. The desire to teach to understand the machine, what a person feels or wants, sets the task of modeling a person's reaction, depending on the conditions in which he is. The successful solution of all these tasks indicates a global goal: predicting the intentions of people by artificial intelligence based on the results of video analytics. The fundamental possibility of such forecasting is provided by the subordination of human behavior to the unified laws of psychophysiology.

An unsolved problem at this stage is the development of a diagnostic tool that provides a quantitative prognostic assessment. Our efforts are aimed at solving this problem. The goal is to develop a probabilistic approach to predicting human behavior based on modeling the pupillary response. The novelty of the research lies in the use of a pupillogram obtained in the process of stimulating material on a person as a prognostic tool.

EXPERIMENTAL METHODS AND TECHNIQUES The possibility of using the pupillary reaction as a prognostic tool is provided by the effect of stimulus material on a person. As the latest research shows, emotions and cognitive processes play a large role in decision-making. When making a choice, a person formulates their cognitive judgments based on their affective experience. Strong emotions decrease or alter cognitive processes that are part of decision making. Therefore, when analyzing decision-making processes, it is necessary to take into account the complexity and importance of emotional processes. Pupillary reaction is one of the indicators of experienced emotions and tensions [1], and GSR is a reliable indicator of the stressed state and cognitive processes [2]. Let's find the parameter of the pupillogram, which can be used as a marker of the stress state.

In total, seven series of experiments were carried out with test objects for various purposes. In total, 40 people took part in the experiments to identify the stress state, including 20 males and 20 females. Representatives of different nationalities were present: Europeans - 33 people, peoples of Asia and the East - 7 people. The participants in the experiment had different visual acuity. The vision of some participants in the experiment was corrected using lenses or glasses, which does not affect the quality of the data obtained.

All experiments are carried out in a separate room equipped with the following equipment: a pupillographic module for registering changes in pupil size, a hard-ware- software complex "Activatiometer 6", three video cameras, two laptops, personal computer (PC), wall-mounted TV for demonstration of test objects. There are no sources of natural light in the room, a constant temperature and other climatic parameters are maintained. After the participants expressed their voluntary consent to participate in the experiment, they were,

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 in turn, invited to enter a special room for measurements. General view of the measuring complex in Figure 1. Figure 1. General view of the measuring complex

The research procedure is as follows. The subject sits down on a chair, puts on a helmet. Camcorder №2 focuses on the pupil. Camcorder specifications: Resolution: 1.2 Megapixel 1280 × 960; sensor: 1/3 "CMOS AR0130CS, pixel size: 3.75 microns; Frame rate: 30 fps at maximum spatial resolution; exposure range: 64 microns-1000s. Recording on both cameras and presentation of presentations from the wall-mounted TV screen is started. on the TV screen by pressing and holding the plates of the "Activatiometer6" device during the entire process of video filming, fits into the interval (1 ± 0.2) (Figure 2). Outliers are caused by blinking and are not taken into account in the analysis. The developed calibration is individual in nature, allows you to take into account the initial psychophysical state of a person [3].

Figure 2. Statistics of the values of the change in the size of the pupils during calibration As you know, information that is significant and important for a person causes
Original PDF Figure 2.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 involuntary attention if an individual appears in the field of perception. If we assume that the test object contains such information, then the reaction of the pupils to such a stimulus is proportional to the intensity of the stress experienced in this case. In accordance with the set goal, images were used as test objects to identify the stress state: calibration slides, slides with images of special symbols, formulas and a target stimulus-photograph of a person, information about which some students hide. The concealment of information by some students was ensured by preliminary staging actions. The young man entered the classroom during classes, when the teacher was absent for a minute (by agreement), manipulated objects on the teacher's desk, and staged a correction of the grade in the journal. A photograph of this guy was used as test objects. Hypothesis: pupillograms of people hiding information are different from pupillograms of people not trying to hide it. The sign by which we determine the stressed state: if the situational indicators of tension measured by GSR increased, then the subject is in a stressed state, therefore, he recognized the person in the photo. People from the control group (5 people) have no idea about the incident and do not know this student. The probability of a stress state in the control group on stimulus material is practically zero.

MAIN RESULTS In order to develop a methodology for assessing the connection between the reaction of the pupil to a test object and the psychophysical state of a person, it is necessary to link together the results of the obtained experiments and compare them with the existing model of emotion (Figure 3a), in which the main mechanism of the emergence of emotions is cognitive processes (thoughts, representation). To do this, consider the curve of the pupillogram with an emotional response (Figure 3b). T. Tomaszewski [4] using the example of the emotion of anger identified 4 phases of the flow of emotion. Let's mark on the pupillogram the areas corresponding to the phases of emotion. The work of the cognitive zone is directly related to mental activity, begins and proceeds earlier than emotion. Then a segment of the curve with a continuous increase can be considered a phase of information accumulation and an assessment of its significance for a person. If so, then this is a transitional process from one state to another, which should be reflected in the galvanic skin reaction. Galvanic skin response was measured in parallel with the pupillary response. In the experiment, the students who concealed from the teacher the fact of an attempt to make corrections in the journal were in a tense state. Confirmation is the excess of the situational indicator over the individual typological in excess of the norm.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Figure 3. a) Four phases of the flow of the emotion of anger (Tomashevsky [4]); b) Pupil-logram - curve of intensity of emotional experience Thus, as a result of measurements, we have a set of pupillograms and corresponding GSR measurements. Gaussian mixture models are probabilistic models and use a soft clustering approach to distribute points in different clusters. Pupillogram cluster analysis allows you to represent data in the form of several Gaussian distributions. Each of these distributions is a cluster. A mixture of k Gaussian distributions (where k is equivalent to the number of clusters), each of which has a certain mean vector and variance matrix. A Gaussian mixture model groups data points that belong to the same distribution together. For each S/Smed of a pupillogram, the probability that it belongs to a cluster is calculated. Thus, as a result of clustering, we obtain the most probable (p) values of the centers of the clusters S/Smed (µ). Comparison of the obtained sets of µ1, µ2 centroids of clusters using the Friedman ANOVA test (Table 1) showed their difference.
Original PDF Figure 3.

Friedman-Anova test showed that at the 0.03 level, the populations ate significantly different. Dunn's Test and Wilcoxon-Nemenyi-McDonald-Thompson Test -At the 0.03 level, the populations are significantly different. Dunn's Test и Wilcoxon-Nemenyi-McDonald- Thompson Test -At the 0.03 level, the populations are significantly different.

Then it is possible to introduce a level classification of the intensity of the psychological state caused by the stimulus material [5]. In addition, in pupillograms, we determine the duration (△t) of a continuous increase or decrease S/Smed in the size of the pupils. Long-term trends of decreasing or increasing S/Smed indicate a stressful state or an increase in attention to stimulus material, respectively. The number of levels is equal to the number of clusters. The results of the correlation analysis of GSR, △t and cluster centers S/Smed (µ) are presented in Table 1.

Table 1: Correlation Coefficients (Pairwise Elimination) N R Δt1 μ1 GSR Δt2 μ2
Original PDF Table 1:

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Correlation analysis showed a strong dependence (p=0.7) between GSR and the duration of the increase in pupil size, which indicates a tense psychological state of the respondents. Moreover, if S/Smed exceeds the most probable individual calibration value, an emotional response to the test object is added to the tense state. Emotional response indicates the significance of the test object for the respondent. Then, based on the result of the cluster analysis of the pupillary reaction and the information contained in the test object, it is possible to predict the likelihood of certain actions by the respondent. Thus, it is possible to implement a probabilistic approach to modeling human behavior using pupillary response as a marker.

MODELING HUMAN BEHAVIOR BASED ON CONTROL OF PUPILLARY RESPONSE The logical scheme "Stimulus-Organism-Response" was developed in behavioral psychology to model the multivariate behavior of living systems. Within the framework of this model, the behavior of the system occurs as a result of decision-making by the system, due to the relationship between the internal state and the state of the environment [6]. This model is applicable to the description of any decision-making system. A fixed set of N mutually exclusive options for possible actions A={a1, a2, ..., aN} is compared with orthogonal basis vectors |𝑎𝑖⟩ of an N - dimensional space that includes all possible behavioral alternatives. Then the decision-making algorithm by a human can be represented in this space with the 𝑝̂ operator: |selected action⟩ =p|external behavioral suggestion⟩.

In the random walk model, the decision-making problem occurs on the basis of limited, uncertain or contradictory information, in which the basic states of the cognitive system can be different degrees of confidence in making a target decision. In the model [7], the basic states are ordered according to the degree of confidence of the solution, so that the change in the degree of confidence corresponds to the "wandering" of the system from one state to another. The time interval of the step, the algorithm for assessing the level of confidence and the condition for making a decision at each step are the parameters of the model. In conditions of uncertainty of the consequences of the decisions made, the decision is made ("wanders") based on their own preferences, taking into account external conditions.

When making a decision under the conditions of our experiment (whether to inform the teacher about the incident or not), the evolution of the cognitive state |Ψ⟩ can be described using the Schrödinger equation

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

𝑖𝑑

𝑑𝑡× |Ψ⟩= 𝐻̂ × |Ψ⟩ (1) The general solution of this equation |Ψ(t)⟩= 𝑒𝑥𝑝[−𝑖× 𝐻̂ × 𝑡] × |Ψ(0)⟩ contains the evolution operator. Dissipative terms describe the interaction of the cognitive state with the information environment, and the dependence on time allows one to model the dynamics of decision-making.

In the described experiment, the subjects did not know whether by chance (A) or not (N) the photograph of the “intruder” was present among the stimulus material. Therefore, at the time of viewing, they were in conditions of two-variant uncertainty of information. The model of "random walk" of the cognitive state of the respondent choosing one of three alternatives of behavior {E, R, K} with two-variant uncertainty of the context {A, N} can be represented in the form of a decision tree (Figure 4).

Figure 4. Comparison of the "random walk" model and clustering results The probabilities of possible options for behavior ((E) - tell everything, (R) - recognized the person, (K) - keep silent) are determined by the relationship be-tween the subject's idea of a fixed state of external conditions and their internal beliefs.
Original PDF Figure 4.
Table 2: Utility table
Original PDF Table 2:

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 possibilities is characterized by a real value. The initial state is described by the Gaussian function. In accordance with the Schrödinger equation, the impact of the stimulus material generates the evolution of the joint state of the cognitive system and the decisions made. Our senses register about 10 Mbps of information per second. Consciousness is able to process only 40 bits per second. Therefore, the brain processes part of the information unconsciously. If there is enough time, then the state of the pointer is split into N spatial beams.

Gaussian Mixture Models cluster analysis allows a set of measured S/Smed to be represented as a sum of probability density functions with weight coefficients, the sum of which is 1. Comparing a decision tree based on a random walk model with the results of clustering measurements S/Smed, one can estimate the probability of making a particular decision the respondent. The pointer value - the decision made - is now correlated with the cognitive state of the subject. Therefore, it is possible to predict the likelihood of human behavior by security systems. The decision-making algorithm is shown in Figure 5.

Figure 5. Algorithm for making decisions by security systems Thus, the decision-making algorithm is based on recording a person's pupillary response to a visual stimulus and takes into account his individual preferences.
Original PDF Figure 5.

CONCLUSIONS

The possibility of using the parameters of the pupillogram obtained in the process of exposing a person to a stimulus material was investigated. As a result of the studies carried out, it was established: - Cluster analysis of pupillograms makes it possible to introduce a classification of pupillary response according to levels corresponding to different psycho-logical states. The results are consistent with the idea of dividing the pointer into N spatial beams. - A strong correlation was found between GSR and the duration of the increase in pupil size S/Smed (p=0.7), which indicates a stressed psychological state of the respondents or an increase in their attention to stimulus material.

On the basis of pupillary response modeling, a decision-making algorithm for security systems is proposed. The results obtained can be useful for working on methods for making

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 decisions based on the results of an automated analysis of the sequence of images received from video cameras.

REFERENCES

Romanova N.M. et al. (2008). Features of human oculomotor reactions when pronouncing true and false information. Izvestiya Saratov University. New series. Philosophy Series. Psy-chology. Pedagogy. 8(1), pp. 65-73. Montagu J. D., Coles E. M. (1966). Mechanism and measurement of the galvanic skin

response. Psychological Bulletin, 65(5), pp. 261. Oksana I., Marina B. (2021). Diagnostics of the Stress State by the Method of Pupillography.

In: Goonetilleke R.S., Xiong S., Kalkis H., Roja Z., Karwowski W., Murata A. (eds) Ad-vances in Physical, Social & Occupational Ergonomics. AHFE 2021. Lecture Notes in Networks and Systems, vol 273. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-80713-9_39. Dudal N.N. (2017). Psychology. Emotional-volitional sphere. Boronenko M., Isaeva O., Boronenko Y., Zelensky V., Gulyaev P. (2021). Recognition of Changes in the Psychoemotional State of a Person by the Video Image of the Pupils. In: Wojtkiewicz K., Treur J., Pimenidis E., Maleszka M. (eds) Advances in Computational Collective Intelligence. ICCCI 2021. Communications in Computer and Information Sci-ence, vol. 1463. Springer, Cham. https://doi.org/10.1007/978-3-030- 88113-9_10. Surov, I. A., and A. P. Alodzhants. (2018). Decision-making models in quantum cognitive

sci-ence. Saint Petersburg: ITMO University. Busemeyer J.R., Wang Z., Townsend J.T. (2006). Quantum dynamics of human decisionmaking. J. Math. Psychol. 50(3), pp. 220-241. Khrennikov A., Basieva I. (2014). Quantum Model for Psychological Measurements: From

the Projection Postulate to Interference of Mental Observables Represented as Posi-tive Operator Valued Measures. NeuroQuantology. 12 (3), pp. 324–336.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Cover of the proceedings volume Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems

Explore the complete digital volume

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-7923-8988-7

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →