Salman Md Khan, Swati Pal (2025). Change in Affective Dynamics and Cognitive State with Time during Multiplayer FPS Games. In: Human Factors and Wearable Technologies. AHFE Open Access, vol. 175, pp. 98–106. https://doi.org/10.54941/ahfe1006363
In the last decade, digital gaming has flourished due to the advancement of computing devices, mobiles and AR/VR. These games are immersive, interactive, cooperative and multiplayer. Different games affect gamer’s cognition, emotions and performance differently. The subjective evaluation such as questionnaires, surveys and standard scales to inspect the gaming experience is based on the retrospective recollection of the gaming experience after the task is complete. Sometimes, the responses from the gamers could be erroneous as it is hard to perceive the gaming experience. The alternative to subjective evaluation is objective evaluation where the real-time collection of the gamer’s behaviour is recorded. This is done by acquiring the heart rate, electrodermal signals or facial expressions of the gamers. Such signals change when subjected to visual stimulations from the immersive and interactive gaming environment. This study investigates emotional and cognitive properties using a facial recognition system and Galvanic Skin Resistance (GSR) while playing mobile-based massively multiplayer FPS game. The game, “Call of Duty: Mobile” was played on iPhone 14 for 15 minutes by each participant. 10 young healthy gamers belonging to the age group of 18-30 years were selected for this study. The GSR activity was analysed based on 4 features; the tonic component, the phasic component, the number of Skin Conductance Response (SCR) peaks and the amplitude of peaks detected during the study. The facial expression analysis (FEA) included emotional and facial parameters. The change in the amplitude of GSR signals with time was observed among all participants. The tonic component and the amplitude of the peaks increased with time. The mean value of the tonic signal increased from 1.782 ± 0.946 to 3.739 ± 1.702. The number of GSR peaks during the task was more than 100 for different participants. However, the number of peaks was constant with time. The peak amplitude of the SCR peak ranged from 0.05-0.5 µSiemens among participants when 100 or more GSR peaks. This indicates that the tonic amplitude and amplitude of the SCR peak can be a better indicator of the cognitive load with time than the number of SCR peaks.The FEA indicated that the change in facial expression was observed only after 8-9 minutes of gaming time. The facial expressions were understood in terms of eye closing and eye-widening, brows raising and attention. Based on the expression, it was found that all participants started gaming with positive valence, higher attention and engagement, indicating that relatively higher critical thinking and focus are required for gaming activity at the start. After 10 minutes of game time, valence, engagement and attention levels decreased. A small increment in negative emotions such as fear, contempt and anger were observed. The correlation analysis between the amplitude of GSR peaks and different emotions strongly correlated with valence, joy and sentimentality.
Video games are a mitigation of the different challenges. Challenges define the difficulty level of the games. The challenges may vary depending on the players experience and exposure to the game. The challenges perceived by a gamer changes player experience leading to change in emotions, cognition and physicality (Denisova et al., 2020). The player experience changes dynamically with time and may vary among gamers. Digital games usually affect emotionally and cognitively. Also, continuous and long term exposure of the digital games lead to change in cognitive load and stress level of an individual (Zhang et al., 2018).
In general; the player experience is recorded by enquiring from the gamers about the game. This can be done using a questionnaire or standard performance evaluation scale. For instance, there are tools such
Change in Affective Dynamics and Cognitive State With Time 99
as NASA-TLX, Challenge Originating from Recent Gameplay Interaction Scale (CORGIS) and Visual Analogue Scale to determine cognitive load perceived by the gamer (Denisova et al., 2020; Peng et al., 2023; Pretty et al., 2024). However, these methods are administered after the gameplay and cannot estimate the dynamic change in the cognition and emotions during the gaming. Also, the responses filled by the gamers can be erroneous and may not indicate accurate indications. To measure the real time change in the cognition of the gamers, the gamer’s behaviour can be measured by two methods: physiological and non-physiological methods. The physiological methods include the sensing of the physiological signals from the human body such as Heart Rate (HR), Skin Conductance/Electrodermal Activity (EDA), Electromyography and Electroencephalography (EEG) (Luo et al., 2024; Park et al., 2024; Vatsal et al., 2024). With the advancement of wearable devices, using HR and EDA measuring devices is easier (Chin et al., 2024; Vatsal et al., 2024). A player’s physiological state can be used to predict the approach adopted to different gameplay situation. Also, the physiological signals can also determine the cognitive state of the player due to prolonged gaming, avoiding frustration. Non-physiological techniques largely include recording affective states during the task to measure their emotional states. These methods are usually used to measure changing emotional states and predict the cognitive state of the gamers. In such cases, the user does not need to wear devices for sensing purposes.
The objective of this study is to investigate the changes in the cognition and emotions while playing fast paced multiplayer first player shooting games. The cognition and emotions are measured using physiological and non-physiological techniques. The EDA signals are acquired to measure the physiological signals and non-physiological technique includes facial expression analysis using camera. The acquired dataset is used to investigate the effect of the challenges experienced by the gamers to achieve the target. Apart from it, subjective responses using questionnaire and executive function test are performed. Questions from CORGIS and NASA-TLX scale were mitigated to design the questionnaire.
RELATED WORK
The objectives designed in the gaming scenario changes the methodology of the challenges incorporated in the games. Different challenges change the player experience in order to overcome the obstacles. The response to the obstacle results in the change of the cognitive and emotional behaviour. Cognitive behaviour is based on an individual’s ability to observe, memorize, plan, reason and solve the problem. The emotional behaviour is dependent on the targets to be achieved, experience of the gamer and cognitive effort applied by the gamer (Peng et al., 2020, 2023).
In general, gamer’s performance in multiplayer games resides on the way the players interact with fellow gamers in the virtual world. Multiple factors influencing the gamers cognition and decision making include factors such as the amount of social interaction, competition and cooperation, coordination of social interaction between players (Baltzar, Hassan and Turunen, 2023).
100 Khan and Pal
Conventionally, the challenges in the games such as First Person Shooting Games were changed according to the “game difficulty”. The difficulty in the multiplayer games is also dependent on the nature of the opposition team such as their expertise, age group and interest level (Caroux et al., 2015).
The effect of the games and associated challenges can be studied evaluating the gamer/ player experience using validated questionnaire (Peng et al., 2023). Some of the validated questionnaire include NASA-TLX, Visual Game Design Scale, Basic Needs in Games Scale (BANGS) and CORGIS (Denisova et al., 2021; Ballou et al., 2024). In general, NASA-TLX form is designed to investigate the physical and cognitive workload distribution of the task based on 7 question scale (Zagermann, Pfeil and Reiterer, 2016). BANGS was designed to evaluate the positive and negative psychological experiences from different video games (Ballou et al., 2024). Multiple recent studies have adopted CORGIS as the medium to investigate the challenges in the digital games (Moschovitis and Denisova, 2023; Peng et al., 2023). Although, these scales are well accepted for retrospective recollection of the gaming experience after the task is complete, a continuous assessment of the gaming experience is not possible. Physiological signals find their application to evaluate gaming experience using emotional arousal, cognitive load and affective states (Greco et al., 2017; Cloude et al., 2022). Apart from it, facial expression recognition can also show changing affective states during a gaming timeline. Mapping them allows identification of changing moods of the gamers such as boredom, anxiety, frustration in real time, assisting gamers to improve gaming experience (Peng et al., 2023).
METHODOLOGY
In this study; realtime data collection of electrodermal signals and facial expression was performed from the experiment to explore the player experience during different gameplay challenges. A questionnaire was filled out by the participants before and after the gameplay to compare the changes perceived by the gamers.
Participants The total number of participants selected for this study was 10 (9 Males and 1 Females). The participants belonged to the age group of 18–30 years. The participants selected for the study included casual players (who usually enjoy playing such games) and hardcore players (who enjoy such games and participated/participate in gaming competitions). Lastly, participants were asked to avoid the gaming task if they had previous neurological or psychological issues.
Data Collection The questionnaire was designed to compare the change in the emotional and cognitive load perceived by the gamers. The CORGIS and NASA-TLX forms were used as the basis to compare their loading conditions. Apart from it, we asked a few demographic questions such as their screen time for the games, frequency of the games and gaming duration. The EDA signals were acquired
Change in Affective Dynamics and Cognitive State With Time 101
using Shimmer GSR device to detect the changing skin conductance level during different gaming scenarios. The EDA signal recording was performed at a sampling rate of 128 Hz. The AgCl unipolar electrodes were placed on the fingers and ground electrode was placed on the earlobe. The facial expressions were measured using high definition Camera (C922 PRO HD WEBCAM). The facial recording is used to compute basic emotions such as anger, contempt, disgust, fear, joy and sadness using Affectiva algorithms. These algorithms computes the likelihood of the emotions based on changing facial landmarks such as eyes, eye brows, lips and nose.
Experimental Protocols The experiment was performed in a room with a temperature maintained at 25 Celsius. Apart from it, the illumination of the screen and the room were kept constant for all participants. All participants were introduced with the purpose of the study before the start of the study. Upon agreeing to perform the task, the participants were asked to sign a written consent form. The first step of the task was the filling of the questionnaire, asking the participants their emotional and physical state. Followed by it, the device was strapped on the wrist of the participant to measure the EDA signals. The facial expressions were recorded using an HD Camera placed in front of the participant, integrated with iMotions. The EDA signals and facial expression data were synchronized using iMotions. A 5 minutes of resting time was provided to all participants before the start of the task. Upon the start of the task, the participant played the game for 15 minutes. The game was played on the IPhone 14 mobile. Upon the completion of the task, the questionnaire was filled by the participants. After the experiment, each participant was given a certain amount of money as an appreciation for the time provided.
Gaming Scenario The game selected in this study was Call of Duty: Mobile, an Online Battle Arena game. This game is popular among youth and provides rich content to understand in-game challenges. The participants were asked to play Multiplayer Online Arena Games. It provides different gaming scenarios based on objectives and gaming maps. In these games, the participant would play in a team of 5 against a team of 5. All other gamers were unknown to the gamers. Two gaming scenarios named Frontline and Deathmatch were selected because of similarity in objective. The objective was to achieve a particular score by killing the opposition member. The target for the Frontline and Deathmatch were 50 and 40 respectively. In this game, the participant can join the game after assumed as dead in the game until the target points are achieved. The participants had the freedom to select the shooting equipment based on their choices.
Analysis The data recorded from the questionnaire were compared the effect of gaming perceived by the gamers in different gaming environment. Apart from it, the electrodermal signal and facial expression analysis was performed. The
102 Khan and Pal
electrodermal signals were used to understand the change in skin conductance with time. The tonic and phasic signals were separated from the skin conductance signals. The mean tonic and phasic signal value was computed for each minute value. Based on these mean values, the normalized values for each minute was computed. The normalization was performed to mitigate participant based variations in GSR activity.
The emotional behaviour was obtained by computing the change in facial expression with time. Different parameters such as eye closure, eye widening, chin movement, brow furrow, lip suck, lip pucker, smile and smirk were computed using iMotions. The time series data of facial expression parameters was used to compute mean value at each minute.
RESULTS
Subjective Responses The subjective responses were recorded using the questionnaire based on Likert Scale. The participants were asked to rate questions regarding their mood, mental stress and physical state before and after the task as listed in Table 1. It was found the participants experienced a little increment in the frustration rating. When asked to select the option which mentions their emotions after the game, none mentioned the calm or neutral emotions. The ratings on the mental stress caused by the task also indicated that it increased by a small number. However, this increment was not significant as the task was not performed was significantly long duration. Lastly, no change in the pattern was observed in the physical state ratings. Apart from it, the participants were asked about the difficulty level task. Most participants reported the difficulty level about the around 5. Although the participants found the task mentally demanding but they felt accomplished after completion of the task.
Table 1: Subjective dataset showing mood, mental state and physical state before and after the gaming task. The average rating shows that there is a small change in mood and mental state perceived by the participant.
Table available in the original PDF paper.
Before After Mood (0 for Frustrated and 10 for Exited) Mood (0 for Frustrated and 10 for Exited) Current mental state (1 for pleasant and 10 for bad) Current physical state (1 for pleasant and 10 for bad) Current mental state (1 for pleasant and 10 for bad)
1.4 1.4 1.3 1.6 1.2 1.5
Current physical state (1 for pleasant and 10 for bad) Average 6.9 4.3 3.8 5.2 5.6 3.9 Standard Deviation
Physiological Signals The change in tonic and phasic activity of electrodermal signals was computed. It was found that the two activities increased with time. The mean value of the tonic activity at the 1st minute of the gaming task was
Change in Affective Dynamics and Cognitive State With Time 103
observed 1.782 ± 0.946. During 15th minute of the task, the mean value of the tonic signal was found 3.739 ± 1.702. The mean value of the phasic activity first and last minute of gaming task was found 0.0216 ± 0.0155 and 0.0467 ± 0.0242 respectively. To further investigate the peak and valley in the tonic and phasic signals, normalized mean value was computed. The normalized mean value for each minute showed that valley in the tonic signal was obtained in initial 3 minutes of the task for all participants except one. On the other hand, the peak in the tonic signal was obtained towards the last 5 minutes of the task as shown in Figure 1. The highest and lowest normalized mean values in the phasic activities were obtained after 10 minutes of the task.
The number of GSR peaks during the task were more than 100 for different participants. However, the number of peaks were constant with time. The peak amplitude of the SCR peak ranged from 0.05-0.5 µSiemens among participants when 100 or more GSR peaks.
Figure 1: Pictorial representation of tonic and phasic component of the EDA signals. The normalized mean absolute value for the phasic component, it is observed that the difference between the subsequent values increases after 5 minutes of the task (mostly after 8th or 9th minute).
Change of Facial Expression With Time The change in facial expression was measured to identify the emotion and valence change during the task. To identify various emotions during the task, time series data of the facial expressions based on eyes, brows, lip and nose are studied as shown in Figure 2. The time series data ranges from 0 (no-expression) to 100 (expression fully present). The first minute value is considered as the baseline value.
It was found that most of the changes were observed in eye closure, brow furrow, lip press, mouth open and lip suck. Apart from it, smaller changes were observed in facial expression such as smile, smirk and chin movement. Rest of the facial expression scores were less than 1 in most cases. The eye closure and lip suck were observed to decline with time. These declines in eye closure and lip suck were observed after 8 minutes and 9 minutes in the task respectively as shown in Figure 2. On the contrary, facial expression such as brow furrow and lip pucker increased towards the end of the gaming task.
104 Khan and Pal
These increments were observed after 9 minutes in the task. These facial expressions were used to indicate the emotional changes taking place during the gaming task. The emotions were ranked on a scale of 0–100. Since significant number of facial expressions was observed after 9 minutes, the emotional changes were compared between 1–9 minutes and 9–15 minutes. The positive emotions included attention, engagement, sentimentality and joy. The negative emotions included fear, anger and surprise. Based on the scale, the emotions attention, engagement and joy were ranked highest. All these emotions are positive emotions. However, all these emotions were observed to decline after 9 minutes of the task as shown in Figure 3. Referring to the negative emotions shows that anger, fear and surprise are scored less. Numerically, the score for fear and surprise increased after 9 minutes. No significant increment was observed in the anger emotion.
Figure 2: Pictorial representation of the facial expression based on eyes, brow, nose and lips. It is observed that the facial expression started to change after 5 minutes of the task and before 10 minutes. For instance, the trend in the FES score decreased from 10 minutes onwards for chin raise. However, the FES score for the brow furrow increased after 10 minutes.Figure 3: Exploration of the different emotions shows that the gamers started the game with positive emotions. (a) Joy, positive valence, engagement and attention was higher at the first half of the experiment time. It reduced in the 2nd time group. (b) The score for negative emotions was minimal. However, the box plot representations shows that the score increased for 2nd time group; indicating increase with time.
Change in Affective Dynamics and Cognitive State With Time 105
DISCUSSION AND FUTURE WORK This study was conducted to obtain preliminary outcomes in the form graphical trends on the player-challenge interaction using physiological and non-physiological signals. Both EDA signals and facial expressions changed trend with time. The tonic component of EDA signals showed increasing trend throughout the gaming task, indicating that the tonic component largely represents cognitive load with time. The phasic component was sinusoidal in nature and the difference between subsequent values increased after 8–9 minutes. Facial expression analysis also showed that facial properties changed after 8–9 minutes such brow furrow, lip suck and eye widen. Emotions computation, based on facial expression analysis also showed that emotions changed during the second half of the task, especially positive emotions such as joy and engagement. However, there was no significant emotion changes in the particular timeline. This trend may indicate that phasic component of the electrodermal signals can be mitigated with facial expression can be used for improved emotion recognition. A study by Ganapathy et al. also used for phasic component for emotion classification by showing images on the screen (Ganapathy, Veeranki and Swaminathan, 2020). In future; statistical analysis and machine learning approaches on a large dataset will be applied to explore emotion recognition during gaming.
Apart from it, multiplayer FPS games are complex in nature as compared to rest of the games. Thus, it becomes necessary to control the gaming environment. For instance; the opposition players may vary according to their expertise. This changes player-challenge interaction based on the objective of the game. For instance; a time control objective becomes hard as compared to the objective required to be achieved without time limits. Lastly, the inter subject variation of the physiological signals depending on the response to stimuli indicates the need to perform subjective specific analysis. Such studies can be used to design real time emotion detection systems to detect cognitive fatigue and mental state of an individual playing games.
REFERENCES
Ballou, N. et al. (2024) ‘The Basic Needs in Games Scale (BANGS): A new tool for investigating positive and negative video game experiences’, International Journal of Human Computer Studies, 188(May). Available at: https://doi.org/10.1016/ j.ijhcs.2024.103289. Baltzar, P., Hassan, L. and Turunen, M. (2023) ‘Social accessibility in multiplayer
games: Theory and praxis’, Entertainment Computing, 47(July). Available at: https://doi.org/10.1016/j.entcom.2023.100592. Caroux, L. et al. (2015) ‘Player-video game interaction: A systematic review of current concepts’, Computers in Human Behavior, 48, pp. 366–381. Available at: https://doi.org/10.1016/j.chb.2015.01.066. Chin, S. C. et al. (2024) ‘Altered Heart Rate Variability During Mobile Game
Playing and Watching Self-Mobile Gaming in Individuals with Problematic Mobile Game Use: Implications for Cardiac Health’, Psychology Research and Behavior Management, 17, pp. 2545–2555. Available at: https://doi.org/10.2147/ PRBM. S469240.
106 Khan and Pal
Cloude, E. B. et al. (2022) ‘Affective Dynamics and Cognition During Game-Based Learning’, IEEE Transactions on Affective Computing, 13(4), pp. 1705–1717. Available at: https://doi.org/10.1109/TAFFC.2022.3210755. Denisova, A. et al. (2020) ‘Measuring perceived challenge in digital games:
Development & validation of the challenge originating from recent gameplay interaction scale ( CORGIS)’, International Journal of Human-Computer Studies, 137(December 2019), pp. 1–15. Denisova, A. et al. (2021) ‘Whatever the emotional experience, it’s up to them: Insights from designers of emotionally impactful games’, Conference on Human Factors in Computing Systems - Proceedings [Preprint], (1). Available at: https:// doi.org/10.1145/3411764.3445286. Ganapathy, N., Veeranki, Y. R. and Swaminathan, R. (2020) ‘Convolutional neural
network based emotion classification using electrodermal activity signals and time-frequency features’, Expert Systems with Applications, 159. Available at: https://doi.org/10.1016/j.eswa.2020.113571. Greco, A. et al. (2017) ‘Arousal and valence recognition of affective sounds based on electrodermal activity’, IEEE Sensors Journal, 17(3), pp. 716–725. Available at: https://doi.org/10.1109/JSEN.2016.2623677. Luo, J. et al. (2024) ‘A mobile device-based game prototype for ADHD: Development
and preliminary feasibility testing’, Translational Psychiatry, 14(1). Available at: https://doi.org/10.1038/s41398-024-02964-2. Moschovitis, P. and Denisova, A. (2023) ‘Keep Calm and Aim for the Head: Biofeedback-Controlled Dynamic Difficulty Adjustment in a Horror Game’, IEEE Transactions on Games, 15(3), pp. 368–377. Available at: https://doi.org/ 10.1109/TG.2022.3179842. Park, K. et al. (2024) ‘The Effect of Time Spent on Online Gaming on Problematic
Game Use in Male: Moderating Effects of Loneliness, Living Alone, and Household Size’, Psychiatry Investigation, 21(2), pp. 181–190. Available at: https://doi.org/10.30773/pi.2023.0027. Peng, X. et al. (2020) ‘A Palette of Deepened Emotions: Exploring Emotional Challenge in Virtual Reality Games’, Conference on Human Factors in Computing Systems - Proceedings, pp. 1–13. Available at: https://doi.org/10.1145/ 3313831.3376221. Peng, X. et al. (2023) ‘ChallengeDetect: Investigating the Potential of Detecting In-
Game Challenge Experience from Physiological Measures’, Conference on Human Factors in Computing Systems - Proceedings [Preprint]. Available at: https:// doi.org/10.1145/3544548.3581232. Pretty, E. J. et al. (2024) ‘Multimodal Measurement of Cognitive Load in a Video Game Context: A Comparative Study Between Subjective and Objective Metrics’, IEEE Transactions on Games, PP (X), pp. 1–14. Available at: https://doi.org/ 10.1109/TG.2024.3406723. Vatsal, R. et al. (2024) ‘An Analysis of Physiological and Psychological Responses in Virtual Reality and Flat Screen Gaming’, IEEE Transactions on Affective Computing, pp. 1–15. Available at: https://doi.org/10.1109/ TAFFC.2024.3368703. Zagermann, J., Pfeil, U. and Reiterer, H. (2016) ‘Measuring cognitive load using
eye tracking technology in visual computing’, ACM International Conference Proceeding Series, 24-October, pp. 78–85. Available at: https://doi.org/10.1145/ 2993901.2993908. Zhang, X. et al. (2018) ‘Evaluating Photoplethysmogram as a Real-Time Cognitive Load Assessment during Game Playing’, International Journal of Human- Computer Interaction, 34(8), pp. 695–706. Available at: https://doi.org/10.1080/ 10447318.2018.1461763.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors and Wearable Technologies
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-51-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional