Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Exploration of Patterns in Emotional Response and Retrospective Cognitive Load during Interaction with a Virtual Product

1 Universidad Autónoma Metropolitana Unidad Xochimilco. CyAD, Departamento de Tecnología y Producción, Ciudad de México, Mexico ↗
2 Universidad Nacional Autónoma de México, Facultad de Artes y Diseño, Ciudad de México, Mexico ↗
Publication year
Digital Object Identifier10.54941/ahfe1004587

Abstract

A cross-sectional study was conducted with 9 students in a classroom, allowing the identification of certain patterns during their interactions with a virtual product used as an educational tool. A correlation was observed among emotional response, cognitive load, and the dominant linguistic system in the virtual product. It was also noted that the prevalent use of a linguistic system can lead to a higher demand for resources for accurate spatial localization, contributing to the generation of high cognitive load. Additionally, these factors were observed to influence the perception of time in these studies. The utilization of self-report methods can offer valuable insights into the user's perspective, enhancing our understanding of how their individual interaction processes are constructed.

INTRODUCTION

Cognitive Load (CL) refers to the mental effort or mental load that a person must allocate to a particular task or activity. Block defined it as the demands for information processing, including both attentional and working memory demands (Block, Hancock, & Zakay, 2010). It is used to describe the amount of mental resources, such as attention, memory, information processing, and concentration, required to carry out a specific task or cognitive process. Among some of the most influential researchers in the study of CL, there are various studies conducted by Sweller, Paas, and Renkl, who developed a Theory of Cognitive Load (Paas, Renkl, & Sweller, 2003), (Sweller, 1988). This theory primarily focuses on the learning of complex cognitive tasks, where users exhibit specific responses related to the quantity of information elements and the interaction process that is executed. Instructive control of this high load, aiming to achieve meaningful learning in complex cognitive domains, has become the theory’s focus. Cognitive load, from Sweller’s per- spective, can be addressed from an intrinsic or extrinsic standpoint. Methods

© 2024. Published by AHFE Open Access. All rights reserved. 362

Exploration of Patterns in Emotional Response and Retrospective Cognitive Load 363

for measuring CL include physiological measures, performance on the pri- mary task, performance on secondary tasks, and surveys of opinion (Block, Hancock, & Zakay, 2010). In this regard, the present study focused on mea- suring intrinsic cognitive load from the perspective of individuals’ awareness of their own activity, in other words, a retrospective perspective. This can pro- vide relevant insights into the construction of individuals’ CL during various interaction processes, where individuals experience CL in correlation with the qualities of the objects they interact with daily. Intrinsic load reflects both the inherent difficulty of a given task and the learner’s prior knowledge or expe- rience with that task. The degree of interaction with the material and their experience with the object also plays a crucial role in intrinsic cognitive load (Fraser, Ma, Teteris, Baxter, & Wrigh, 2012); Therefore, CL is influenced by multiple resources. The correlation between emotional response and CL is a complex activity to observe. Recent studies suggest that emotions can be considered from three perspectives: as an extrinsic CL, as an effector that alters the way information is encoded, stored, and retrieved even before the individual becomes aware of the material, and as a factor that influences intrinsic cognitive load (Plass & Kalyuga, 2019). Plass and Kaplan proposed the Integrated Model of Cognitive-Affective Learning with Media (ICALM), giving emotion processing the same priority as visual and verbal information processing and considering emotion as a separate processing channel (Plass & Kaplan, 2016). The intention of these studies is to observe possible corre- lations between the individual responses given by a group of people in a local environment and the cognitive load derived from the exploration of an every- day virtual product in an academic context. Based on this, it was determined, through a cross-sectional study, to observe two components of Emotional Response during the interaction processes: valence and intensity. The first component determined through self-report, and the second component based on the number of emotional components determined by nuances.

METHODOLOGY

The study was divided into seven phases. In the first phase, an empirical approach was used where volunteers self-reported their age, cultural context, level of education, gender, and emotional response. The context was the class- room, classified as an environment where certain control elements have been incorporated, such as the number of people, lighting, sound, furniture, among other factors. The responses were recorded in the first phase on their techno- logical device. In the second phase, the group of individuals was assigned a specific task: 1) Access the class website, 2) Navigate the website, 3) Locate the exercise that they found most interesting. The analyzed artifact is a vir- tual product corresponding to the Fundamental Fields of Design module. In the third phase, the cognitive load components to be observed were deter- mined. In this regard, ten components were identified as relevant: processing speed, factors related to the somatic nervous system, executive attention, spa- tial perception, navigation, usability, effort, time perception, open and closed linguistic system. In the fourth phase, cognitive load was classified as low, moderate, and high, with the corresponding NASA TLX scale ranges being:

364 Tejeda and Pineda

low = 0–33.33, moderate = 34–66.66, high = 67–100. In the fifth phase, emotions were categorized, along with their nuances. A total of 10 emotions were determined, which were categorized into positive, negative, and neutral valence. Additionally, 35 nuances were considered in the evaluation. Each emotion was recorded through self-report using the NASA TLX scale. In the sixth phase, an interview was conducted with each user, and the questions were linked to the classification of cognitive load. In the seventh phase, the correlation between emotional responses and CL was analyzed based on a subjective assessment of the interaction processes with the assigned virtual product. The correlation of emotional responses recorded on the day and those recorded regarding the analyzed artifact was also observed. It is impor- tant to mention that the technological devices used for color reproduction were personal.

PROCESS DESCRIPTION A total of 9 students participated, with 44.44% representing the Women’s sector and 55.55% the Men’s sector. Among the Women’s sector, 22.22% self-reported being confident about their emotions, while 44.44% of men reported being confident about their emotions. The emotional state recorded at the beginning of the activity showed 11.11% of individuals with a posi- tive valence and 44.44% with a negative valence. Within the Women’s sector, 0% registered a positive valence emotion, while 11.11% of the Men’s sec- tor recorded a positive valence emotion. Additionally, 44.44% of all users recorded a neutral emotion. Of the women, 11.11% recorded a neutral valence emotion, and 33.33% of the men recorded a neutral valence. At the beginning of the activity, students in the classroom were asked if they wanted to participate in a research experiment, where they would need to provide some data, to which all 9 students in the classroom raised their hands. The digital working instrument was then provided, through which they recorded self-reported data, including age, gender, place of residence, level of educa- tion, and initial emotional state. Afterward, they were assigned the described task, which essentially involves an exploratory activity with the virtual prod- uct used in the activities described in the subject’s curriculum. Therefore, it is an educational tool not unfamiliar to them, as they had been working with it for 7 weeks, with an average use of 3 days per week. Regarding the emo- tional response obtained through self-report and linked to the interaction with the virtual product, 55.55% defined their emotional response as posi- tive, while 44.44% defined it as negative. In this sense, it was observed that the initial emotional state was directly linked to the emotional response to the product as follows: 22.22% of individuals as users determined an emotional response different from the one recorded as their initial emotional state, while 77.77% linked a similar emotional response to the product with their initially recorded emotional state.

The classification of emotions, determining their valence and nuances, was very useful for generating a more detailed record in trying to differ- entiate, if possible, the initial emotional state from what they felt when interacting with the virtual product. Each nuance was defined, providing

Exploration of Patterns in Emotional Response and Retrospective Cognitive Load 365

users with an anchor point regarding whether they perceived any of the emotional nuances described during the interaction process. Regarding the 35 emotional nuances, the following observations were made: The average intensity of the emotion relative to the object in most users was low, with an average of 17.694. The valence in most users was negative, with an aver- age of –1.222. Likewise, it was observed that 66.666% of users recorded an emotion with negative valence, 11.111% of users recorded an emotion with positive valence, while 22.222% of users recorded a neutral valence. In this sense, it is noted that although the object does not have an emotional impact on users, the average prevails as a negative response. Similarly, regarding cognitive load, it can be observed that the result of the interaction process generated an average CL that can be classified as moderate, with an average of 55.971. In terms of cognitive load, retrospection through self-report was considered for 34 factors classified into 10 categories.

Figure 1: The figure illustrates the retrospective construction of the emotional response of the user group.
Original PDF Figure 1:

The construction of retrospective cognitive load was carried out through direct interviews, where each user was asked a series of structured questions in the same way. However, if they had doubts, clarification was provided regarding the concepts used in the 10 components selected as references to measure CL based on the mentioned scale Fig. 1. A pattern observed in this analysis is that CL was linked to users in relation to six categories primarily: time, navigation, usability, closed linguistic system, spatial factors, and effort. Additionally, in the presence of moderate CL, the intensity of the emotional response behaved in a low range in relation to the scale determined for these studies. Among the six categories, time, closed linguistic system, and space were the most influential factors in the construction of retrospective cognitive

366 Tejeda and Pineda

load and in correlation with the object they interacted with Fig. 2. In this sense, it is observed that although CL was recorded as moderate, various factors intervened in its perception.

Figure 2: The figure shows the cognitive load and its relationship with other factors in the interaction.
Original PDF Figure 2:

Likewise, it is worth noting that the task was exploratory in nature and lasted for 300 seconds. The results presented indicate that it is a digital tool that demands various complex cognitive processes due to its configu- ration and composition characteristics. These processes are systematically intertwined with gross and fine motor interaction processes in the working system. Regarding time perception, 55.5% of the user group showed a fast perception of time, while 44.445% had a moderate perception of time. This may suggest that the perception of this factor influences high levels of cog- nitive load. Additionally, it was observed that in factors requiring complex cognitive processing, such as meaning construction, the intensity of the emo- tional response remained at a low level. On the other hand, a correlation was observed between time perception and the perceived dominance of the linguistic system in the virtual product. In this sense, 88.888% of users per- ceived a large amount of text in the product they interacted with. It was also noted that there is a correlation between text perception and emotional response; with a higher perception of text, the intensity of the emotional response remains low for most users. Regarding space, 66.666% of users made use of various spatial components at a high level during interactions with the virtual product. It is observed that the lower the perception of cog- nitive load derived from spatial components, the more intense the emotional response.

Exploration of Patterns in Emotional Response and Retrospective Cognitive Load 367

In relation to the nodes, there were a total of 1586, integrated by the enti- ties: Gross Motor Interaction (GMI); Visual Interaction (VI); Fine Motor Interaction with Fingers (FMI-F); Digital Pressure (DP); Sound Interaction (SI); and processes of Focused Attention (FA), Break Points (BP). It was observed that the most active mechanisms throughout the process were those related to fine motor interaction, with a total of 1153 interactions, consist- ing of the dyad of FMI-F and DP. This indicates a high investment of mental resources in these interactions and correlates with Visual Symbols (VS) as components of virtual products, with a total of 1048 interactions. It is impor- tant to clarify that these are hierarchical processes, and for each press (DP), there is a prior process related to the location of the VS. Additionally, a total of 14371 interactions were observed with all components of the process, of which 44.83% corresponded to the location of the VS, including processes of GMI, VI, Prop, and spatial location; 52.898% to fine motor interaction, including finger movement and digital pressure. Likewise, a correlation was observed between the intensity of emotional valence and users with a higher number of interactions with VS and fine motor interaction. Below is the net- work of User 9, who had the highest number of nodes with a total of 393, a number of connections of 69605, and a network density of 0.9036 Net. 1.

Net. 1: The network displays the interaction process of U9 for 300 seconds, with a network density of 0.9036.

DISCUSSION

Variability in emotional responses is observed; however, patterns that can help understand the interaction processes perceived by users and the cate- gories determined for this study could be detected. Additionally, a correlation between cognitive load and emotional response can be observed, where arti- facts that users identify with considerable cognitive load may tend to generate a low intensity of emotional response. Regarding the valence of the emo- tional response, which, despite being of low intensity, had a negative valence,

368 Tejeda and Pineda

this result can also be linked to the understanding of information because several users expressed during the interview that they did not fully compre- hend the information. The type of analysis conducted by each user, both in their self-report and during the interview, implies being aware of the activity performed with the object. On the other hand, a very interesting pattern was observed in relation to three categories that can be influential in the interaction with virtual products: interaction time, closed linguistic system, and space. The pattern was detected that the interaction process with closed linguistic systems, such as text, generates a perception in the majority of users that time passes very quickly, and possibly, this type of linguistic resource implies the greatest use of spatial localization resources. The richness of observed emotional responses reveals complexity in the user experience during interaction. A revealing correlation between cognitive load and emotional response suggests that artifacts demanding considerable CL tend to generate more subdued emotional responses. This phenomenon could indicate a relationship between the complexity of the interaction and the experienced emotional intensity. Regarding the valence of the emotional response, although of low intensity and a tendency towards negativity, this finding could be associated with the initial emotional state with which a person begins a task, which may interfere with the understanding of information.

CONCLUSION

The study suggests that factors such as gender, initial emotional state, time perception, closed linguistic system, and the use of spatial components play an important role in the user experience during interactions with virtual products. Additionally, the existence of a relationship between cognitive load and emotional response was observed, which is important to consider when designing this typology of virtual products used as educational tools. It also helps to understand that users are not cognitively passive during interactions with certain virtual products; in other words, CL occurs from the moment of interaction.

On the other hand, an impact of the initial emotional state was observed, where most users maintained a similar emotional response to their initial emotional state during the interaction with the virtual product. This suggests that the initial emotional state plays an important role in the perception of the experience, with a predominance of negative emotional valence in most users after interacting with the virtual product. This also suggests that the qualities of the virtual product that required moderate cognitive load in users con- tributed to maintaining the negative valence of emotion. Additionally, it is observed that not only a state of flow with positive emotional valence gen- erates a fast retrospective perception of time but also occurs with negative emotional valence linked to specific product qualities, such as the linguistic system. Among these, time, the use of spatial resources, and the closed lin- guistic system were the most influential in generating moderate CL, linking components related to fine motor interaction and complex processes for the localization of the VS (Visual Symbols).

Exploration of Patterns in Emotional Response and Retrospective Cognitive Load 369

REFERENCES

Block, R., Hancock, P., & Zakay, D. (2010). How cognitive load affects duration judgments: A meta-analytic review. Acta psychologica, 134(3), 330–343. Fraser, K., Ma, I., Teteris, E., Baxter, H., & Wrigh, M. (2012). Emotion, cogni-

tive load and learning outcomes during simulation training. Medical education, 46(11), 1055–1062. Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist (38), 1–4. Plass, J., & Kaplan, U. (2016). Emotional design in digital media for learning. (I. S.

Gartmeier, Ed.) Emotions, technology, design, and learning, 131–162. Plass, J., & Kalyuga, S. (2019). Four ways of considering emotion in cognitive load theory. Educational Psychology Review (31), 339–359. Sweller, J. (1988). Cognitive load during problem solving. Cognitive Science, 12,

257–285.

Cover of the proceedings volume Human Interaction and Emerging Technologies (IHIET-AI 2024): Artificial Intelligence and Future Applications

Explore the complete digital volume

Human Interaction and Emerging Technologies (IHIET-AI 2024): Artificial Intelligence and Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-96-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →