Valentin Wunsch, Knut Möller, Verena Wagner-Hartl (2025). Avatar-based emotion representation with varying degrees of realism. In: Intelligent Human Systems Integration (IHSI 2025): Integrating People and Intelligent Systems. AHFE Open Access, vol. 160, pp. 184–193. https://doi.org/10.54941/ahfe1005824
In the age of digitalization with its progressive visualization of a wide variety of content, avatars are becoming increasingly important. Consequently, virtual human representations are also being used in healthcare, for example in therapeutic applications. The use of technologies such as virtual reality or virtual characters like avatars can support people who find social interactions challenging. This is often the case for people with autism spectrum disorder (ASD). The aim of the presented study was to evaluate the suitability of different avatar types for the development of a future digital application for people with ASD. Based on the results of a previous study, two avatar types that differed in their degree of realism were analysed: a cartoonish, simple avatar and a stylized, moderately detailed avatar. First, it was investigated how participants set different basic emotions on these avatars and whether these settings are comparable between the different types of realism as well as with previously defined values for the basic emotions. Afterwards, a threshold test was used to determine at what point the emotions displayed by the avatars became recognizable and at what point they were rated as pronounced. Overall, ten men and ten women (M = 24.9 years old, SD = 3.64) participated in the study. The results show that the average values set by the participants correspond to the previously defined values for the basic emotions, with a relatively high degree of variation in some cases. Overall, the insights gained within this experiment provide a basis for future research on this topic.
Avatars are representations of human behaviour in simulated and virtual environments. They allow people to participate in virtual actions by imitating the user’s real intentions and actions in a virtual world (Rahill and Sebrechts, 2021). Because of the progress in digitalization avatars are being used in more and more contexts: In education, for example, avatars enable collaborative and synchronized learning in virtual environments (Jovanovi´c and Milosavljevi´c, 2022). The context in which avatars are used seems to play a role in their design by users (Wu et al., 2023). Wagner-Hartl et al. (2024) showed in an exploratory study that there are preferences in the scope of the
Avatar-Based Emotion Representation With Varying Degrees of Realism 185
desired representation of avatars within different environments, for example the gaming context and the professional context.
Avatars are also used in the area of healthcare for therapeutic purposes, for example for the supportive treatment of people with schizophrenia spectrum disorder in order to be able to manifest schizophrenic impressions as an avatar (Vass et al., 2024) or in the treatment of Posttraumatic Stress Disorder (PTSD) in the form of role-playing games to deal with traumatic events (Davis and Alexanian, 2024).
Role of Avatars in Therapeutic Applications for Individuals With ASD Avatars also play a role in therapeutic applications for subjects with autism spectrum disorder (ASD). The National Institute of Mental Health (NIMH, n.d.) defines ASD as “[…] a group of complex neurodevelopmental disorders […]”. There are different estimations available: Worldwide, it is estimated that one in 100 children is diagnosed with ASD (Zeidan et al., 2022). Rylaarsdamm and Guemez-Gamboa (2019) note, that it is “[…] one of the most prevalent neurodevelopmental disorders.” (ibid. p. 1). People with ASD may exhibit repetitive behaviours and/or restricted interests and activities as well as difficulties in communication and social interaction, or the perception that these skills are challenging (American Psychiatric Association, 2013; Lord et al., 2018; National Institute of Mental Health, n.d.).
Social skills include verbal and non-verbal behaviours that are necessary for effective communication between individuals in complex situations (Freeth and Morgan, 2023; Rao et al., 2008). Especially for people with ASD these behaviours can be perceived as challenging in social situations as they often lack the ability to recognise emotions in others which can make it difficult to participate in social situations (Hopkins et al., 2011; Lord et al., 2018; Tebartz van Elst et al., 2021). Addressing these issues, modern therapy approaches for people with ASD regarding social skills training may use avatars as the virtual representation of human interaction to offer a save environment in varying contexts (Nojavanasghari et al., 2017). The work from Kellems et al. (2020) suggests that avatars might also lead to a higher engagement in the therapeutic intervention as the interaction with them is preferred in comparison to interaction with humans.
One aspect of social skills training is the training of expression and recognition of emotions. The outwardly visible components of emotions such as gestures or facial expressions are understood on an inter-individual basis (Brandstätter et al., 2013). The facial expression of emotions can be categorised into various basic emotions. Following Ekman, the basic emotions are happiness, sadness, surprise, disgust, fear and anger (Ekman, 1982). The underlying basis for the expression of emotions can be described with the Facial Action Coding System (FACS; Ekman and Friesen, 1978). The FACS characterises the visible facial movements and does so by classifying the individual facial moving parts into groups. The expression of certain emotions leads to the movement of the groups (Ekman, 1982). Pino et al. (2021) showed that children with ASD had a higher exploration rate towards
avatars compared to real faces as well as a higher accuracy in emotion recognition with avatar faces than with real faces.
Development of Avatars for Therapeutic Application for Individuals With ASD The presented study contributes to a project which aims to develop a digital therapeutic application for children with ASD (Abdulbaki Alshirbaji et al., 2024). The final goal of the project is to create a tool which provides the user with situation-specific and individual feedback during the procedure based on his reaction to social training situations (Arabian et al., 2023, 2024). As the tool addresses individuals with ASD in particular, it is relevant that the virtual presentations achieve a quality that in no way contributes to the reinforcement of negative feelings, as this might impair the achievement of the therapy goal. To investigate whether the avatars planned to be used for the application are suitable for expressing emotions, a study was conducted to answer the following research questions:
(1) How do participants adjust the facial parameters of the avatars in order to achieve certain target emotions? (2) Is it possible to transfer the adjustments of the facial expressions from one avatar to the other avatar while sustaining the underlying meaning of the emotion? (3) What minimal strength (feature thresholds) of an emotional expression is required for a participant to recognize an emotion? From which value participants recognize the expressed emotion for the first time and when is it perceived as pronounced?
METHOD
Participants Overall, 20 participants (10 male, 10 female) aged between 19 and 35 years (M = 24.9, SD = 3.64) participated in the exploratory study. 18 participants were university students and two participants were university employees. The participants were recruited at Furtwangen University through personal contact. Regarding their gaming experience 12 participants stated to play video games seldom or rather seldom, four participants stated to play videogames occasionally and four participants stated to play them rather often or often. The samples mean value of the Affinity for Technology Interaction Short (ATI-S) Scale (Franke et al., 2019) was 4.43 (SD = 1.21, Cronbachs α =.91; 4 items using a 6point rating scale ranging from 1 = completely disagree to 6 = completely agree). The participants took part in the study voluntarily and received no compensation. They provided their informed consent at the beginning of the study.
Materials
Three different types of avatars were developed for use in the project: (1) a cartoonish, simple avatar with complexity reduced facial features (allowing the movement of eyebrows as well as the change of size and shape of eyes
Avatar-Based Emotion Representation With Varying Degrees of Realism 187
and mouth), (2) a stylized, moderately detailed avatar with more complex facial features (e.g. introducing additional parameters such as nostril and cheek movement) and (3) a realistic, highly detailed avatar that extended the beforementioned adjustment options. Just two (cartoonish, stylized) out of the three representations of avatars were used in this study, to keep the duration of the experiment to a minimum. The selection was based on the results of a previous study which investigated the used avatars with a paper- based approach (Wagner-Hartl et al., 2024). The displayed emotions of avatars analysed in this study were chosen based on the study as well, as it revealed that the avatar representation of the emotions happiness, anger and surprise were recognized best.
An application was developed with Unity 2022.03.17 to manipulate the avatars facial features and to express different emotions, see Figure 1 and Figure 2. The FACS classification was used to derive target emotions according to the emotion classification from Ekman (1982). Between a neutral facial expression and the target emotions blend shapes were created enabling the continuous change of the facial features. This also allowed to manipulate single facial parts along the defined blend curves.
Procedure During the study the participants performed three different tasks, emotion adjustment, crossover and threshold test: For the emotion adjustment both avatar types were presented to the participants one after the other and were introduced to the handling of the specific interface (Figure 1) and the different variables were explained. Participants started from a neutral expression and adjusted the facial parameters of the avatars using sliders for each facial feature until a certain emotion (anger, happiness and surprise, presented in a randomized order) was reached.
Figure 1: Interface – Adjustment of facial features.
For example, to display the emotion sadness the participants could move a slider to move the corners of the mouth downwards if they felt that this corresponded better to the respective emotional expression. Final settings were saved when the participant was satisfied by his result.
Second, for the crossover task the final values from the emotion adjustment task were transferred to the other avatar type, respectively. The participants then had to rate how much of the emotional strength and quality was transferred to the other avatar on a scale from zero to 100 percent. They were also asked to name the specific adjustments that would need to change in their opinion in order to get as close as possible to the target emotion.
The final task consisted of a threshold experiment where the participants were asked to move a slider for each of the six different used target emotions (anger, disgust, fear, happiness, sadness and surprise) that changed the avatars facial expression from a neutral expression to a strong emotional expression, see Figure 2. For each of the target emotions presented in random order they were asked to report at which slider value they would perceive the expression for the first time and at which slider value they would perceive the emotion as pronounced. The avatars were presented in a randomized order.
Figure 2: Interface – Threshold experiment.
RESULTS
Avatar Representation of Emotion Regarding the first research question, how the participants would adjust the facial parameters of the avatars in order to achieve certain target emotions, the mean values of each adjustment were calculated and transferred to the avatar interface. Figure 3 shows the mean values of the facial parameter adjustment task. They were compared with the 70 percent slider values of the threshold experiment which were used as a reference value based on expert ratings prior to the presented experiment.
Avatar-Based Emotion Representation With Varying Degrees of Realism 189
Figure 3: Mean values of adjustment task in comparison with the respective reference values from a prior expert rating.
Emotion Transferability Between Avatars To investigate whether the adjustments of the facial expressions from one avatar could be transferred to the other avatar the participants rated the transferred values for correspondence with the intended emotion. Figure 4 shows the ratings of the transferred emotion settings with “pts” standing for the values of the cartoonish avatar being transferred to the stylized avatar and “stp” meaning the opposite.
Figure 4: Transfer task – Rating of the emotion representation (pts_x: transfer of emotion information from cartoonish to stylized avatar, stp_x: vice versa).
Threshold Experiment for Emotion Detection To answer the third research question the participants had to indicate the level at which the emotion was just noticeable and when they would perceive the emotion as pronounced. Figure 5 shows the distribution of the individual
values for the emotion anger. The mean values and standard derivations for all emotions are presented in Table 1 for the cartoonish avatar and in Table 2 for the stylized avatar.
Figure 5: Threshold task – Participants values and mean value of emotion anger.Table 1. Mean values for noticeable and pronounced threshold for the cartoonish avatar.
The aim of the presented study was to evaluate the suitability of different avatar types for the development of a future digital application for people
Avatar-Based Emotion Representation With Varying Degrees of Realism 191
with ASD. Regarding the first research question how participants adjust different facial parameters of the two analyzed avatar types (a cartoonish, simple avatar and a stylized, moderately detailed avatar) to achieve the used basic emotions anger, happiness and surprise, different facial expressions were found. The descriptive comparisons with previous expert ratings showed no major differences. Based on these results, it can be concluded that the basic representation of emotions through the avatars is possible. Furthermore, considering the results, it can be reasonably inferred that there is a comparable comprehension of the emotional expressions associated with the three analysed emotions. Differences in the mean values adjusted by the participants in comparison to the reference values might be explained with interindividual differences how emotions are interpreted by human beings. For example, for the emotion “surprise” there is a slight deviation between the mean value of the cartoonish, simple avatars mouth and the value set in the prior expert rating. Following the results, some participants interpreted surprise as a positive emotion, resulting in a more o-shaped mouth form and some interpreted it as a negative emotion, which resulted in a more c-shaped mouth form. Such possible interpretations of the emotion surprise shall be considered in future studies.
Furthermore, regarding the second research question, the results show that overall, the transfer between the avatars resulted in a loss of information. This effect can be found especially when transferring the parameters from the cartoonish, simple avatar to the stylized, moderately detailed avatar. This may be due to the reduced complexity of the cartoonish avatar which left some values of the stylized avatar unchanged. Additionally, the results show that the emotion anger performed poorly in both transfer directions. Nonetheless, the effect was more pronounced for the transfer direction stylized to cartoonish avatar than vice versa. For this specific emotion an explanation could be that the differences in the way the facial parameters could be altered were too big to transfer its meaning onto the other avatar. For future application the compatibility of the used parameters across the avatars should be considered.
A threshold experiment was used to answer the third research question. Following the results, the participants recognized the expressed emotions rather late with values often close or above 50 percent of the range of the presented emotional spectrum. Especially the emotion representations of the stylized avatar were recognized rather late. Following this effect, the ratings of perceiving the emotional expressions as pronounced were also chosen rather high on the given emotional spectrum within the threshold experiment. Therefore, in comparison to the cartoonish avatar it could be assumed that the emotional features of the stylized avatar were changing too slow at the beginning of the presented range to be comparable with the cartoonish avatar or, on the other hand, relatively small changes in emotional expression can be more clearly represented using a simpler avatar type. However, further experiments are needed for a better understanding and validation of these initial results.
As with any research, the study has some limitations. First, based on the exploratory approach the sample size was relatively small. Second, just
two types of avatars were compared. Third, because such evaluations may be challenging for people with ASD they were not included as subjects of the presented study. But of course, especially people with ASD should be considered as participants in subsequent studies with an application of a higher maturity level. Nevertheless, the results will help to gain more insight and will improve further design of avatars that can be used for the development of a future therapeutic application like described in (Abdulbaki Alshirbaji et al., 2024).
ACKNOWLEDGMENT
This work was funded by the Federal Ministry of Education and Research (BMBF) as part of the project LESSON [“FH-Impuls I: Situationsspezifisches Emotionstraining (Learning Emotions: Situation Specific ONline) (Lesson)”; 13FH5E10IA]. The authors would like to thank Marcel Igel and Philipp Hartz for their support regarding the used avatars as well as the participants involved in the realisation of the study.
REFERENCES
Abdulbaki Alshirbaji, T., Arabian, H., Wunsch, V., Wagner-Hartl, V., Möller, K., 2024. LESSON: Situation-specific emotion training for autism spectrum disorder children. Presented at the CoHMed Final Symposium Proceedings, 12th September 2024, Villingen-Schwenningen, Germany - Abstract Proceedings, pp. 43–44, 2024. American Psychiatric Association, DSM-5 Task Force, 2013. Diagnostic and
statistical manual of mental disorders: DSM-5TM, 5th ed. American Psychiatric Publishing, Inc., Arlington, VA, US. Arabian, H., Abdulbaki Alshirbaji, T., Chase, J. G., Moeller, K., 2024. Emotion Recognition beyond Pixels: Leveraging Facial Point Landmark Meshes. Applied Sciences 14, 3358. Arabian, H., Schmid, R., Wagner-Hartl, V., Moeller, K., 2023. Analysis of EDA
and Heart Rate Signals for Emotional Stimuli Responses. Current Directions in Biomedical Engineering 9, 150–153. Brandstätter, V., Schüler, J., Puca, R. M., Lozo, L., 2013. Motivation und Emotion, Springer-Lehrbuch. Springer Berlin Heidelberg, Berlin, Heidelberg. Davis, D., Alexanian, S., 2024. Role-playing recovery in social virtual worlds:
Adult use of child avatars as PTSD therapy. Computer Methods and Programs in Biomedicine Update 5, 100129. Ekman, P., 1982. Methods for Measuring Facial Action. In: Handbook of Methods in Nonverbal Behavior Research. Cambridge University Press, Cambridge, pp. 45–90. Ekman, P., Friesen, W. V., 1978. Facial Action Coding System. Consulting
Psychologists Press. Franke, T., Attig, C., Wessel, D., 2019. A Personal Resource for Technology Interaction: Development and Validation of the Affinity for Technology Interaction (ATI) Scale. International Journal of Human–Computer Interaction 35, 456–467. Freeth, M., Morgan, E. J., 2023. I see you, you see me: the impact of social presence
on social interaction processes in autistic and non-autistic people. Philos Trans R Soc Lond B Biol Sci 378, 20210479.
Avatar-Based Emotion Representation With Varying Degrees of Realism 193
Hopkins, I. M., Gower, M. W., Perez, T. A., Smith, D. S., Amthor, F. R., Casey Wimsatt, F., Biasini, F. J., 2011. Avatar Assistant: Improving Social Skills in Students with an ASD Through a Computer-Based Intervention. J Autism Dev Disord 41, 1543–1555. Jovanovi´c, A., Milosavljevi´c, A., 2022. VoRtex Metaverse Platform for Gamified
Collaborative Learning. Electronics 11, 317. Kellems, R. O., Charlton, C., Kversøy, K. S., Gy˝ori, M., 2020. Exploring the Use of Virtual Characters (Avatars), Live Animation, and Augmented Reality to Teach Social Skills to Individuals with Autism. Multimodal Technologies and Interaction 4, 48. Lord, C., Elsabbagh, M., Baird, G., Veenstra-Vanderweele, J., 2018. Autism spectrum
disorder. Lancet 392, 508–520. National Institute of Mental Health, n.d. Autism Spectrum Disorder [WWW Document]. Autism Spectrum Disorder. URL https://www.nimh.nih.gov/health/ topics/autism-spectrum-disorders-asd (accessed 7.12.24). Nojavanasghari, B., Hughes, C. E., Morency, L.-P., 2017. Exceptionally Social:
Design of an Avatar-Mediated Interactive System for Promoting Social Skills in Children with Autism. In: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. Presented at the CHI ‘17: CHI Conference on Human Factors in Computing Systems, ACM, Denver Colorado USA, pp. 1932–1939. Pino, M. C., Vagnetti, R., Valenti, M., Mazza, M., 2021. Comparing virtual vs real faces expressing emotions in children with autism: An eye-tracking study. Educ Inf Technol 26, 5717–5732. Rahill, K. M., Sebrechts, M. M., 2021. Effects of Avatar player-similarity and player-
construction on gaming performance. Computers in Human Behavior Reports 4, 100131. Rao, P. A., Beidel, D. C., Murray, M. J., 2008. Social Skills Interventions for Children with Asperger’s Syndrome or High-Functioning Autism: A Review and Recommendations. J Autism Dev Disord 38, 353–361. Rylaarsdam, L., Guemez-Gamboa, A., 2019. Genetic Causes and Modifiers of
Autism Spectrum Disorder. Front Cell Neurosci 13, 385. Tebartz van Elst, L., Fangmeier, T., Schaller, U. M., Hennig, O., Kieser, M., Koelkebeck, K., Kuepper, C., Roessner, V., Wildgruber, D., Dziobek, I., 2021. FASTER and SCOTT&EVA trainings for adults with high-functioning autism spectrum disorder (ASD): Study protocol for a randomized controlled trial. Trials 22, 261. Vass, E., Gerlinger, L., Egervári, L., Kilencz, T., Csukly, G., Hermán, L., Réthelyi,
J., Farkas, K., Mariegaard, L., Glenthø, L. B., Simon, L., 2024. Exploring the acceptability, and feasibility of a modified virtual reality-based AVATAR therapy in schizophrenia spectrum disorders: A case series report. Acta Psychologica 250, 104520. Wagner-Hartl, V., Igel, M., Schmid, R., Gleichauf, K., Wunsch, V., Arabian, H., Zimmermann, M., Möller, K., Hartz, P., 2024. Exploring Virtual Reality Avatars: Assessing Basic Emotion Representations for Mental Health Applications. Annual Conference of the German Society for Biomedical Engineering. IFAC IFAC- PapersOnLine 2024, 58–24, 448–453. Wu, S., Xu, L., Dai, Z., Pan, Y., 2023. Factors Affecting Avatar Customization
Behavior in Virtual Environments. Electronics 12, 2286. Zeidan, J., Fombonne, E., Scorah, J., Ibrahim, A., Durkin, M. S., Saxena, S., Yusuf, A., Shih, A., Elsabbagh, M., 2022. Global prevalence of autism: A systematic review update. Autism Research 15, 778–790.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2025): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-36-6
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional