Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

AVATAR: Realistic customized virtual agents as emotional tutors for children with ASD

1 CRECIENDO, Center of Early Stimulation and Alternative Therapies, Cuenca, EC010105, Ecuador ↗
2 Universidad Politécnica Salesiana, Cuenca, EC010105, Ecuador ↗
3 ADACAPIA, Association of Families and Friends Seeking Autism, Training Assistance Programs, and Research, Cuenca, EC010105, Ecuador ↗
4 CRECIENDO, Center of Early Stimulation and Alternative Therapies, Cuenca EC010105, Ecuador ↗
Publication year
Digital Object Identifier10.54941/ahfe1002751

Abstract

This work describes the generation process of a Human-Computer Interaction interface, intended to provide support in therapeutic interventions for children with Autism Spectrum Disorder (ASD). For the interface, realistic avatars were generated based on photogrammetry techniques, based on real human models. Resulting interface employs the "Wizard of Oz" method, whereby avatars become virtual puppets controlled by a remote operator. Using facial recognition, the avatars mimic the gestures and facial movements of human models while the voice is transmitted without any modification. In this way, interaction is completely promoted by the human model, avoiding the loss of attention that could be generated by delays produced by an automatic interaction system based on Artificial Intelligence. For the experimental process, we developed a mobile application that uses generated avatars as emotional trainers based on 6 basic emotions. App works like a serious game that allows users to identify emotions expressed by avatars by interpreting their gestures. Results showed a great acceptance by children with ASD to interact with the application and an excellent perception by therapists who considered this tool as a very useful support during therapeutic processes.

INTRODUCTION

Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that occurs in the early years of childhood, which generates deficiency in social interaction, accompanied by repetitive and stereotyped behaviors (DSM-V, 2013). According to data presented in 2018 by the United States Center for Disease Control and Prevention (CDC), the worldwide prevalence is that it occurs in 1 in 59 children (CDC, 2018). Previous data indicates that the percentage of people diagnosed with ASD shows a tendency to increase, rea- ching almost 2% of the population. This has caused autism to be considered a priority attention problem within public medical entities.

© 2022. Published by AHFE Open Access. All rights reserved. 340

AVATAR: Realistic Customized Virtual Agents as Emotional Tutors 341

The deficiency in social skills of people with ASD prevents them from star- ting or maintaining interaction with peers (Martos, 2012). Among the main social deficits of autism are the difficulty to detect, interpret or predict the emotions, intentions and beliefs of others (Baron-Cohen et al., 1985). The acquisition of these skills at an early age is essential to develop reciprocal affective interactions and ensure adequate social functioning in adult life. Faced with this reality, and taking advantage of the rapid global technolo- gical deployment, various tools have been developed based on Information and Communications Technologies (ICT) and focused on the acquisition or improvement of social skills. Reviewing the literature we find several tech- nological tools designed to support the screening, diagnosis, assistance and social learning of people with ASD (Ahmad et al., 2019; Baldassarri et al., 2020; Liu et al., 2017; Spiel et al., 2017). Although there are several techno- logies types that can be used in the development of therapeutic supports, one that has received special attention is Virtual Reality (VR). VR allows the deve- lopment of human-computer interfaces and intelligent environments, which emulate real situations, controlling the main variables that could generate social anxiety in people with ASD (Bernardes et al., 2015; Bond et al., 2021; Dechsling et al., 2021).

Within VR we find avatars, which are virtual representations of people and have proven to be useful tools for overcoming communication barriers in people with ASD (Guerrero-Vasquez et al., 2019, 2018; Harrop et al., 2019; Milne et al., 2010; Tanaka et al., 2016). Several authors have demonstra- ted the effectiveness when working with avatars as emotional tutors, used in teaching, recognition and self-regulation of feelings. (Chen et al., 2019; Her- ring et al., 2017; Karnes and Grünke, 2021). In addition, avatars are easy to develop and allow modifications with less difficulty than other types of tech- nology. These characteristics provide the possibility of generating resources with a certain level of customization according to the user, taking into account that autism is characterized by the heterogeneity of people, even when they have similar diagnoses.

In this context, this paper presents the emotional tutor development based on realistic avatars generated from a photogrammetric reconstruction of real human models. Based on the importance of users feeling identified with ava- tars (Guerrero-Vasquez et al., 2020), our methodology allows us to generate models based on real people belonging to the users environment and with appropriate anthropomorphic features.

REALISTIC AVATARS GENERATION Avatars generation was carried out in two stages, the first consisted of the photogrammetric reconstruction and the second in the adaptation as an animation object. Each of the stages is detailed below.

3D Photogrammetric Reconstruction Photogrammetry is three-dimensional reconstruction of an object from mea- surements made on photographic images. When taking pictures, it is impor- tant that the environment has correct lighting without creating shadows.

Figure 1: 3D model photogrammetric reconstruction process.
Original PDF Figure 1:

Shadows could complicate the model reconstruction or its texture. For a good reconstruction, a minimum of 60 photographs are required, taken around the model and with different angles, trying to capture all profiles. It is recom- mended to place the camera at an average distance between 20 and 30 cm. If we capture more photographs, the model detail increases, but computational cost to process information also increases.

The 3D model reconstruction process is shown in Figure 1. There are seve- ral software options, both free and licensed, which allow the process to be carried out. It is important to maintain quality in each step, so the result can be used as an interaction interface. When generating the point cloud, it is likely that there are several elements with additional information that should be eliminated, leaving only the points that make up the model. It is important that the model does not have additional floating elements, so as not to gene- rate distraction for users. The final model must be exported with the mesh and texture to be used within the animation software.

3D Model Adaptation and Animation Once the 3D model is available, it is necessary to make some adaptations that make it useful for animations, whether preset or in real time. For this process there are software options that could be used, in our case we use Blender, which in addition to being open source, allows interaction with Python.

Figure 2 shows a summary of the process developed to polish the 3D model. The result is the animated avatar with facial expressions and vocal movements.

AVATAR: Realistic Customized Virtual Agents as Emotional Tutors 343

Figure 2: Adaptation process in blender.
Original PDF Figure 2:

Facial Expressions Generation With the generated avatar it is possible to develop the animation basically in two different ways. On the one hand, the first way is to use facial recognition to detect the movements of a person and transfer them to the avatar. This methodology is known as the “Wizard of Oz” and can be used for real-time interaction. In Figure 3, some examples of animations made following the human model expressions are shown.

On the other hand, the second methodology consists of developing preset animations as short videos with basic emotional states. This second methodo- logy was evaluated in this work with a group of children with ASD, obtaining promising results.

In Figure 3, the animations used for avatar evaluation are shown. For this, a mobile application was developed with two modes of use; the first is a learning mode where the user is guided to learn the emotions and the second is a game mode to test the user’s ability to identify each emotion.

RESULTS

Avatars facial expressions were evaluated with two experimental groups: one with Typical Development (TD) children (TD = 14 children, 3 girls, 11 boys)

Figure 3: Adaptation process in blender.
Original PDF Figure 3:
Figure 4: Basic emotions generated with avatar a) angry b) happy c) sad d) afraid e) surprised f) confused.
Original PDF Figure 4:

and another with children with ASD (ASD = 15 children, 15 boys). The evaluation consisted of a 14-question survey answered by an observer who was present during the interactions, rated on a Likert scale from 1 to 5, with 5 being the most positive response. Table 1 shows statistical values of mean, mode and variance of the respon- ses generated in survey in both experimental groups. Taking into account the results of TD group as reference values, it is possible to analyze the responses obtained from ASD group. It is observed that the level of interest is lower in children of the group with ASD, in the same way the perception is that

AVATAR: Realistic Customized Virtual Agents as Emotional Tutors 345

Table 1. Variables in observer interview.

Table available in the original PDF paper.

Variables TD Group ASD Group Media Mode Var. Media Mode Var.

Initial interest 4.71 5 0.35 4 4 0.53 Interaction interest 4.86 5 0.12 3.87 4 0.78 Complexity 4.7 5 0.3 3.9 5 1 Recognition level 3.8 3 1 3.3 3 0.7 Realism 3.8 4 0.7 3.7 3 0.7 Comfort 4.93 5 0.07 4.47 5 0.92 Emotional learning 4.57 5 0.24 3.73 4 0.46 Physical appearance 4.4 5 0.7 3.9 4 0.3 Response time 4.1 5 0.7 3.5 3 0.5 Avatars in interventions 4.9 5 0.1 4.7 5 0.2 Insertion of avatars 5 5 0 4.9 5 0.1 Being an avatar model 4.1 4 0.5 4.5 5 0.2 Family members as avatar 4.36 5 1.09 4.47 4 0.25 Use at home 5 5 0 4.13 4 0.25

interaction with this group is more complex. Variables that show the greatest difference are: emotional learning, physical appearance, and response time; however, they are not significantly different values.

It is noteworthy that the TD group evaluators give greater importance to becoming models of new avatars and also consider it valuable that family members are the new avatars.

CONCLUSION

Avatars use as emotional tutors for people with ASD has been widely tested and its efficiency confirmed in several previous studies. However, realistic avatars are a more accurate approximation of reality because their expressi- ons are more authentic without the exaggeration that other cartoon avatars often exhibit. We can consider emotional coaches based on realistic avatars, as a more complex level of emotional identification. In addition, conside- ring that the objective is to provide training for real life, it is appropriate to approximate therapeutic resources, at least in appearance, to real people.

The possibility of recreating human models of user’s family or therapeutic circle could represent an advantage when overcoming communication bar- riers. Although people with ASD show a great openness to interact with technological resources, seeing a familiar face could improve the confidence level and the engagement to interact by answering questions and following directions.

One of the common mistakes that is usually made when developing sup- ports for ASD intervention, is the tools generalization. It is necessary to remember the heterogeneity of this disorder and seek the therapeutic inte- rventions personalization, with the aim of achieving greater effectiveness.

With our work we have shown that the generation of avatars from photo- graphs of real people, offers the possibility of quickly expanding the number of available resources, increasing the options when deciding on a model.

REFERENCES

Ahmad, J.I., Yusof, S.M., Talib, N.H.A., 2019. Multimedia learning tools for Autism children, in: Proceedings of the Third International Conference on Computing, Mathematics and Statistics (ICMS2017). Springer, pp. 579–584. Baldassarri, S., Passerino, L., Ramis, S., Riquelme, I., Perales, F.J., 2020. Tow-

ard emotional interactive videogames for children with autism spectrum disorder. Univ Access Inf Soc. https://doi.org/10.1007/s10209-020-00725-8 Baron-Cohen, S., Leslie, A.M., Frith, U., 1985. Does the autistic child have a “theory of mind”? Cognition 21, 37–46. Bernardes, M., Barros, F., Simoes, M., Castelo-Branco, M., 2015. A serious game

with virtual reality for travel training with Autism Spectrum Disorder. IEEE, pp. 127–128. https://doi.org/10.1109/ICVR.2015.7358609 Bond, M., Powers, D.M.W., Raghavendra, P., 2021. Teaching skills with virtual humans: Lessons from the development of the thinking head whiteboard, Cogn. Sci. Technol., Cognitive Science and Technology. Springer. CDC, 2018. Community report on autism 2018. Autism and Developmental Disa-

bilities Monitoring Network. Retrieved from Centers for Disease Control and Prevention. Chen, F., Wang, L., Peng, G., Yan, N., Pan, X., 2019. Development and evaluation of a 3-D virtual pronunciation tutor for children with autism spectrum disorders. PLoS ONE 14, e0210858. https://doi.org/10.1371/journal.pone.0210858 Dechsling, A., Shic, F., Zhang, D., Marschik, P.B., Esposito, G., Orm, S.,

Sütterlin, S., Kalandadze, T., Øien, R.A., Nordahl-Hansen, A., 2021. Virtual reality and naturalistic developmental behavioral interventions for children with autism spectrum disorder. Research in Developmental Disabilities 111, 103885. https://doi.org/10.1016/j.ridd.2021.103885 Diagnostic and statistical manual of mental disorders (DSM-V), 2013. . American Psychiatric Association Washington DC, Washington DC. Guerrero-Vasquez, L.F., Chasi-Pesantez, P.A., Castro-Serrano, R., Robles-Bykbaev, V.E., Bravo-Torres, J.F., Lopez-Nores, M., 2019. AVATAR: Implementation of a Human-Computer Interface Based on an Intelligent Vir- tual Agent, in: 2019 IEEE Colombian Conference on Communications and Computing (COLCOM). Presented at the 2019 IEEE Colombian Conference on Communications and Computing (COLCOM), IEEE, Barranquilla, Colombia, pp. 1–5. https://doi.org/10.1109/ColComCon.2019.8809170 Guerrero-Vasquez, L.F., Landy-Rivera, D.X., Bravo-Torres, J.F., Lopez-Nores, M.,

Castro-Serrano, R., Vintimilla-Tapia, P.E., 2018. AVATAR: Contribution to Human-Computer interaction processes through the adaptation of semi- personalized virtual agents, in: 2018 IEEE Biennial Congress of Argentina (ARGENCON). Presented at the 2018 IEEE Biennial Congress of Argen- tina (ARGENCON), IEEE, San Miguel de Tucumán, Argentina, pp. 1–4. https://doi.org/10.1109/ARGENCON.2018.8646055 Guerrero-Vasquez, L.F., Robles-Bykbaev, V.E., Lopez-Nores, M., Bravo-Torres, J.F., Jara-Quito, H.J., Tapia-Vintimilla, M.A., Pazos-Arias, J.J., Gil-Solla, A., 2020. Assessing Children’s Perceptions of Live Interactions With Avatars: Prepara- tions for Use in ASD Therapy in a Multi-Ethnic Context. IEEE Access 8, 168456–168469. https://doi.org/10.1109/ACCESS.2020.3023737

AVATAR: Realistic Customized Virtual Agents as Emotional Tutors 347

Harrop, C., Amsbary, J., Towner-Wright, S., Reichow, B., Boyd, B.A., 2019. That’s what I like: The use of circumscribed interests within interventions for individuals with autism spectrum disorder. A systematic review. Research in Autism Spectrum Disorders 57, 63–86. https://doi.org/10.1016/j.rasd.2018.09.008 Herring, P., Kear, K., Sheehy, K., Jones, R., 2017. A virtual tutor for children with

autism. Journal of Enabling Technologies 11, 19–27. https://doi.org/10.1108/JET- 01-2016-0006 Karnes, J., Grünke, M., 2021. The Effects of a Multicomponent Motivational System Intervention Using Peer-Tutoring for Implementation on the Automation of Single- Digit Addition Tasks of Four Struggling Elementary Students. Education Sciences 11. https://doi.org/10.3390/educsci11060265 Liu, X., Wu, Q., Zhao, W., Luo, X., 2017. Technology-Facilitated Diagnosis

and Treatment of Individuals with Autism Spectrum Disorder: An Engineering Perspective. Applied Sciences 7, 1051. https://doi.org/10.3390/app7101051 Martos, J., 2012. Autismo y trastornos del espectro autista. Definición, génesis del trastorno y explicaciones psicológicas. Valdez, D. & Rugeri, L.(Comps), Autismo del diagnóstico al tratamiento.(cap. I pp 19-45) Buenos Aires Valdez, D. & Rugeri, L. Comps. Ed Paidós. Milne, M., Luerssen, M.H., Lewis, T.W., Leibbrandt, R.E., Powers, D.M.W., 2010.

Development of a virtual agent based social tutor for children with autism spe- ctrum disorders. IEEE, pp. 1–9. https://doi.org/10.1109/IJCNN.2010.5596584 Spiel, K., Frauenberger, C., Fitzpatrick, G., 2017. Experiences of autistic children with technologies. International Journal of Child-Computer Interaction 11, 50–61. https://doi.org/10.1016/j.ijcci.2016.10.007 Tanaka, H., Sakriani, S., Neubig, G., Toda, T., Negoro, H., Iwasaka, H.,

Nakamura, S., 2016. Teaching Social Communication Skills Through Human-Agent Interaction. ACM Trans. Interact. Intell. Syst. 6, 1–26. https://doi.org/10.1145/2937757

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET 2022): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET 2022): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-44-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →