Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Social Empathetic Cognitive Robotics for Autism (SECRA): a preliminary study

Department of Human Sciences, Link Campus University, Via del Casale S. Pio V 44, Rome, Italy ↗
Publication year
Digital Object Identifier10.54941/ahfe1005709

Abstract

In the last two decades, Socially Assistive Robotics (SAR) has emerged as a promising approach in treating Autism Spectrum Disorders (ASD). SAR involves using social robots to provide assistance in social interaction settings. Although much research in this field is still preliminary, SAR has shown significant potential for achieving effective outcomes in ASD treatment. Despite these promising results, there are still unanswered questions about the effectiveness of SAR for ASD, especially regarding how social robots should be designed to optimize the complex interactions among therapists, children, and robots. The primary aim of the present project is to address these limitations through a large-scale, randomized controlled trial that can provide clear answers to the above questions. The project has two main objectives: (a) to develop robust psychosocial protocols for robot-assisted therapy tailored for children with ASD, and (b) to evaluate whether the QTrobot (LuxAI), along with eye-tracking technology, can improve cognitive and socio-emotional skills in these children in various environments. The project will be implemented in two phases. In the first phase, psychosocial protocols will be developed and tested preliminarily to refine their effectiveness. Based on the results of Phase 1, a rigorous randomized controlled study will conduct in the second phase. Currently, the project is at first phase. We are conducting the preliminary study to develop the psychosocial protocol and to understand what factors can facilitate the interaction human-robot. In this case, children with ASD and the QTrobot. This project is funded by the European Union-Next Generation EU.

INTRODUCTION

Autism Spectrum Disorder (ASD) is a complex neurobiological condition characterized by significant difficulties in communication and social interaction, often accompanied by repetitive and stereotyped behaviors (American Psychiatric Association, 2013). Children with ASD face notable challenges in both verbal and non-verbal communication, such as recognizing facial emotions and processing social and emotional information (Hage et al., 2021; Lord et al., 2018; Nagy et al., 2021). These challenges underscore the importance of developing effective therapeutic interventions to improve socio-emotional abilities of these children, and consequently, their quality of life.

© 2024. Published by AHFE Open Access. All rights reserved. 1416

Social Empathetic Cognitive Robotics for Autism (SECRA): A Preliminary study 1417

Over the past two decades, the field of ASD therapy has seen growing interest in innovative interventions, including socially assistive robotics (SAR), which has emerged as one of the most promising areas (Holeva et al., 2024; Vagnetti et al., 2024; Yun et al., 2017). SAR employs social robots to promote the development of socio-emotional skills through structured and controlled interactions (Scassellati et al., 2018). Research has shown that children with ASD are particularly engaged by robots (Alabdulkareem et al., 2022; Pennisi et al., 2016; Raptopoulou et al., 2021). Using robots as mediators in communication between the child and therapist can enhance engagement and improve clinical outcomes in terms of attention, involvement, imitation, social, and communication skills (Damianidou et al., 2020; Soares et al., 2019; Yuan et al., 2021). However, despite the promising results of SAR interventions, the field faces several limitations. Current evidence is based on studies with small sample sizes, limited statistical power, and variable methodologies (Duradoni et al., 2021; Ismail et al., 2019). Follow-up assessments to evaluate the long- term impact of these interventions are often lacking (Salimi et al., 2021). Additionally, the robots used in these studies vary widely in technological features such as mobility, availability, safety, and acceptability (Puglisi et al., 2022). There remain unanswered questions about the effectiveness of SAR for ASD, particularly regarding how social robots should be designed to optimize the complex interactions between therapists, children, and robots.

In this paper, we present a preliminary study in progress that is part of a larger research project, called “Social Empathetic Cognitive Robotics for Autism” (SECRA). This project has two main objectives: (a) to develop robust psychosocial protocols for robot-assisted therapy tailored for children with ASD, and (b) to evaluate whether a robot, along with eye-tracking technology, can improve cognitive and socio-emotional skills in these children across different settings: therapy labs, schools, and home environments. This project will be implemented in two phases. The first phase will involve the development and preliminary testing of psychosocial protocols to refine their effectiveness. Based on the results of the first phase, the second phase of the project will consist of a rigorous randomized controlled trial.

Currently, the project is at first phase, we are conducting the preliminary study to develop the psychosocial protocol and to understand what factors can facilitate the interaction human-robot. In this case, children with ASD and a new social robot, called the QTrobot (LuxAI S.A).The main aims of this preliminary study are two: a) to develop a Cognitive Behavioural Therapy (CBT) protocol for ASD focused on the comprehension and recognition of emotions, their properly attribution to the contexts and the learning of a basic emotional vocabulary; b) to examine the feasibility and clinical usability of this CBT protocol in the context of a social and cognitive robot-assisted treatment for children with ASD.

1418 Caprì and Picciotto

METHODS

Study Preparation of Protocol In order to design a valid and effective psychosocial protocol, the following three preparatory steps were carried out. Firstly, a comprehensive literature review was conducted to determine theoretically derived implementation components that were likely to be relevant to the protocol (Kouroupa et al., 2022; Papakostas et al., 2021; Vagnetti et al., 2024). Secondly, based on the results of literature review, a multidisciplinary group of professionals with extensive experience in the field of ASD, composed of a neuropsychiatric, a clinical psychologist and an engineer, examined and adapted for QTrobot the CBT protocol used in previous studies obtaining positive and significant results (Marino et al., 2020). Third, the protocol design was completed and was piloted in a preliminary study examining its feasibility and clinical usability.

Protocol Design The protocol was created based on CBT, the principles of Theory of Mind and Applied Behavior Analysis (ABA). It was specifically designed to be implemented with the assistance of the QTrobot and to develop the recognition, the labelling of five basic emotions and their properly attribution to the contexts in children with ASD. The CBT protocol was divided into 12 sessions of 50 minutes each, administered once a week (1 for the pre- assessment, 10 for the treatment and 1 for the post-assessment). This protocol, mediated by the QTrobot, was structured into three main phases including four specific activities related to five basic emotions (happiness, sadness, anger, fear, and disgust).

The first phase aimed to acquire the ability to recognize and label basic emotions; for each basic emotion, the robot described a situation, and the child was asked to label correct emotion, i.e. the robot said: “I’m in a good mood. What’s my emotion?”.

The second phase aimed to teach the context-emotion association, for example the robot presented the following scenario: “Hi, I’m Andrea. Today, it is my birthday. I’m at my party with my friends. How do I feel?”. Thus, the child was stimulated to attribute the proper emotion associated with the presented scenario.

The third phase aimed to teach the discrimination between thoughts and emotions; again, the robot described a situation, and the child was asked to develop the proper thought associated with the presented situation: i.e. “Hi, I’m Marco. I’m in my bedroom with sore throat and cold. What do I think?”.

As stated above, during each session, the QTrobot presented the activity to the child, and it interacted with him or her providing verbal cues, prompts, and reinforcements. If a response of the child was incorrect, the robot provided prompts; if correct, it randomly provided a verbal positive reinforcement with joyful facial expression and arm movements or a positive verbal expression without associated emotional facial expression and arm movements, i.e., QTrobot said: “Congratulations, correct answer (with or without joyful facial expression and arm movements).

Social Empathetic Cognitive Robotics for Autism (SECRA): A Preliminary study 1419

The protocol intervention was conducted by a qualified clinical psychologist using the QTrobot. The psychologist controlled the robot through the application from the table. This application allowed the psychologist to choose the activity introduced by the QTrobot, and based on the child’s responses, the psychologist gave “prompts” or “reinforcements” through the robot. For each phase of the protocol, different prompts were designed and the QTrobot can randomly provide them. A summary of the protocol is reported in Table 1.

Table 1. Protocol summary.

Table available in the original PDF paper.

Phases Activity Expected Outcome Prompt ---- ---- ---- Pre- assessment

Labelling correct basic emotion

First Games on emotion recognition

Attributing the proper emotion associated with a given scenario of daily life

I’m going to give you another help. I’m in … (prompt changes in accordance with basic emotion) Second Games on context- emotion association

Attributing the proper thought associated with a given socio-emotional situation

I’m going to give you another help. I’m in … (prompt changes in accordance with basic emotion) Third Games on discrimination between thoughts and emotions ---- ---- ----

I’m going to give you another help. I’m in … (prompt changes in accordance with basic emotion) Post- assessment

Preliminary Study This study is currently in progress. Until now, five children were recruited by the Institute for Biomedical Research and Innovation (IRIB) of Messina and National Research Council of Italy (CNR). The inclusion criteria for the sample were: age between 5 and 10 years; a diagnosis of ASD according to Manual of Mental Disorders, Fifth Edition (DSM-5); ADOS Level: Module 2, Griffiths: QS>75; absence of aggressive behaviors or severe oppositional tendency; no auditory, visual, or physical impairments; not being on psychiatric therapy; and not undergoing any other emotional and/or social skills treatment during the study. All the parents signed a written consent form for the participation of their child at this study.

Implementation The robot-mediated protocol was administered in a quiet room of IRIB- CNR. The intervention setting involved the use of the QTrobot, an expressive

1420 Caprì and Picciotto

humanoid robot measuring 63 cm in height and weighing 5 kg, specifically designed and built to assist therapists in teaching new skills (cognitive, social, communicative, and emotional) to children with ASD. Produced by LuxAI S.A., the QTrobot is equipped with 12 degrees of freedom that allow for high mobility in the neck and hands. The robot features (a) a facial display to emulate basic emotional expressions, (b) a 3D Intel RealSense camera for facial and gesture recognition in space, and (c) microphones for sound orientation and speakers that enable the robot to produce verbal communications or play sounds (Figure 1). An internal Raspberry PI (QTPI) board controls motors, displays and sensors, all connected to a Linux PC (QTPC), which uses ROS to send commands to the Raspberry board. QTPC and Raspberry board make up the robot connected to each other via an internal LAN, allowing easy configuration and programming which can be directly sent (via Web) to company manufacture for information exchange.

Figure 1: QTrobot (LuxAI).
Original PDF Figure 1:

This tool provides an opportunity to translate robot-assisted therapy on the Internet of Things Data (IoT) domain. Programming can be done using the web app interface provided by the manufacturer (LuxAI), which offers an intuitive block-type utility. This allows routines to be created and executed on the robot using the Android tablets that come with the robot. For customizing specific behaviors, QTrobot allows the use of the RealSense software. This software has been installed into the robot and allows it to recognize gestures or faces through the assignment of key point data in space. It is possible to write specific commands using Python and C++ that invokes the APIs already installed in the robot’s QTPC. QTrobot provides a visual programming interface for non-experts to control the robot by an Android application from tablets and smart phones. During the intervention, the QTrobot acts as a co-therapist, providing visual, motor, and communicative prompts in a partially autonomous manner. It automatically records the children’s responses through an online platform designed to collect quantitative and graphical data.

Preliminary Data We present descriptive data related to the number of correct responses of all children for four sessions of the protocol. From descriptive statistics (Table 2), we observed an increasing trend of all participants for each session,

Social Empathetic Cognitive Robotics for Autism (SECRA): A Preliminary study 1421

from minimum values (0-5) to maximum values (2-7) and mean values (1.00- 6.00). As shown in figure 2, we also observed an increasement of mean of correct responses from session 1 to session 4. These descriptive data suggested a positive impact of the CBT protocol in our sample of children with ASD.

Table 2. Descriptive statistics.

Table available in the original PDF paper.

Session Minimum Maximum Mean Standard Deviation

1 0 2 1.00 .707 2 2 3 2.40 .548 3 4 5 4.40 .548 4 5 7 6.00 .707

Figure 2: Means of correct response of participants in each session.
Original PDF Figure 2:

Moreover, during sessions we observed that when the feedback provided of the QTrobot was the presentation of a facial emotional expression through its screen with arm movements, children looked the face of the robot. Whereas, when the feedback of the robot was only a verbal expression, children did show no reactions. This indicated that the displays of the QTrobot captured the attention of children, and it was more attractive than a verbal prompt. In terms of the feasibility and clinical usability, these preliminary data suggested that the CBT protocol mediated by the QTrobot can be considered a suitable protocol intervention for ASD in field of socio-emotional understanding.

CONCLUSION

In this paper, we have presented preliminary results suggesting that the CBT protocol implemented so far was functioning well. However, these data are descriptive and represent a first step of the study. Statistical analysis will be carried out when a larger sample will complete all the phases of the protocol intervention. Moreover, since this study is in progress, further steps are required. Precisely, it is needed to expand the sample, to complete all planned activities of the protocol, and to introduce the second phase of the SECRA project that will involve the conducting of a rigorous randomized controlled study.

1422 Caprì and Picciotto

From these preliminary results, in terms of human-robot interaction, we observed that social robots are attractive and motivating for children with ASD in the therapy context, particularly the robot used in this preliminary study has a strength respect to other robots used in previous studies on ASD. The strength is a specific technical feature, the display positioned on the face of the QTrobot allows the presentation of animated faces and emotional facial expressions combined with arm movements and voice; in this study we observed that these aspects captured the attention of children with ASD respect to only verbal reinforcements. However, we also noted that the QTrobot has a limitation in terms of child–robot interaction, because in some activities the response of the child was necessarily mediated by a digital tablet for children that could create an overstimulation for the child (Puglisi et al., 2022). Hence, in the next steps of this research project, an object will be to address this limitation and to maximize strengths of the CBT protocol mediated by QTrobot. In the second phase of the SECRA project, a large sample of children with ASD will be recruited and divided into three groups: a robot group receiving therapy with the QTrobot, a traditional treatment group undergoing conventional methods, and a waiting list group that will receive therapy later. An ABAA experimental research design will be used to evaluate the effects of the robot-mediated intervention in the three settings (lab, school, and home).

ACKNOWLEDGMENT

The authors would like to acknowledge the teams of Institute for Biomedical Research and Innovation (IRIB), National Research Council of Italy (CNR).

FUNDING The SECRA project (20229R3JMA CUP:B53D23014420008) is funded by the PRIN2022 European Union-Next Generation EU.

REFERENCES

Alabdulkareem, Amal. Alhakbani, Noura and Al-Nafjan, Abeer. (2022). A systematic review of research on robot-assisted therapy for children with autism, SENSORS (BASEL) Volume 22, No. 3. American Psychiatric Association. (2013). Diagnostic and statistical manual of

mental disorders. 5th edn. Arlington, VA: American Psychiatric Association. Damianidou, Despoina. Eidels, Ami and Arthur-Kelly, Michael. (2020). The use of robots in social communications and interactions for individuals with ASD: A systematic review, ADVANCES IN NEURODEVELOPMENTAL DISORDERS, Volume 4. Duradoni, Mirko. Colombini, Giulia. Russo, Paola Andrea and Guazzini, Andrea.

(2021). Robotic psychology: A PRISMA systematic review on social robot-based interventions in psychological domains, JOURNAL MDPI Volume 4, No. 4. Hage, Simone Vasconcelos Rocha. Sawasaki, Lidiane Yumi. Hyter, Yvette and Fernandes, Fernanda Dreux Miranda. (2021). Social communication and pragmatic skills of children with autism spectrum disorder and developmental language disorder, CODAS Volume 32, No. 2.

Social Empathetic Cognitive Robotics for Autism (SECRA): A Preliminary study 1423

Holeva, Vasiliki. Nikopoulou, Vasiliki-Aliki. Lytridis, Chris et al., (2024). Effectiveness of a robot-assisted psychological intervention for children with autism spectrum disorder, JOURNAL OF AUTISM AND DEVELOPMENTAL DISORDER Volume 54. Ismail, Luthffi Idzhar. Verhoeven, Thibault. Dambre, Joni and Wyffels, Francis.

(2019). Leveraging robotics research for children with autism: A review, INTERNATIONAL JOURNAL OF SOCIAL ROBOTICS Volume 11, No. 3. Kouroupa, Athanasia. Laws, Keith. Irvine, Karen. et al. (2022). The use of social robots with children and young people on the autism spectrum: A systematic review and meta-analysis, PLOS ONE, Volume 17, No. 6. Lord, Catherine. Elsabbagh, Mayada. Baird, Gillian and Veenstra-Vanderweele,

Jeremy. (2018). Autism spectrum disorder, THE LANCET Volume 392, No. 10146. Marino, Flavia. Chilà, Paola. Sfrazzetto Trusso, et al. (2020). Outcomes of a robotassisted social-emotional understanding intervention for young children with autism spectrum disorders, JOURNAL OF AUTISM AND DEVELOPMENTAL DISORDERS Volume 50, No. 6. Nagy, Emese. Prentice, Louise and Wakeling, Tess. (2021). Atypical facial emotion

recognition in children with autism spectrum disorders: Exploratory analysis on the role of task demands, PERCEPTION Volume 50, No. 9. Papakostas, George. Sidiropoulos, George. Papadopoulou, Cristina et al. (2021).

Social robots in special education: A systematic review, ELECTRONICS Volume 10, No. 12. Pennisi, Paola. Tonacci, Alessandro. Tartarisco, Gennaro. Billeci, Lucia. Ruta, Liliana. Gangemi, Sebastiano and Pioggia, Giovanni. (2016). Autism and social robotics: A systematic review, AUTISM RESEARCH Volume 9 No. 2. Puglisi, Alfio. Caprì, Tindara. Pignolo, Loris. et al. (2022). Social humanoid robots

for children with autism spectrum disorders: A review of modalities, indications, and pitfalls, CHILDREN (BASEL) Volume 9, No. 7. Raptopoulou, Anastasia. Komnidis, Antonio. Bamidis, Panagiotis and Astaras, Alexandros. (2021). Human-robot interaction for social skill development in children with ASD: A literature review, HEALTHCARE TECHNOLOGY LETTERS Volume 8, No. 4. Salimi, Zohreh. Jenabi, Ensiyeh and Bashirian, Saeid. (2021). Are social robots ready

yet to be used in care and therapy of autism spectrum disorder: A systematic review of randomized controlled trials, NEUROSCIENCE & BIOBEHAVIORAL REVIEWS Volume 129. Scassellati, Brian. Boccanfuso, Laura. Huang, Chien-Ming. Mademtzi, Marilena.

Qin, Meiying. Salomons, Nicole. Ventola, Pamela and Shic, Frederick. (2018). Improving social skills in children with ASD using a long-term, in-home social robot, SCIENCE ROBOTICS Volume 3, No. 21. Soares, Filomena. Costa, Sandra. Santos, Cristina. Pereira, Ana Paula. Hiolle, Antoine R. and Silva, Vinícius. (2019). Socio-emotional development in high functioning children with autism spectrum disorders using a humanoid robot, INTERACTION STUDIES Volume 20, No. 2. Vagnetti, Roberto. Di Nuovo, Alessandro. Mazza, Monica and Valenti, Marco.

(2024). Social robots: A promising tool to support people with autism. A systematic review of recent research and critical analysis from the clinical perspective, REVIEW JOURNAL OF AUTISM AND DEVELOPMENTAL DISORDERS.

1424 Caprì and Picciotto

Yuan, Fengpei. Klavon, Elizabeth. Liu, Ziming. Lopez, Ruth Palan and Zhao, Xiaopeng. (2021). A systematic review of robotic rehabilitation for cognitive training, FRONTIERS IN ROBOTICS AND AI Volume 8. Yun, Sang-Seok. Choi, JongSuk. Park, Sung-Kee. Bong, Gui-Young and Yoo,

HeeJeong. (2017), Social skills training for children with autism spectrum disorder using a robotic behavioral intervention system, AUTISM RESEARCH Volume 10, No. 7.

Cover of the proceedings volume Human Factors in Design, Engineering, and Computing

Explore the complete digital volume

Human Factors in Design, Engineering, and Computing

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-35-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →