Ilona Buchem, Niklas Baecker (2022). NAO Robot as Scrum Master: Results from a scenario-based study on building rapport with a humanoid robot in hybrid higher education settings. In: Training, Education, and Learning Sciences. AHFE Open Access, vol. 59, pp. 65–73. https://doi.org/10.54941/ahfe1002385
Educational robots have been used as an effective intervention mainly in STEM education. A wide range of educational robots, including programmable toys and DIY electronics, have been used to support computational, engineering and problem solving skills, mainly of school children. However, the application of humanoid robots to support learning, especially in higher education, is still at its initial stage. Recent studies provide promising evidence about the potential of humanoid robots for motivation and learning outcomes in higher education. Humanoid, educational robots, such as the NAO robot, provide a multimodal interface which uses touch, speech, gestures and eye gaze for interaction, which is similar to human interaction in the physical world. Rapport is one of the key factors for a successful interaction with humanoid robots and a modulating factor for learning (Westlund & Breazeal, 2019). Rapport has been defined as a dynamic structure of mutual attentiveness, positivity, and coordination (Tickle-Degnen & Rosenthal, 1990). Rapport with humanoid robots can be established through verbal and nonverbal behaviour including dialogues, gestures and movements (Omokawa et al., 2019; Brown & Howard, 2014).In this paper we describe a scenario-based application of the NAO robot and the results from an online survey with 47 students (58% male, 42% female) who participated in two consecutive exploratory studies in hybrid learning settings (66% participated online, 34% F2F). The primary goal of our research was to evaluate to what extent students established rapport with NAO and to explore whether the quality of the rapport differed depending on the mode of participation (on the campus vs. online). The study was conducted with students in the Agile Project Management course in their third semester of Digital Business (B. Sc.). The robot-led scenario was designed as Daily Scrum, which is a 15-minute meeting held by teams using the Scrum framework. NAO was programmed using Choregraphe and Python to facilitate the Daily Scrum as the Scrum Master, whose task was to help students learn how to keep a daily scrum.Building rapport with NAO was assessed using the rapport scale by Gratch, et al. (2007), which was already applied in studies with robotic embodied agents. This scale builds on research revealing the potential of embodied agents to establish rapport and familiarity with humans through verbal and nonverbal behaviour. Additionally, students’ perceptions of the robot were measured using the Human-Robot Interaction Evaluation Scale (HRIES), which is used to evaluate how humans attribute human characteristics to robots (Spatola, et al., 2021). The results of the study indicate that students perceived NAO as a humanlike, anthropomorphic agent and that this perception positively affected student-robot rapport. Additionally, the paper explores the relationships between the rapport scores and the perception of students of key features of the robot including appearance, voice quality, pauses in speaking, pace of speaking, facial expression, movements of the body, head and hand, light effects and tactile interaction. Based on the findings, we discuss both our research contributions and practical implications for programming of humanoid, educational robots to enhance student-robot rapport as part of HRI.
Educational robots have been used as an effective intervention mainly in STEM education. A wide range of educational robots, including program- mable toys and DIY electronics, have been used to support computational, engineering and problem solving skills, mainly of school children. However,
the application of humanoid robots to support learning, especially in higher education, is still at its initial stage. Recent studies provide promising evidence about the potential of humanoid robots for motivation and learning outco- mes in higher education. Humanoid, educational robots, such as the NAO robot, provide a multimodal interface which uses touch, speech, gestures and eye gaze for interaction similar to human interaction.
In this paper we describe a scenario-based application of the NAO robot and the results from an online survey with 47 students (58% male, 42% female) who participated in two consecutive exploratory studies in hybrid learning settings (66% participated online, 34% F2F). The primary goal of our research was to evaluate to what extent students established rapport with NAO and to explore whether the quality of the rapport differed depending on the mode of participation (on the campus vs. online). The study was con- ducted with students in the Agile Project Management course in their third semester of Digital Business (B. Sc.). The robot-led scenario was designed as Daily Scrum, which is a 15-minute meeting held by teams using the Scrum framework. NAO was programmed using Choregraphe and Python to faci- litate the Daily Scrum as the Scrum Master, whose task was to help students learn how to keep a daily scrum.
Rapport with Humanoid Robots Rapport has been explored in the context of Human Robot Interaction (HRI) in a number of research studies. Rapport was defined as a dynamic structure of mutual attentiveness, positivity, and coordination by Tickle-Degnen and Rosenthal (1990). Rapport has been considered as one of the key factors for a successful interaction with humanoid robots and a factor for learning (Westlund & Breazeal, 2019). Rapport has been defined as a dynamic stru- cture of mutual attentiveness, positivity, and coordination (Tickle-Degnen & Rosenthal, 1990). Rapport with humanoid robots can be established through verbal and nonverbal behavior including dialogues, gestures and movements (Brown & Howard, 2014). The sense of rapport is linked to effective commu- nication, persuasion, liking, trust, interaction and engagement (Gratch et al., 2006). Studies show that rapport can be established in a similar manner betw- een humans and embodied agents, including humanoid robots, through gaze, feedback, facial expressions, gestures, movements and other types of respon- sive behavior which indicates positive emotions, attention and coordination (Gratch et al., 2006). The study by Aroyo et al. (2018) showed that buil- ding rapport with robots results in the disclosure of sensitive information as well as conformity with robot’s suggestions. In their experimental study, Art- stein et al. (2017) investigated the effect of the relational dialogue combined with the functional dialogue between humans and the NAO robot on crea- ting rapport and exerting social influence. NAO was used as an example of a physically embodied agent and compared with a virtually embodied agent designed as a virtual human as a conversational partner (Artstein et al., 2017). The study showed that participants prefered interacting with NAO and felt higher rapport with the robot compared to a virtual human, which seems to
NAO Robot as Scrum Master: Results From a Scenario-Based Study 67
support results from psychological research about physical embodiment as a factor enhancing social influence (Artstein et al., 2017). Lucas et al. (2017) investigated building rapport with NAO through social dialogues, such as ice-breakers, and highlighted the negative influence of errors in robot’s dia- logue for rapport building. Li et al. (2017) indicated that reactive emotions such as surprise, agreement, sympathy, and approval should be expressed by a robot to achieve rapport, and suggested that making humanoid robots emo- tional is needed to make humans feel comfortable. For example, participants tend to nod and smile when a robot expresses human-like reactive emotions (Li et al., 2017). Also the study by Omokawa et al. (2019) showed that phatic dialogues compared to query dialogues cause participants to laugh more as a response. Human reactions such as nodding, smiling or laughing may be an indicator of rapport.
Design of the Daily Scrum Scenario The Daily Scrum scenario was designed as a hybrid (on campus and online), 15-minute event for small groups (max. 5 students) facilitated by the NAO robot. NAO had the role of a Scrum Master and its task was to help students learn how to keep a daily scrum in an agile team. The setting of the scenario was specified as a laboratory or a seminar room at the university, with parti- cipants including NAO, students (both as active daily scrum participants and passive observers), and lecturers (one participating on campus, one online). NAO was placed on a movable pentagon table. Up to five students were standing around the table, having eye-level contact with NAO, which is only 58 cm/1,9 feet high. The daily scrum scenario was designed in English and in German, taking language specific expressions of humor into consideration. The design inclu- ded the programming of NAO as Scrum Master (see below) and the pre- paration of students for a participation in the robot-led daily scrum. The preparation of students included a short video which was sent to students before the experiment to convey the first impression of NAO and possibly reduce fear of interacting with a robot (Hiroi and Ito, 2011). In the video, NAO introduces itself as Scrum Master and invites students to participate in the experiment. The preparation also included the script for each active participant. The script was an example text from a daily scrum interaction. During the daily scrum with NAO, students read their script. The aim of the script was to reduce cognitive workload in a new HRI, hybrid situation (Gittens, 2021).
The design of the daily scrum scenario was pre-tested twice before the first implementation in November 2021, and then tested again after modifications before the second implementation in January 2022. The aim of all pre-tests was to obtain student feedback and test the programming of NAO as Scrum Master.
Programming of NAO as Scrum Master The NAO robot, i. e. NAO Power V6 Educator Pack, was programmed by one student (the second author of this paper) using the Choregraphe software
68 Buchem and Bäcker
Figure 1: The programming of the NAO robot as Scrum Master in choregraphe.
(Version 2.8.6), Python and C++. The aim was to design a highly intera- ctive scenario in which NAO as Scrum Master facilitates a daily scrum. At the same time it was important to take into consideration that noisy conditi- ons in human-computer interaction can be challenging for speech recognition (Heinrich and Wermter, 2011). Our experience shows that since NAO does not perform reliably if a participant speaks too softly, is too far away from the robot or there are too many background noises. Therefore, the implementa- tion of the scenario included not only interaction through dialogues but also through tactile sensors, such as foot bumpers located at the tip of each foot and on the top of the head of NAO. The clear tactile feedback of pressing one of the foot bumpers was combined with dialogues to optimize user experie- nce. Compared to the first experiment, dialogues were revised and/or changed based on student feedback. After changes, NAO also had a clearer pronunci- ation, especially in German, and could recognise more vocabulary. Students could now ask NAO to repeat what has been said. Also, timekeeping has been added to the exercise, i. e. the daily scrum session lasts 15 minutes, while the time per participant depends on the number of participants. To keep track of time, NAO’s eye-LEDs light up in different colors during the exercise, i. e. green, when 10 minutes left, yellow when only 5 minutes left, red when the time is up (see Figure 1).
The programming of the NAO robot as Scrum Master also included the expression of a personality through speech and gestures. The aim was to evoke the feeling that the conversation with NAO is realistic, personal and enjoyable. Studies showed that a humorous personality of a robot tends to stimulate empathy and that phatic dialogues, such as small talk, create a personality effect (Omokawa et al. 2019).
METHODS
Study Design The study took place at Berlin University of Applied Sciences in hybrid set- tings and comprised two experiments, one in November 2021 and one in January 2022. The participation in both experiments was voluntary. Each
NAO Robot as Scrum Master: Results From a Scenario-Based Study 69
student could participate only in one of the two experiments on the campus. Online participation was possible in both experiments. The same survey was administered during both experiments The key three research questions were: How do students perceive NAO as Scrum Master? How effective is NAO for establishing rapport with students? Are there any differences in establishing rapport with NAO in hybrid settings, i. e. students participating on campus and online? In order to collect the data, an online survey was designed and administered during two consecutive experiments with NAO.
Research Instruments and Statistical Analysis The online survey included validated scales and self-designed items. To explore the perception of NAO as Scrum Master, students were asked to assess whether they would recommend using NAO as Scrum Master in the future. General perceptions of NAO were assessed using the Human-Robot Interaction Evaluation Scale (HRIES) by Spatola et al. (2021). HRIES is a composite questionnaire used to evaluate how humans perceive robots and includes four subscales: Sociability, Agency, Animacy, and Disturbance, each comprising a set of four semantic items. The reliability and validity of HRIES were confirmed in a number of studies (Spatola et al., 2021). Building rap- port with NAO was assessed using the rapport scale by Gratch, et al. (2007), which was previously applied in studies with robotic agents. This 15-item scale includes items related to cognitive, affective, and interactional rapport. The analysis of the data was conducted using IBM SPSS Statistics 27. The methods included both descriptive and inferential statistics. The results for the HRIES and rapport scales are presented as mean values and standard deviations. The independent samples t-test with the significance level α=0.05 was used to determine whether rapport differed for students participating on-campus compared to students participating online.
RESULTS
Participants Participants were undergraduate students in the Digital Business (B. Sc.) pro- gram at Berlin University of Applied Sciences enrolled in the course on Agile Project Management (APM) in the 3rd semester of their studies. Altogether 47 students participated in the online survey, i.e. 27 students during the first experiment and 20 students during the second experiment. Among the students who answered the online survey during the first experiment 13 parti- cipated on campus (48.1%) and 14 online (51.9%). Among the students from the second experiment 4 participated on campus (20%) and 16 online (80%). The low number of students participating on campus during the second expe- riment was due to the pandemic situation. In total, 27 out of 47 participants were male (57.4%) and 20 were female (42.5%). Most participants in the first experiment had no previous experience in interaction with humanoid robots (81.5%), while 50% of participants in the second experiment stated they already had this experience. Most participants were 20 to 24 years old (46.5%), followed by students in the age group of 25 to 29 (30%).
70 Buchem and Bäcker
Student Perceptions To answer the first question: How do students perceive NAO as Scrum Master?, students were asked to assess whether they would recommend using NAO as Scrum Master on a scale from 1 “fully disagree” to 7 “fully agree”. 81% of participants stated that they would recommend using NAO as Scrum Master in the future. The assessment of NAO as Scrum Master was compa- red with other possible roles of the robot in the context of agile methodology and higher education. The application of NAO as Scrum Master received the strongest recommendation (M = 4.74) followed by the role of a Student Buddy (M = 4.11), Teacher Assistant (M = 4.09), Agile Coach (M = 3.34), Team Member (M = 3.17) and Student Mentor (M = 3.02).
Human-Robot Interaction Evaluation Scale (HRIES) Furthermore, perceptions of the NAO robot were measured using the Human-Robot Interaction Evaluation Scale (HRIES) and the items were assessed on the scale from 1 “not at all” to 7 “totally”. After the recoding of results for the Disturbance subscale, which expresses negative perceptions, the total score for HRIES was M = 4.529 (Min 3.128; Max 5.851). The highest score was reached for the Sociability subscale with M = 5.011 (Min 4.511; Max 5.596) and the lowest for the Disturbance subscale (before the recoding) with M = 2.367 (Min 2.149; Max 2.872). The item with the high- est score was “likeable” (M = 5.60), followed by “friendly” (M = 5.17) in the Sociability, and “rational” (M = 4.83) in Intentionality subscale (see Figure 2).
Figure 2: Results of the human-robot interaction scale (HRIES), n = 47.
The results show that students perceived the NAO robot as a sociable agent (likeable, friendly, trustworthy and warm) as well as rational (beha- vior is based on the reasons for action) and alive (agent which is able to respond). The comparison of the results for both experiments shows that the highest values in the first experiment were reached for the Animacy subscale
NAO Robot as Scrum Master: Results From a Scenario-Based Study 71
(M 4.07), while the highest values in the second experiment were reached for the Sociability subscale (M 4.85). This means that while students in the first experiment perceived NAO primarily as a living agent with human- like traits, the perceptions in the second experiment focused on the social traits. This shift in the perception may be related to the familiarity effect, which may enhance the acceptance of a technology (Jankowski et al., 2020), and/or it can be due to improved programming of NAO which may have influenced a positive evaluation of the capability of the robot to socially inte- ract with humans. These effects could be explored in more depth in further studies.
Rapport Scale To answer the second question: How effective is NAO for establishing rap- port with students? students were asked to rate the 15 items in the rapport scale from 1 “fully disagree” to 7 “fully agree” The results indicate that the programming of NAO was effective for creating an impression of NAO being likeable (M = 5.59), warm and caring (M 4.59), as well as an interesting (M = 5.36) and exciting (M 4.96) interaction. The results after reversing negative items also show that participants felt that NAO was interested in participants (M = 5.10), involved participants (M = 4.12) and that NAO by all means was not boring (M = 5.80). However, the programming of NAO was less effective in creating a connection (M = 3.11) and a relationship (M 3.15) with the robot. The total score of the scale was M = 4.31 (Min:3.06, Max:5.81). Additionally, a factor analysis with Varimax rotation for the rap- port scale extracted three components which may correspond to the three aspects of rapport described by Gratch et al. (2007), i.e. cognitive, affective and interactional rapport. The measured consistency of the scale was high with α=.890 (Cronbach’s Alpha).
Hybrid Participation To answer the third research question: Are there any differences in establi- shing rapport with NAO in hybrid settings, i. e. students participating on campus and online? three different methods of comparing means were used for the comparison of two groups, i. e. group 1 “online” (n = 30) and group 2 “oncampus” (n = 17). The means were compared both for each of the 15 items in the rapport scale and for the total score of the scale. Since sample sizes in both groups differed, the Levene’s Test was computed. The result was insignificant and the homogeneity could be assumed. To compare the means, the independent sample T-Test, Welch and Brown-Forsythe Test were computed at the 95% confidence level. The comparison of means showed that mean values for all positively formulated items in the rapport scale were higher for the oncampus group, while mean values for all negatively for- mulated items were lower in the oncampus group except for R9 “NAO did not involve participants” and R15 “The interaction with NAO was frustra- ting”. This indicates that oncampus participants experienced the interaction as more frustrating and felt less involved. The feeling of frustration may be
72 Buchem and Bäcker
due to the fact that the interaction with NAO did not run smoothly and stu- dents struggled with the delay in robot response. The feeling of being less involved may be due to the fact that some oncampus students were passive observers and NAO was programmed to interact only with active daily scrum participants. However, the results also revealed that none of these differences were statistically significant (p<.05). This means that both students on the campus and online could establish rapport with the robot. An exception was item R8 “I felt NAO was not interested in participants” (p=.048; M1=3,27; M2=2.24), which similar to R9 may express that the passive observers on the campus wished to be involved more in the interaction and receive more interest from NAO.
CONCLUSION
The primary goal of this study was to evaluate the rapport of students with the humanoid robot NAO in its role of Scrum Master in a daily scrum scena- rio. The results from the online survey show that the programming of NAO was effective for establishing rapport with students. Most participants percei- ved NAO as an agent with social traits (e. g. likeable, friendly, warm and caring). However, establishing a connection or a relationship with NAO was less effective. There were differences in how students perceived NAO during the first and the second experiment and how students established rapport depending on their mode of participation. However, there were no signifi- cant differences in establishing rapport in the online and the oncampus group which means that rapport with NAO could be established in both settings. Further studies should have a more in-depth look into necessary conditions for relationship-building with humanoid robots in the context of (hybrid) education and the effectiveness of different programming options, such as humor, emotions and phatic dialogs for establishing rapport with different groups of users.
REFERENCES
Aroyo, A.M., Rea, F., Sandini, G. and Sciutti, A. (2018), “Trust and Social Engineering in Human Robot Interaction: Will a Robot Make You Disclose Sensitive Information, Conform to Its Recommendations or Gamble?” IEEE Robotics and Automation Letters, 3, 3701–3708. Artstein, R., Traum, D.R., Boberg, J., Gainer, A., Gratch, J., Johnson, E., Leuski,
A. and Nakano, M. (2017), “Listen to My Body: Does Making Friends Help Influence People?” FLAIRS Conference, 430–435. Brown, L. and Howard, A.M. (2014), “Gestural behavioral implementation on a humanoid robotic platform for effective social interaction.” IEEE International Symposium on Robot and Human Interactive Communication, 471–476. Gittens, C.L. (2021), “Remote HRI: a Methodology for Maintaining COVID-
19 Physical Distancing and Human Interaction Requirements in HRI Studies.” Information Systems Frontiers, 1–16. Gratch, J., Okhmatovskaia, A., Lamothe, F., Marsella, S., Morales, M., van der Werf, R. J. and Morency, L.-P. (2006). Virtual Rapport. In: Gratch, J. Young, M., Aylett, R., Ballin, D., Olivier, P. (eds.) Intelligent Virtual Agents, pp. 14–27. Marina Del Rey, CA, USA.
NAO Robot as Scrum Master: Results From a Scenario-Based Study 73
Gratch, J., Wang, N., Gerten, J., Fast, E. and Duffy, R. (2007), “Creating Rapport with Virtual Agents.” In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K. and Pelé, D. (eds.), Intelligent Virtual Agents, Springer, Berlin, Heidelberg, 125–138. Heinrich, S. and Wermter, S. (2011), “Towards Robust Speech Recognition for
Human-Robot Interaction.” IROS2011 Workshop on Cognitive Neuroscience Robotics (CNR), 29–34, San Francisco, CA, USA. Hiroi, Y. and Ito, A. (2011), “Influence of the Size Factor of a Mobile Robot Moving Toward a Human on Subjective Acceptable Distance.” In: Z. Gacovski (ed.), Mobile Robots - Current Trends, IntechOpen, London.. Huang, L., Morency, L. and Gratch, J. (2011), “Virtual Rapport 2.0.” In: Vilh-
jálmsson H. H., Kopp S., Marsella S., Thórisson K.R. (eds.) Intelligent Virtual Agents. IVA. Lecture Notes in Computer Science, 6895, 68–79. Springer, Berlin, Heidelberg. Jankowski, N., Ivanova, E., Wiehe, L. and Wahl, M. (2020), “Long-term changes in technology acceptance of a robotic system in stroke treatment: a pilot study.” Current Directions in Biomedical Engineering, 6(2), 2020–2012. Li, Y., Ishi, C.T., Ward, N.G., Inoue, K., Nakamura, S., Takanashi, K. and Kawahara,
T. (2017), “Emotion recognition by combining prosody and sentiment analysis for expressing reactive emotion by humanoid robot.” Asia-Pacific Signal and Infor- mation Processing Association Annual Summit and Conference (APSIPA ASC), 1356–1359. Lucas, G.M., Boberg, J., Traum, D.R., Artstein, R., Gratch, J., Gainer, A., Johnson, E., Leuski, A. and Nakano, M. (2017), “The Role of Social Dialogue and Errors in Robots.” The 5th International Conference on Human Agent Interaction, 431– 433. Omokawa, R., Kobayashi, M. and Matsuura, S. (2019), “Expressing the Personality
of a Humanoid Robot as a Talking Partner in an Elementary School Classroom. Universal Access in Human-Computer Interaction. Theory, Methods and Tools.” In: M. Antona and C. Stephanidis (eds.) HCII 2019. Lecture Notes in Computer Science, 11572, 494–506, Springer, Cham, Spatola, N., Kühnlenz, B. and Cheng, G. (2021), “Perception and Evaluation in Human–Robot Interaction: The Human–Robot Interaction Evaluation Scale (HRIES)—A Multicomponent Approach of Anthropomorphism.” International Journal of Social Robotics, 1–23. Tickle-Degnen, L. and Rosenthal, R.K. (1990), “The Nature of Rapport and Its
Nonverbal Correlates.” Psychological Inquiry, 1, 285–293. Westlund, J.K. and Breazeal, C. (2019), “A Long-Term Study of Young Children’s Rapport, Social Emulation, and Language Learning With a Peer-Like Robot Playmate in Preschool.” Frontiers in Robotics and AI, 6:81.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Training, Education, and Learning Sciences
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-35-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional