Jonas Birkle, Verena Wagner-hartl (2024). Requirements for Virtual Reality-based Trainings of Human-Robot Interaction. In: Cognitive Computing and Internet of Things. AHFE Open Access, vol. 124, pp. 64–74. https://doi.org/10.54941/ahfe1004704
Nowadays, the use of robots has grown into a standard in industry. In this context, the interaction of humans and robots is intended to combine the relevant capabilities in order to achieve the highest possible efficiencies. This points to the need for an appropriate training to reduce fears and increase trust and acceptance. Virtual realities (VR) can be a helpful platform for such trainings. The aim of the study was to examine subjective impressions and suggestions for implementing a VR training in the industrial context. A simple interaction with an industrial robot, conducted in the three scenarios "reality", "virtual reality" and "hybrid reality", was used. The interviews revealed a large pool of information and concrete suggestions for the implementation of VR training. The information obtained provides a useful basis for designing different training scenarios.
The use of robots has grown into a standard in industry. Following the results of a market research (Statista, 2023), a steady global growth is expected for robotics. The IFR (International Federation of Robotics, 2024) reported that “(...) the world hit a new record of 3.9 million operational robots in 2022.” They stated that the top nations regarding robot density were Korea, Singa- pore and Germany, followed by Japan and China. Especially in industry, the interaction of humans and robots is intended to combine relevant capabilities in order to achieve higher efficiency and productivity (Robla-Gómez et al., 2017). Furthermore, in production, robots are not only seen as “tools” any- more but received a new status as team-members which are for example able to expand the capacities and work abilities of the human members with their strengths (Weber et al., 2018). Even in the private sector, interaction with robots is already common in many places (e.g. Broadbent, 2017).
Acceptance, trust, and perceived emotions regarding robots vary widely depending on the respective context. Following Hancock et al. (2011), espe- cially trust in the robot during a direct collaboration seems to be a crucial factor. Furthermore, it was shown by Wagner-Hartl et al. (2020, 2022) that the acceptance of robots differs in relation to the task that should be performed together with the robot, as well as the task complexity of the performed task. Additionally, Nomura et al. (2008) reported that fear in
Requirements for Virtual Reality-Based Trainings of Human-Robot Interaction 65
human-robot interaction can have a decisive influence on the interaction itself. For example, it can result in an avoidance of touching the robot. Also, indications regarding an influence of the robot’s movement speed on the emo- tional reaction of the users were shown (Birkle et al., 2022; Wagner-Hartl et al., 2023). This points to the need for an appropriate training to reduce fears and increase trust and acceptance. Research in this area is currently very limited. Virtual realities (VR) have frequently proven to be helpful platforms for the implementation of trainings (e.g., Takac et al., 2019; Lee et al., 2019).
The aim of the study is to examine subjective impressions and sugges- tions for implementing a virtual reality training in the industrial context. The research question that should be answered was: Which aspects are helpful or hindering when implementing a VR training within this context and what should be considered from a future users’ point of view?
METHOD
The experiment was part of a series of studies to evaluate the suitability of VR for trainings to reduce fears and increase trust and acceptance. The study was divided into an experimental and an exploratory part. This paper focuses exclusively on the exploratory part. For the purpose of completeness, the entire study design is described. During the experimental part of the study important experiences for the exploratory part were gained. A mixed design was chosen. The independent variables for the experimental part were gender, type of interaction (see procedure for more details) and the repeated measure- ment factor interaction number. As a first contact with all types of interaction was required for the exploratory part, the unseen types of interaction were also presented to all participants before the start of the exploratory part. The study was approved by the ethics committee of Furtwangen University.
Sample A total of 16 males and 14 females aged between 21 and 40 years (M = 24.83, SD = 3.82) participated in the study. All participants provided their informed consent at the beginning of the study.
Materials, Measures & Procedure As mentioned before, the study consists of two different parts: First, the par- ticipants had to perform a simple interaction task with an industrial robot. The performed task included the robot first passing over a wooden cube to the participant. This was followed by a visual inspection of the cube by the participant and its subsequent transfer back to the robot. The interaction task was conducted in three different scenarios “reality”, “virtual reality” and “hybrid reality” (see also Figure 1). All participants were exposed to all three scenarios in randomized order. In the first scenario exposed, the interaction task was done six times and used for psychophysiological mea- surements (not part of this paper). The next two scenarios were only short exposures to bring everyone up to the same knowledge level as preparation for the following part. The second part of the study took place in the form of a non-standardized semi-structured interview after completing all interaction
66 Birkle and Wagner-Hartl
tasks within the three scenarios. The main areas of interest were “positive and negative experiences during the interaction”, “design of a training scenario”, “gamification in the training situation” and “usage of different modalities in VR training”.
Statistical Analysis To answer the research question, the interviews were evaluated following a qualitative content analysis based on Mayring (2022). For this purpose, the raw material from the interviews was categorized. The categories were created using the inductive approach. This means that categories and subcat- egories were derived directly from the data during the categorization process. The statements in the categories were then checked for duplication and sum- marized into common terms. The frequency of the respective statements was also recorded.
RESULTS
Positive and Negative Experiences During the Interaction First, results from questions on highly positive and negative experiences are shown. Many of the participants’ statements related to the implementation of technical aspects. Among other things, it was noted that the VR scenery was well designed and implemented. The fact that all objects visible in VR were tactilely present in the same place as in reality was also mentioned as a posi- tive aspect. The most frequent positive statement was about the accuracy of the consistency between the virtual and real scenes. In addition to the techni- cal aspects, one participant also made statements about the general situation during the study. It was mentioned that there was a learning effect, that the perceived fun increased when a cross was discovered during the visual inspec- tion and that the virtual interaction was suitable as a scenario for practicing in a safe environment.
On the other hand, some negative experiences were also reported. These can be divided into the categories “poor body perception”, “poor technical aspects” and “poor environment and task design”. One of the complaints
Requirements for Virtual Reality-Based Trainings of Human-Robot Interaction 67
about body perception was that the participants body and hands were not visible in VR. The participants reported that this increased the fear of colli- sions with objects as distances were difficult to estimate. Reported deficits in the technology were mainly related to the head-mounted display. The partic- ipants indicated that the reasons were the blurring of the image, the pressure induced by wearing the head mounted display (HMD) and the flickering of the screen. In addition, in some cases, the occurrence of errors in the phys- ical behavior of the VR objects and an insufficient correspondence between VR and reality were reported. The presence of the measuring devices for the psychophysiological recordings also contributed to a participants’ nega- tive experience. In the category “poor environment and task design”, there were statements about effects like tiredness, concentration problems, bore- dom, heat in the room and eye strain due to excessive light. These statements were made regardless of the scenario.
Usage of Different Modalities in VR Training Regarding the modalities in the various types of interaction, the state- ments made by the participants can be divided into two main categories: “auditory aspects” and “haptic aspects”. These were divided into further subcategories. Regarding the “auditory aspects” category, the subcategories “positive aspects of the integration of auditory feedback”, “negative aspects of auditory feedback”, “differences in the order of interaction types” and “suggestions for concrete implementation”can be defined. The most frequent statement on the positive aspects of auditory feedback was the possible close- ness to reality. The participants noted that this could possibly lead to better training successes, which would tend to be missed in the absence of such feed- back. Two participants also stated that they experienced greater immersion, and three other participants emphasized the promotion of multimodality as particularly important in order to be able to reproduce perceptions as accu- rately as possible and give the participants more confidence in the virtual environment. Better situation awareness was also mentioned by four par- ticipants and was described as particularly important in the presence of a real robot. It was also noted that auditory feedback from the robot can also be a crucial factor in the development of anxiety and should therefore not be neglected during training. While a total of 34 statements were made about positive aspects, the negative aspects were limited to four statements. These stated that without auditory feedback of the robot there could be a better sense of security and less anxiety as well as more pleasant training situations. Furthermore, in six cases, participants who initially received no auditory feedback (no movement of the real robot; virtual scenario) stated that they were not aware that the real robot was not moving. Seven partic- ipants did not register that only the digital twin was moving in the virtual scenario until they had experienced auditory feedback for the first time. In contrast, participants who first received auditory feedback (reality or hybrid scenario) described the lack of feedback in the virtual interaction as strange. Specific recommendations for the implementation included choosing a qui- eter volume and indicating that simulated auditory feedback instead of real
68 Birkle and Wagner-Hartl
devices causing the feedback are sufficient if spatial perception is not affected. Furthermore, it was also considered as essential that the sound matches the robot’s movements. Four participants raised questions about the relevance for a training situation, while one person suggested a progression of training sessions without and with auditory feedback. It was also pointed out that the relevance of auditory feedback strongly depends on the person and the task to be accomplished during training.
The mentioned aspects of haptics can be divided into the subcategories “positive aspects of the integration of haptic feedback”, “technical possibili- ties of the implementation”and “comments on the implementation”. As with the auditory feedback, the most frequently mentioned positive aspect was the possible closeness to reality. Real collision objects in the places where virtual objects are shown to the users were mentioned to enable bumping. Further positive aspects were that the training is then more serious and that there is the opportunity to get a better understanding of the robot in VR. As it was already the case with auditory feedback, one person noted that the possibil- ity of a collision could be a factor of fear and should not be neglected for this reason. Seven participants stated that they considered haptics to be less important for a training context. For the technical implementation of hap- tic feedback, eight participants suggested the use of vibrations, two the use of external forces on the fingers and one a better simulation of weight in general. Regarding the general implementation, it was noted that collision objects are primarily important in places where there is a real risk of injury (five participants), that the scene is generally too unreal without them (one participant) and that the actual implementation of haptics depends on the planned interaction (three participants). One participant assumed that the interaction would be improved by haptic feedback. According to one, the complexity of the relevant objects must also be considered when developing the haptic feedback. As with auditory feedback, one participant suggested step-by-step training in which initially no feedback is given, and then more and more feedback is provided.
Design of a Training Scenario in VR To address the question of what a good training design should look like, three categories were defined based on the answers provided: “design of the train- ing task”, “design of the training environment”and “applicable technology”. The statements on the training task were divided into two subcategories: “important aspects of the training”and “suggestions for its implementation”. The implementation of an introduction scenario was mentioned most fre- quently (five times) as an important aspect of the training. Two statements referred to the use of storytelling, the clarification of the aim/sense of the training, the use of a real task and the limitation to an adequate duration. Three participants also emphasized the importance of simplicity so that the training will be suitable for all different user groups. Furthermore, a broad range of variety, an interesting design, the use of different content and a very accurate representation of the robot were mentioned as important points when designing a training task. When it comes to the specific implementation
Requirements for Virtual Reality-Based Trainings of Human-Robot Interaction 69
of a training, a total of 16 participants stated that a course with different lev- els would be particularly suitable. Examples given included the continuous addition of new modalities or increasing the task complexity. Five partici- pants stated that a real robot should be present, while two did not think this is necessary. It was also suggested that the training should be able to be cus- tomized specifically for the current user. It was also noted that malfunctions of the robot and distractions should play a central role in the training. Two participants thought it would be useful if the trainer was visible in VR.
Statements were also made about important components for the VR train- ing environment. These related primarily to the highest possible level of detail for the robot. It was also pointed out that it is important to avoid program errors, implement correct physical behaviour and ensure that the scene is true to scale. Concrete suggestions for the implementation can be found in Figure 2.
Figure 2: Suggestions for designing the training environment.
With respect to the technology that can be used for VR trainings, responses included statements about hand tracking, controllers, and virtual reality in general. Regarding hand tracking, the closeness to reality (seven participants) and the importance to provide a corresponding prototype of the object being interacted with (four participants) were mentioned. Furthermore, three par- ticipants pointed out that the usefulness of hand tracking strongly depends on the type of task to be performed. However, the statements varied regard- ing the appropriate level of detail of the hand tracking: One thought that a low level of detail is sufficient, another participant highlighted that an exact tracking is important, and another one considered the exact representation of the hands as essential. Also, some participants were more comfortable with hand tracking than using a controller because they empathized better with the scenario and assumed an improvement in body awareness. The suitability of hand tracking for complex tasks was considered to be a possible limitation
70 Birkle and Wagner-Hartl
and it was noted that this could lead to additional anxiety causes. Regard- ing the use of the controller, such as in this study, eleven participants stated that this form was completely suitable for them and two found the controller easy or pleasant to use. Again, one participant perceived the controller as a foreign object, two others found that it could only be used sensibly with pre- vious experience and a another had doubts about its usefulness for complex tasks. The unrealistic nature and weight of a controller was also criticized. With regard to virtual reality in general, it was noted that the HMD has to fit well and that the screen resolution can have an impact on the experience. In addition, according to some comments, a familiarization phase is necessary and things like the latency of the image play a major role. It was also pointed out that using VR can be exhausting for the body and eyes. The additional weight on the head and having to look continuously at a screen were cited as reasons for this.
Gamification in the Training Situation Information about gamification in trainings has been divided into three cate- gories. These are “advantages of gamification”, “limitations of gamification” and “possible implementation of gamification”. The most popular statement concerning the benefits of gamification was the increased motivation of the participants. In addition to motivation, it was also mentioned that interest and openness towards the training can be increased, and monotony can be reduced using gamification within this context. Furthermore, gamification enables the participants to forget their fears during the training or at least experience them in a less intense way. It was also indicated that training and learning success may increase. According to their statements, gamification can also help users to associate positive experiences with the robot, make the initial contact with it easier and the general situation more pleasant (see Figure 3).
Figure 3: Statements about the advantages of Gamification.
Requirements for Virtual Reality-Based Trainings of Human-Robot Interaction 71
In the category “limits of gamification”, it was mentioned quite often that a suitable balance between fun and professionalism needs to be found in order to make the training efficient. According to three participants, the focus should always remain on the actual task and gamification should only be used as a support. Two participants also noted that the training should not drift too far away from reality and that too much gamification can lead to a distorted perception of the seriousness of the situation. According to one par- ticipant, incorrect implementation of gamification can have a negative impact on the success of the training or, as six other participants noted, result in a subsequent surprise, as the robot was initially perceived differently during training. It was also pointed out that the success of gamification depends on the respective user.
The usage of competitive training, the provision of individualization options (for example: changing the appearance of the robot) or the use of a reward system were suggested as possible implementations. It was also suggested that the training progress could be visualized. Otherwise, one par- ticipant thought that the idea of a scoring system during training will not be very useful. The possibility of staged training was also highlighted at this point, in which a high amount of gamification is used in the beginning which then decreases step by step.
DISCUSSION
To answer the research question “Which aspects are helpful or hindering when implementing a VR training within this context and what should be considered from a future users’ point of view?” it should first be noted that the participants agreed with each other in most cases, and that few con- tradictory statements were made. The implementation of an introductory phase was considered extremely important. This type of introductory sce- nario was also carried out as part of this study and was well suited in this context to bring everyone up to a similar level and enable them to get to know the technology. The participants’ statements are therefore confirmed by the observations made in this study. It was also suggested that the duration of the training should be as short as possible but still sufficient. In addition, the importance of a simple implementation of the training task and clear communication of the goals and purpose of the training was emphasized.
The frequently suggested implementation of training levels must be high- lighted. Initially, training should be carried out with a simple task, with few distractions and environmental influences and excluding other modalities such as auditory or haptic perception. As the training level increases, these aspects should also be adjusted accordingly. These suggestions are already used in practice, for example in the form of adaptive training, in which the actual stimulus adapts to the individual performance of the participant (Kelley, 1969). The relevance of adaptive training is therefore confirmed in practice and should be implemented in the development of the future VR training. Opinions differ in the interviews about whether a real robot should be present in the training setup or not. Some consider this to be especially useful, while others see no necessity. Under certain circumstances, this aspect
72 Birkle and Wagner-Hartl
could also be included in an adaptive training. If sufficient progress is made, a real robot could then be present in the room and, if even more progress is made, the user could interact with it directly. Finally, the visibility of the trainer in virtual reality was noted as important for a successful train- ing environment. Possible implementations include the presentation of the trainer as an avatar or a detailed integration of the moderator via full-body tracking. The participants broadly agreed on the specific design suggestions for VR training (see Figure 2). Many participants mentioned the importance of ensuring that the implementation is as realistic as possible. Only when it comes to the evaluation of contextual information such as environmen- tal noises, distraction, visual representation of a working atmosphere (e.g. a machine hall) or the integration of olfactory information do participants’ opinions differ greatly. No uniform suggestion can be identified. In a fur- ther study, these aspects should therefore be revisited and explored in greater depth to draw the best possible conclusions.
Even when it comes to the technology that can be used to implement VR training, the use of techniques to increase realism was emphasized most fre- quently. The focus here was on hand tracking and the controllers used. There was disagreement regarding the necessary level of detail in the representa- tion of hands and the accuracy of hand tracking. However, a study by Kim et al. (2017) showed an increase in immersion and presence when using a hand tracking system that allows haptic feedback. The use of this type of sys- tem should therefore be reconsidered for the implementation of VR training. However, there was a general agreement in this study that the use of a con- troller is usually sufficient if it is easy to understand and use to perform the task realistically. The elaboration of the respective advantages and disadvan- tages of different technologies was not part of the interviews. Therefore, the technologies can only be compared with each other to a very restricted extent. This should be given greater focus in subsequent studies. Regarding gamifi- cation during a VR training in this topic, a large amount of information was collected. Overall, the participants strongly agreed regarding both the advan- tages and disadvantages of gamification and its specific implementation. The extraordinarily large amount of information obtained shows the relevance of the topic in society as well as for the implementation of VR training. In partic- ular, the advantages “motivation increases”, “interest increases”, “monotony decreases” and “helps to overcome anxiety” can be directly applied to prob- lems that occurred in this study. The appropriate use of gamification can be a solution to this. VR training will later be used, among other things, to reduce anxiety, whereby the gamification effect “helps to overcome anxiety” can be particularly helpful when approaching the robot for the first time. Further research is needed to evaluate whether such effects can be achieved using a VR training for human-robot interaction. Nevertheless, the use of gamifica- tion already suggests great potential. This is also confirmed by recent studies (Lumsden et al., 2016; Vermeir et al., 2020).
When considering the research question, it is important to point out the background and limitations of these interviews. As they were only intended to obtain an initial general opinion on aspects of the design of VR train- ing, no universally valid conclusions can be drawn from the statements made
Requirements for Virtual Reality-Based Trainings of Human-Robot Interaction 73
here. But they can and should be used as orientation for subsequent data col- lections. The available data appears to contain a sufficiently high density of information to serve as such an orientation. Finally, it should be noted that this study is only an exploratory approach and can be seen as a first step towards the development a VR training in this context. The relatively young sample strengthens this statement.
To sum it up, the interviews revealed many concrete suggestions and requirements for the implementation of a VR-based training of human-robot interaction. They provide a useful basis for the future training development.
AUTHOR’S STATEMENT The authors state no conflict of interest. The study was approved by the ethics committee of Furtwangen University. Informed consent has been provided from all participants of the study.
REFERENCES
Birkle, J., Vogel, A., Wagner-Hartl, V. (2022). “Impact of Distance and Movement Speed on the Acceptance of Human-Robot Interaction – Method and First Eval- uation”, in: HCI Int. 2022 – Late Breaking Posters, Stephanidis, C. et al. (Eds.). pp. 483–490. Broadbent, E. (2017). Interactions With Robots: The Truths We Reveal About
Ourselves, ANNU. REV. PSYCHOL. Volume 68, 627–652. Hancock, P. A., Billings, D. R., Schaefer, K. E., Chen, J. Y. C., de Visser, E. J., Parasuraman, R. (2011). A meta-analysis of factors affecting trust in human-robot interaction. HUMAN FACTORS Volume 53 No. 5, 517–527. International Federation of Robotics. (2024). Global Robotics Race: Korea, Sin-
gapore and Germany in the Lead. International Federation of Robotics Web- site: https://ifr.org/ifr-press-releases/news/global-robotics-race-korea-singapore-a nd-germany-in-the-lead. Kim, M., Jeon, C., Kim, J. (2017). A study on immersion and presence of a portable hand haptic system for immersive virtual reality, SENSORS Volume 17, 1141. Lee, H. S., Park, Y. J., Park, S. W. (2019). The effects of virtual reality training on func-
tion in chronic stroke patients: A systematic review and meta-analysis, BIOMED RESEARCH INTERNATIONAL Volume 2019, 1–12. Lumsden, J., Edwards, E. A., Lawrence, N. S., Coyle, D., Munafò, M. R. (2016).
Gamification of cognitive assessment and cognitive training: A systematic review of applications and efficacy, JMIR SERIOUS GAMES Volume 4, e11. Mayring, P. (2022). Qualitative Inhaltsanalyse: Grundlagen und Techniken [Qualitative content analysis: basics and techniques]. Beltz. Nomura, T., Kanda, T., Suzuki, T., Kato, K. (2008). Prediction of human behaviour
in human-robot interaction using psychological scales for anxiety and negative attitudes toward robots, IEEE TRANSACTIONS ON ROBOTICS Volume 24, 442–451. Robla-Gómez, S., Becerra, V. M., Llata, J. R., González-Sarabia, E., Torre-Ferrero, C., Perez-Oria, J. (2017). Working Together: A Review on Safe Human-Robot Col- laboration in Industrial Environments, IEEE ACCESS Volume 5, 26754–26773. Statista. (2023). Volume of robotics market worldwide from 2016 to 2028, by region.
Takac, M. et al. (2019). Public speaking anxiety decreases within repeated virtual reality training sessions, PLOS ONE Volume 14, e0216288. Vermeir, J. F., White, M. J., Johnson, D., Crombez, G., Van Ryckeghem, D. M. L.
(2020). The effects of gamification on computerized cognitive training: Systematic review and meta-analysis, JMIR SERIOUS GAMES Volume 8, e18644. Wagner-Hartl, V., Gleichauf, K., Schmid, R. (2020). “Are we ready for human robot collaboration at work and in our everyday lives? - an exploratory approach”, in: Human systems engineering and design II, Ahram, T. et al. (Eds.). pp. 135–141. Wagner-Hartl, V., Schmid, R., Gleichauf, K. (2022). “The influence of task com-
plexity on acceptance and trust in human-robot interaction - gender and age differences”, in: Cognitive computing and internet of things, Paletta, L., Ayaz, H. (Eds.). pp. 118–126. Wagner-Hartl, V., Nakladal, S., Koch, T., Babajic, D., Mazur, S., Birkle, J. (2023).
“Influence of Movement Speed and Interaction Instructions on Subjective Assess- ments, Performance and Psychophysiological Reactions During Human-Robot Interaction”, in: HCI Int. 2023 – Late Breaking Papers, Kurosu, M., et al. (Eds.). pp. 461–175. Weber, M.-A., Schüth, N. J., Stowasser, S. (2018). Qualifizierungsbedarfe für die Mensch-Roboter-Kollaboration [Qualification needs for human-robot collabora- tion]. Zeitschrift Für Wirtschaftlichen Fabrikbetrieb, Volume 113, No. 10, pp. 619–622.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Cognitive Computing and Internet of Things
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-00-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional