Werner Ravyse, Kimmo Tarkkanen, Mika Luimula, Jouni Smed (2024). A Data-Driven Approach to Identify Outcomes That Require Immediate Feedback for Effective VR Training. In: Human Factors, Business Management and Society. AHFE Open Access, vol. 135, pp. 257–267. https://doi.org/10.54941/ahfe1004952
Over the past decade Virtual reality (VR) technology has become more readily available. Although VR still suffers from inducing simulation sickness and having cumbersome controllers, industrial companies have embraced the technology as an alternative to overly complex and expensive simulators. A primary pedagogical consideration for effective VR training is feedback. While research has proven the positive training impact of feedback over having no feedback, very few studies explore techniques to help pinpoint the exact training outcomes that require feedback in a VR training simulator. This study illustrates how collecting human interaction metrics data from a VR fire extinguisher training application without a feedback mechanism can identify the areas where feedback for effective training is most required. We conducted a descriptive comparison study with 36 participants. We collected and analysed the participants’ VR interaction data across nine validated metrics and compared the results to their practical fire extinguishing evaluation on the same metrics. Our results show that the interaction data from the VR application presents curious behaviour regarding the distance participants kept between themselves and the fire, and their regard for wind direction when approaching the fire. The same elements surfaced in the practical evaluation. We conclude that collecting and analysing metrics from a VR training application is a suitable technique for identifying training aspects or outcomes that require a feedback mechanism.
methods often struggle to provide (Naranjo et al., 2020). VR simulations enable trainees to interact with lifelike scenarios, replicating real-world envi- ronments and tasks. This hands-on approach enhances experiential learning, allowing users to practice and refine skills in a risk-free virtual setting (Guo et al., 2020). Moreover, VR-based training programs have demonstrated improved knowledge retention, reduced training time, and increased over- all performance, contributing to a more efficient and competent workforce (Radhakrishnan, Koumaditis and Chinello, 2021).
Despite the promising advantages of VR in industrial training, challenges persist. Issues such as the initial cost of implementation (Holuša et al., 2023), the need for specialized content development (Turner et al., 2016), and con- cerns about motion sickness among users (Saredakis et al., 2020) need to be addressed for widespread adoption. Additionally, interoperability and stan- dardization remain important considerations (Naranjo et al., 2020) to ensure seamless integration with existing training programs and platforms.
To prevent challenges from outweighing the advantages of using VR headsets for industrial training, designers and developers of such applica- tions must follow a sound pedagogical approach to ensure effective learning (Somerkoski et al., 2020). A key element of effective learning within a dig- ital environment is feedback (Van der Kleij, Feskens and Eggen, 2015). While much research has proved the positive training impact of the various forms of feedback (Donalek et al., 2014), such studies are typically based on known factors that will influence learning, or with a blanket approach of giving all possible feedback and comparing it to having no feedback. The greater difficulty and value, when it comes to building VR simulation training applications, lies in pin-pointing the training elements that require feedback.
This study illustrates how collecting human interaction metrics data from a VR training application that has no feedback mechanics can identify the areas where feedback for effective training is most required and reflects on the imminent danger of not including sufficient feedback in VR simulation training for hazardous tasks.
FEEDBACK IN VR TRAINING A comprehensive feedback system is integral to optimizing the efficacy of VR training programs, ensuring a dynamic and engaging learning experi- ence. A robust feedback system encompasses various elements that deliver constructive insights and foster continuous improvement.
The first user encounters with training-specific feedback typically come when the application provides real-time performance responses, offering instantaneous feedback to trainees’ actions within the VR environment. Through visual cues, audio signals, and haptic stimuli, the training appli- cation can replicate real-world consequences, thereby aiding immediate understanding (Schuster et al., 2015).
As a second tier of feedback, immediately after completing a virtual train- ing scenario, users expectantly receive a concise overview of their engagement metrics that were collected throughout their interaction with the training application. This summative feedback information can be applied to scor- ing and assessment criteria based on the specific skills or competencies being trained (Lin et al., 2002).
A Data-Driven Approach to Identify Outcomes That Require Immediate Feedback 259
Thirdly, storing summative information such as completion time, accu- racy, and adherence to safety protocols in a database extends the value of the quantified performance metrics by enabling the generation of detailed progress reports and analytics for debriefing purposes (Hall and Tori, 2017). With debriefing reports, trainees can track and reflect on their improvement, while instructors gain insights into areas requiring additional attention. These reports also provide a catalyst for constructive peer-to-peer and teacher- trainer discussions toward achieving deeper learning or enriched training outcomes (Ravyse et al., 2017).
Beyond the three core feedback tiers (immediate, summative and reflec- tive), adaptive learning paths, adjusted difficulty levels based on individual performance (Lopes and Lopes, 2022), and instructor feedback mechanisms are particularly valuable in complex or specialized training scenarios (Huber et al., 2021). Ultimately, a continuous improvement mechanism, informed by user feedback and system analytics, ensures the ongoing enhancement of VR training.
This study focuses on applying an analysis of human interaction metrics toward identifying training outcomes that require an immediate feedback mechanic.
THE RESEARCH PROTOTYPE Our research prototype was built within the TUAS-led Virtual Training Cer- tifications (VTC, 2022) project funded by Business Finland in which our industrial partners have developed a European Hot Work Certification to test the effectiveness of digital learning and assessment. The research pro- totype is a first iteration of a VR training application that aims to instruct participants how to put out a fire with a foam and liquid fire extinguisher. This first iteration has no discernible feedback mechanic for the user, but does collect and store virtual interaction metrics (see Table 1) at a rate of ten times per second. The metrics were verified and validated by a certified fire safety expert.
Table 1. Stored virtual interaction metrics of the fire extinguisher research prototype.
Table available in the original PDF paper.
Metric Name Metric Description Unit of Measure Seconds TimeFromStart Elapsed time since picking up the virtual fire extinguisher
DistanceToFire Distance between the user and the fire Meters (to the ninth decimal) PinSuccess Indicator that the pin is successfully removed Yes/No ExtinguisherEngaged Indicator that the user is triggering the extinguisher Yes/No HeightOfNozzle Height at which the user is holding the extinguisher nozzle
Meters (to the ninth decimal) SprayToFireDistance Distance on the z-axis that the extinguisher spray lands from the fire
Meters (to the ninth decimal) BottleContents Remaining spray volume in the extinguisher bottle Percentage Approach With Respect To Wind
Absolute angle to the wind at which the user approaches the fire
Degrees (0 is directly into the wind) FireIntensity Strength of the fire Number of active fires
The virtual fire extinguishing training scenario mimics the practical evalu- ation environment and process that occupational health and safety students at TUAS undergo. The virtual scenario starts with a tutorial phase where users have an opportunity to interact with a virtual bottle (pick up the bottle, remove the pin, activate the spray mechanic, and drop the bottle into a desig- nated area) and are presented with a diagram of a labelled fire extinguisher. No fire extinguishing instructions are given during the tutorial phase.
For the virtual extinguishing phase (see Figure 1), users see a burning fire and are expected to: (a) retrieve the fire extinguisher; (b) remove the pin; (c) approach the fire from an upwind direction, which participants could determine from a visual smoke cue; (d) stand a safe distance from the fire; (e) extinguish the fire by activating the spray mechanic and spraying in a left-to-right sweeping motion at the base of the fire.
Figure 1: Fire extinguishing scene from a first-person user perspective.
METHODOLOGY
The ultimate aim within the VTC project is to compare the effectiveness of traditional controllers with a more lifelike custom bottle controller (see Figure 2) designed and made by Ade Oy (one of our VTC project part- ners). To this end we are following an iterative process of systematically including the three feedback tiers (immediate, summative and reflective) and determining how each feedback tier impacts the training effectiveness for the two controllers. The only differences between the two virtual fire extin- guisher groups were the controllers and the controller-dependant tutorial that prepared participants for the virtual interaction data collection phase. This paper discusses how we utilised combined user engagement metrics from both the conventional and custom controller groups to identify where immediate feedback is required.
Research Design This study is a descriptive quantitative analysis of user-interaction data met- rics we collected from a virtual fire extinguishing application. Each of the nine metrics from Table 1 were individually extracted and analysed to determine
A Data-Driven Approach to Identify Outcomes That Require Immediate Feedback 261
characteristics such as averages, trends, and spread to find data irregularities that may indicate a requisite for immediate feedback. The descriptive analyses of the virtual scenario data were compared with the summarised aggregated results from an evaluation rubric that we applied to the practical (extinguish- ing a real fire) assessment of the research participants. We completed (through observation) the evaluation rubric while participants were completing their exercise. To ensure the validity of the evaluation rubric data, the fire safety expert verified each participant’s evaluation immediately after their practical session. The rubric measured the same metrics as the virtual interaction, but also included observations about the physical ease or difficulty participants had with removing the extinguisher’s safety pin and how much instruction they received from the fire safety expert.
Figure 2: VR controllers for the two participant groups.
Research Protocol and Data Collection We recruited participants from a cohort of third- and fourth-year ICT engi- neering students studying at TUAS. The participants were randomly assigned to either the conventional or custom controller groups. We collected valid data from 20 participants in the conventional controller group and 16 in the custom bottle controller group. The virtual interaction data collection took place over two days. The conventional controller group engaged in the fire extinguisher application on the first day. We asked participants to arrive at staggered intervals and upon arrival they filled out a profile questionnaire about their fire extinguishing and VR experiences. Immediately after the profile questionnaire, each participant entered a room where no other par- ticipants could observe them and they completed the virtual scenario from where we collected the interaction metrics. On the second experiment day, we repeated the process with the bottle controller group.
Eight days after the virtual experience, 17 of the conventional controller and 11 of the bottle controller group participants came to the site where they underwent a practical fire-extinguishing exercise. The exercise was car- ried out under strict safety protocols, with a controlled gas fire by a certified
fire safety expert—participants wore regulation personal protection equip- ment. The researchers assessed each participant against a validated evaluation rubric in their competence to extinguish a real fire. To ensure validity and reliability, the fire safety expert verified each individual participants’ evalua- tion rubric assessments against their own observations. To once more avoid measurement bias, each participant partook in the extinguishing exercise out of sight from the others. The participants did not receive any pre-exercise instructions from the fire safety expert or researchers. We placed the fire extinguisher at the entrance to the fire site, the fire safety expert ignited the fire, and we told the participants they could start with their attempt to extin- guish the flames. The fire safety expert only intervened in cases of unsafe behaviour (by closing the gas valve to stop the fire) or inability to operate the fire extinguisher (by giving succinct instructions).
RESULTS
No participants had previous fire-extinguishing experience and none of them had attended any such courses prior to this research. Given the nature of their ICT studies, all participants had at least some previous experiences with VR headsets.
Virtual Data Results Of the nine virtual interaction metrics we collected and analysed, four (TimeFromStart, DistanceToFire, SprayToFireDistance, and ApproachWith- RespectToWind) showed unusual results. Unfortunately, we had to omit the SprayToFireDistance metric from this study because we found an anomaly in the calculation algorithm that caused all the captured readings to be negative (i.e., the spray was recorded as landing some distance in front of the fire). We could not determine a reliable coefficient to resolve the error and will therefore not report on the spray metric in this study.
The average time taken for each of the groups to complete the virtual fire extinguishing exercise was significantly different. The conventional con- troller group completed the fire extinguishing phase in an average time of roughly 27 seconds, while the bottle controller group took 84 seconds on average. Our observation during the interventions found this measurement unusual and further examination of the coding revealed that the starting time trigger for the bottle controller started much sooner than for the conventional controllers. To have some sensible comparison of the efficiency for each of the controllers we decided to compare the time it took to extinguish the fire from the moment that the safety pin was removed. We conjectured that this was the point in time that participants had a real intent to extinguish the fire. From removing the safety pin until the fire was extinguished, the con- ventional controller group took 17 seconds and the bottle controller group required 20 seconds on average. This slight difference may have been due to the conventional controllers having a more efficient teleport activation, or that the conventional controller group (who were predominantly interactive technology students) were more proficient in using VR equipment. We felt that this difference is negligible enough to not warrant a deeper investigation.
A Data-Driven Approach to Identify Outcomes That Require Immediate Feedback 263
The data showing the virtual distance that participants maintained between themselves and the fire presented no difference between the two groups, but does show that participants as an aggregated group were not conscious of the appropriate distance from which to safely extinguish the fire. The recommended safe distance to extinguish a fire of this nature and size, with a foam fire extinguisher, is approximately 1.5m to 2.2m (accord- ing to the fire safety expert). To determine how close participants came to the fire, we extracted and analysed the minimum distance each participant had between themselves and the fire. In Figure 3 we see that 21 participants did not enter into the optimum extinguishing distance (i.e., they remained too far away from the fire) and seven participants went beyond the optimum zone (i.e., they came unsafely close to the fire).
Figure 3: Proximity to which participants approached the fire.
The ApproachWithRespectToWind metric measures the angle between the participant and the direction of the wind vector with regard to the position of the fire (see Figure 4). An angle measurement of zero degrees implies that the participant was on the hot side of the fire directly into the wind—the most dangerous direction. An angle measurement of 180 is on the completely safe side of the wind with regard to the fire. The angle of approach data is only recorded when a participant comes within a distance of 5m from the fire. In the VR application, participants could establish the wind direction by looking at where the fire smoke was blowing toward. The wind blew from a random direction for each of the participants.
Figure 4: Unsafe angles to approach the fire with regard to wind direction.
We analysed the average and standard deviation of the angle that each par- ticipant maintained with respect to the wind direction and fire position. While 16 out of 36 participants kept an average angle that was on the danger side of the wind, the standard deviation analysis revealed that only two participants changed their direction by more than 15 degrees during the virtual exercise. All other participants from both groups did not alter their angle of approach by more than 15 degrees. This implies that all, but two, participants merely picked up the bottle and walked directly to the fire. We therefore believe that the 20 participants who appeared to be safely upwind from the fire, may have had the wind direction at their backs from the beginning of their extinguisher training scenario. That is, we have grounds to believe these par- ticipants had a “lucky” start. We are confident that the participants did not heed the wind direction enough to alter their direction for safely approaching the fire.
Practical Extinguishing Results From the 28 participant rubric recordings, we found three metrics that indicated incorrect or unsafe practices while extinguishing the fire: (a) the distance that participants had between themselves and the fire; (b) the regard for wind direction; and (c) the spray motion for extinguishing the fire.
The distance between the fire and the participant was observed and esti- mated on a scale of too close (less than 1.5m from the fire), ideal (between 1.5m and 3m from the fire), and too far (greater than 3m from the fire). Our observations noted that 12 participants came within 1m of the fire (caus- ing the fire safety expert to close the gas valve) and a further 12 went to a distance between 1m and 2m from the fire—24 out of 28 participants were unsafe regarding their distance to the fire.
On the day of the practical exercise there was a perceptible wind strong enough to impact the heat direction of the fire. From the 28 participants, eight approached the fire directly into the wind and another 16 walked from the starting position in a straight line to the fire without contemplating the wind direction—24 out of 28 participants were unsafe regarding the wind direction in their approach to the fire.
The spray technique for extinguishing a fire should be a continuous spray at the base of the fire in a left-to-right sweeping motion. In our experiment we observed 15 out of 28 participants either spraying arbitrarily at the fire or keeping the spray aimed at a constant location in the fire. Although the spray motion was haphazard, it was predominantly aimed at the base of the fire.
In addition to the evaluation rubric, we also asked the fire safety expert their opinion on the performance of the participants. The fire safety expert’s responses stated that, in general, the participants: (a) came too close to the fire; (b) did not pay attention to the wind direction; and (c) struggled more than they should to remove the extinguisher safety pin. On the latter, the comment continued that the participants were already squeezing the handle of the fire extinguisher while trying to remove the pin, which causes the pin to be tightly wedged in the handle, making it difficult to remove.
A Data-Driven Approach to Identify Outcomes That Require Immediate Feedback 265
Data Collection Summary We collected data from a virtual fire extinguishing scenario and a practical fire extinguishing exercise along nine validated metrics and compared the results. In analysing the virtual data we noticed that participants kept erratic distances between themselves and the virtual fire and appeared to show no concern regarding the wind direction and their approach to the fire. Both of which would be alarming behaviours when trying to extinguish a real fire. During the real fire extinguishing exercises, participants exhibited these exact behaviours. In addition, their spray technique appeared to depart some- what from the effective sweeping motion. We were unable to detect this data from the virtual intervention because we had an algorithmic error that we will correct in future iterations. As a further confirmation, the fire safety expert corroborated that participants were unsafe in their distance and wind direction considerations when extinguishing the real fire.
CONCLUSION AND FUTURE RESEARCH Feedback systems can be complex and require careful consideration when being built into a virtual training application. In this research we showed that collecting and analysing interaction metrics from a virtual training sce- nario that has no feedback can identify the elements that trainees would find challenging in real situations. More importantly, we reflect (as many studies before us have proven) that without effective feedback, trainees could trans- fer unsafe practices to real life scenarios, placing them in physical harm’s way. This paper gives designers of industrial training applications a data- driven approach that not just identifies the elements to build feedback systems for, but could also guide the mechanics of those feedback systems. We fur- ther conjecture that such a data-driven approach would be suitable across other interactive technology training modes that incorporate some form of extended reality (XR).
We acknowledge that the fire extinguisher use case results we report here cannot be generalised to all VR training scenarios. Our contribution lies in providing an empirical methodology that researchers and designers can draw upon to create impactful and effective feedback systems for their VR training applications, as well as presenting practical insight into the value of immediate feedback.
Future Research The next iteration of our VR fire extinguisher application will include an immediate feedback system that communicates the dangers of being too close to the fire and approaching the fire from a downwind angle. For this we plan to use a systematically darkening vignetting technique that is exagger- ated as the downwind angle becomes more dangerous. We will also build in a stronger visual cue to indicate wind direction. We plan to repeat the experiment and compare the results of the practical evaluations to deter- mine whether the immediate feedback system impacts how a homogenous participant group with no fire-fighting experience responds in a real-life fire extinguishing exercise. Beyond this, we intend to build two more versions of
the application: (a) with only a second tier summative feedback system; and (b) with only a third tier debriefing (reflective) feedback system. Our goal with this would be to ascertain the different impacts each of these feedback tiers have on real scenario behaviour.
ACKNOWLEDGMENT
The authors would like to acknowledge Business Finland for funding the Virtual Training Certifications project being led by TUAS. We also wish to thank the Futuristic Interactive Technologies development engineers and stu- dent assistants who worked on creating the application, as well as our project consortium partners, in particular Kiwa and Ade Oy, for their valuable insight and reference material throughout the research process. Finally, thank you to the research participants who willingly gave their time and effort to join this study.
REFERENCES
Anthes, C., García-Hernández, R. J., Wiedemann, M. and Kranzlmüller, D., 2016, March. State of the art of virtual reality technology. In 2016 IEEE aerospace conference (pp. 1–19). IEEE. Donalek, C., Djorgovski, S. G., Cioc, A., Wang, A., Zhang, J., Lawler, E., Yeh, S.,
Mahabal, A., Graham, M., Drake, A. and Davidoff, S., 2014, October. Immersive and collaborative data visualization using virtual reality platforms. In 2014 IEEE International Conference on Big Data (Big Data) (pp. 609–614). IEEE. Guo, Z., Zhou, D., Zhou, Q., Zhang, X., Geng, J., Zeng, S., Lv, C. and Hao, A., 2020. Applications of virtual reality in maintenance during the industrial product lifecycle: A systematic review. Journal of Manufacturing Systems, 56, pp. 525–538. Hall, K. and Tori, K., 2017. Best practice recommendations for debriefing in
simulation-based education for Australian undergraduate nursing students: An integrative review. Clinical Simulation in Nursing, 13(1), pp. 39–50. Holuša, V., Vanˇek, M., Beneš, F., Švub, J. and Staša, P., 2023. Virtual Reality as a Tool for Sustainable Training and Education of Employees in Industrial Enterprises. Sustainability, 15(17), p. 12886. Huber, T., Mertes, S., Rangelova, S., Flutura, S. and André, E., 2021, December.
Dynamic difficulty adjustment in virtual reality exergames through experience- driven procedural content generation. In 2021 IEEE Symposium Series on Computational Intelligence (SSCI) (pp. 1–8). IEEE. Lin, F., Ye, L., Duffy, V. G. and Su, C. J., 2002. Developing virtual environments for industrial training. Information sciences, 140(1–2), pp. 153–170. Lopes, J. C. and Lopes, R. P., 2022, October. A review of dynamic difficulty
adjustment methods for serious games. In International Conference on Opti- mization, Learning Algorithms and Applications (pp. 144–159). Cham: Springer International Publishing. Naranjo, J. E., Sanchez, D. G., Robalino-Lopez, A., Robalino-Lopez, P., Alarcon- Ortiz, A. and Garcia, M. V., 2020. A scoping review on virtual reality-based industrial training. Applied Sciences, 10(22), p. 8224. Radhakrishnan, U., Koumaditis, K. and Chinello, F., 2021. A systematic review of
immersive virtual reality for industrial skills training. Behaviour & Information Technology, 40(12), pp. 1310–1339.
A Data-Driven Approach to Identify Outcomes That Require Immediate Feedback 267
Saredakis, D., Szpak, A., Birckhead, B., Keage, H. A., Rizzo, A. and Loetscher, T.,
2020. Factors associated with virtual reality sickness in head-mounted displays: a systematic review and meta-analysis. Frontiers in human neuroscience, 14, p. 96. Schuster, K., Plumanns, L., Groß, K., Vossen, R., Richert, A. and Jeschke, S.,
2015. Preparing for Industry 4.0–Testing collaborative virtual learning environ- ments with students and professional trainers. International journal of advanced corporate learning, 8(4), p. 14. Somerkoski, B., Oliva, D., Tarkkanen, K. and Luimula, M., 2020, January. Digital learning environments-constructing augmented and virtual reality in fire safety. In Proceedings of the 2020 11th International Conference on E-Education, E- Business, E-Management, and E-Learning (pp. 103–108). Turner, C. J., Hutabarat, W., Oyekan, J. and Tiwari, A., 2016. Discrete event simula-
tion and virtual reality use in industry: new opportunities and future trends. IEEE Transactions on Human-Machine Systems, 46(6), pp. 882–894. Van der Kleij, F. M., Feskens, R. C. and Eggen, T. J., 2015. Effects of feedback in a computer-based learning environment on students’ learning outcomes: A meta- analysis. Review of educational research, 85(4), pp. 475–511.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors, Business Management and Society
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-11-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional