Michael Schneeberger, Lucas Paletta, K Wolfgang Kallus, Lilian Reim, Christian Schönauer, Andreas Peer, Richard Feischl, Georg Aumayr, Martin Pszeida, Amir Dini, Stefan Ladstätter, Anna Weber, Alexander Almer, Dietmar Wallner (2022). First Responder Situation Reporting in Virtual Reality Training with Evaluation of Cognitive-emotional Stress using Psychophysiological Measures. In: Cognitive Computing and Internet of Things. AHFE Open Access, vol. 43, pp. 73–82. https://doi.org/10.54941/ahfe1001841
First responders engage in highly stressful situations at the emergency site that may induce stress, fear, panic and a collapse of clear thinking. Staying cognitively under control under these circumstances is a necessary condition to avoid useless risk-taking and particularly to provide accurate situation reports to remote units to be able to organize appropriate support in time. This work applied a flexible virtual reality (VR) training environment with the purpose to investigate the performance of reporting under rather realistically simulated mission conditions. In a pilot study, representative emergency forces of the Austrian volunteer fire brigade and paramedics of the Johanniter organization were subjected to a test program that tested a formalized reporting schema (LEDVV), inducing equivalent strain in both, real environment and VR-based training scenarios. Wearable psychophysiological measuring technology was applied to estimate the cognitive-emotional stress level under both training conditions. The results indicate that both situation reports achieve a rather high level of cognitive-emotional stress and should be thoroughly trained. Furthermore, the results motivate the use of VR environments for the training of stress-resilient decision-making behavior of emergency forces.
Figure 1: Situation reporting under cognitive-emotional stress, following the LEDVV command scheme, (a) under ‘Reality Condition’ (RC) and (b) under ‘Virtual Reality Con- dition’ (VR). VR is embedded within a commercially available first responders training environment. The results indicate that stress in the context of situation reporting in the simulated environment was comparable with stress in the real environment scenario.
1995). Their physical and cognitive readiness is of highest importance to ena- ble appropriate decision making (Frye & Wearing, 2014). Staying cognitively under control is a necessary condition to avoid useless risk-taking, to empo- wer those in need, and particularly to provide accurate situation reports to remote units to be able to organize appropriate support in time. First respon- ders maintain situation awareness by following the international LACES framework (Lookouts, Communications, Escape Routes, and Safety Zones; see Alexander et al., 2015; AFAC, 2016). In Austria, LACES is applied within the LEDVV command schema (ÖBFV, 2016) for situation reporting. LEDVV includes information by attention to a current situation, decision, execution, operation support system, and support by the operation controller.
Training the first responders in the LEDVV routine improves resilience towards stressors in hazard conditions. Training in real environments requi- res a high expenditure of resources and allows only limited training frequency and variability. For these reasons, virtual reality (VR) training environments with realistic simulations are necessary, in which appropriate levels of dif- ficulty can be adaptively set to adapt the stressful scenarios in a targeted manner. Advanced training of first responders and emergency staff with typical operation scenarios is one of the upcoming challenges of the near future.
In a pilot study, representative emergency forces (N = 13) of the Austrian volunteer fire brigade and paramedics of the Johanniter organization were subjected to a test programme that investigated the LEDVV schema, indu- cing equivalent strain in both, real environment training scenarios, i.e., under ‘Reality Condition’ (RC), and VR-based training scenarios, i.e., under ‘Vir- tual Reality Condition’ (VR). The cognitive-emotionally challenging phase included the LEDVV situation report with (i) the presentation of an emerge- ncy scenario with critical events (ES; mediated either by TV or by VR), (ii) giving the request by the team leader (RQ), and (iii) the presentation of the LEDVV situation report by the first responder (SR).
First Responder Situation Reporting in Virtual Reality Training 75
Metadata that indicate stress were derived from wearable psychophysi- ological measuring technology (ECG, EDA, respiration, and eye tracking). We computed stress-indicating features from the EDA signal following Bou- csein et al. (2012) and Braithwaite et al. (2015); with individually normalised SCR frequency, resulting in medians of 0.00% for resting phase, 58.40% (ES), 73.44% (RQ) and 83.99 (SR) for the situation report phase in RC and 70.77% (ES), 81.83% (RQ), and 75.85% (SR) for VC, respectively. These figures of the descriptive statistics indicate that LEDVV reporting leads to increased psychophysiological activation in both conditions due to cognitive- emotional stress. Similar figures were attained from measurements of heart rate variability (HRV, using SDNN following Salahuddin et al., 2007) and mean heart rate (HR). HRV was determined from the ECG signal using the standard deviation of the interbeat intervals of normal sinus beats (SDNN) after a semi-automatically cleaning of the ECG signal from artefacts.
In summary, the results indicate that both situation reports achieve a rather high level of cognitive-emotional stress and should be thoroughly trained. Furthermore, the results motivate the use of VR environments for the training of stress-resilient decision-making behavior of emergency forces.
VIRTUAL REALITY TRAINING SYSTEM
Related Work
Recently, VR-based training in disaster preparedness has been increasin- gly recognized (Freeman et al., 2001) as an important additional modality to traditional real-life skill training. Multiple studies (Cone et al., 2011; Kurenov et al., 2009) have highlighted VR applications in disaster training. The increased realism in the practice enables first responders to reinforce their individual performance, in particular, to execute tasks appropriately under stress and apply decision making under conditions close to reality. The immersive capability of VR-based training offers more realism than classroom-based instructive teaching and substantially reduces the resou- rce burden of real-life drills and tabletop exercises. Mills et al. estimated that a mass casualty triage training of paramedic students in a real-world simulation is about 13 times more expensive than in VR, while the simu- lation efficacy has been found near identical (Mills et al., 2019). Recent research has even indicated superior performance in simple search tasks following VR and augmented reality (AR) training of first responders as opposed to traditional classroom and real world training in an ambulance bus (Koutitas et al., 2020).
VR Training System The training system in this work is based on the VROnSite platform (Mossel et al., 2021) that supports immersive training of first responder units’ on-site squad leaders. This training platform is fully immersive, entirely untethered to ease use and provides two means of navigation — abstract and natural walking — to simulate physical stress and exhaustion. The development of
Figure 2: Scenario design applied using the iterative approach for training scenario development from Mossel et al. (2021). Scenes were specifically adjusted for the ses- sion on cognitive-emotional strain in both conditions (RC and VR): (a) car accident with injured persons and emission of inflammable liquid in VR, (b) the same scenario presented in RC.
the software has been closely interlocked with stakeholders from multiple fire brigades to gather early feedback in an iterative design process and is comme- rcially available by the Austrian company M2D MasterMind Development GmbH.
Scenario Development In the context of the presented work, VROnSite was used to develop several scenarios that were specifically adjusted for the session on cognitive- emotional strain in both conditions, i.e., RC and VR, respectively (see Figure 2). Four scenarios were prepared by means of videos each with a dura- tion of 2 minutes including virtual drive experience through the scenario. These scenarios were determined to be representative to evaluate the first responders’ situation awareness.
LEDVV Command Scheme Firstly, each video was presented to the operator (phase ‘monitoring’, see Figure 4, Figure 5), then the first responder was informed by the mission lea- der to prepare a situation report in mind (phase ‘anticipating’, duration of one minute), and finally, the first responder reported on the scenario within one minute (phase ‘reporting’). This procedure was designed in parallel to the Trier Social Stress Protocol (Kirschbaum et al., 1993) and through psycho- physiological recordings allowed the assessment of strain. The reports had to be structured following the LEDVV command scheme that represents a stan- dard structure for Austrian first responders. The acronym LEDVV represents the key aspects of ‘L’ for ‘Lage’ (situation awareness), ‘E’ for ‘Entschluss’ (decision making), ‘D’ for ‘Durchführung’ (execution), ‘V’ for ‘Versorgung’ (supply), and ‘V’ for ‘Verbindung’ (communication). The first responder par- ticipants considered the videos highly suitable for training of decision making in complex first responder scenarios.
First Responder Situation Reporting in Virtual Reality Training 77
PSYCHOPHYSIOLOGICAL DATA ANALYTICS In order to draw conclusions from quantitative measures about the psycholo- gical and physiological stress of the subjects within the study, all participants were equipped with wearable biosensors. The acquired bio-signals inclu- ded EDA (electro dermal activity), ECG (electrocardiogram), and respiratory data.
Methodology of EDA-Based Stress Analytics There exist many ways to extract statistical features from an EDA signal that allow conclusions about psychophysiological stress (Cacioppo et al., 2007). Besides features like SCL (Skin Conductance Level), which rely on the tonic part of the EDA, there are those based on the phasic part, like SCR (Skin Con- ductance Response). In this work, we focused on the latter as they proved to be very helpful in our study setting. In particular, we used the freque- ncy of detected SCRs in the raw EDA signal. For his purpose, we counted the number of SCRs within time intervals of interest - e.g., ‘monitoring’, ‘anticipating’, ‘reporting’ - and normalized to SCRs per minute. Since there were no explicitly triggered stimuli in our test setup, we did not distinguish between event-related (specific) and non-specific SCRs. The determination of SCRs was in principle applied according to the recommendations of Boucsein et al. (2012) and Braithwaite et al. (2015). In a first step we low-pass filtered the raw EDA signal to remove any high-frequency interferences and down- sampled the signal from 1,000 Hz to 50 Hz to reduce the computational effort for subsequent filtering and analysis steps. Then, we semi-automatically che- cked for any remaining artefacts. If necessary, artefacts were corrected or excluded from further evaluation. In a next step, we derived the phasic EDA component by subtracting the median-filtered tonic EDA from the unfiltered one. We used a baseline estimation window of 5 seconds and a threshold of 0.02 µs for minimum conductance changes in SCR detection, which lies within the range of 0.01 µS and 0.05 µS proposed in Boucsein et al. (2012).
Analysis of Heart Rate Variability In addition to EDA, ECG-based features can provide information about psychophysiological stress as well. The heart rate (HR) derived from the R-wave in the ECG and in particular, the heart rate variability (HRV) are sui- table for this purpose (Shaffer & Ginsberg, 2017). To determine R-waves and derived characteristics, we recorded a simple and robust 1-lead ECG based on three electrodes. In preliminary tests, we found that conventional labo- ratory electrodes often detach during prolonged sweat-inducing workouts. For this reason, we used Ambu BlueSensor® long-term electrodes with bet- ter adhesive properties. Deriving HR and its mean (HR mean) from ECG is straightforward, as follows. First, we semi-automatically checked the ECG for artefacts and corrected them. Then, we automatically detected the exact peak location of all R-waves, determined the IBIs (inter-beat intervals, i.e., the time between R-waves), and finally derived the mean HR over each time interval of interest. There are several methods for calculating heart rate varia- bility (HRV), all based on the quantitative assessment of differences between
successive IBIs or their statistical spread (Shaffer & Ginsberg, 2017). In our study, we focused on SDNN, the standard deviation of IBIs in a defined time interval.
Finally, other bio-signals, such as, respiratory rate and eye movement fea- tures from VR-based eye tracking may also provide information about a per- son’s psychophysiological stress. However, the results of further investigation are in the context of future work.
EXPERIMENTAL RESULTS Study Design
Participants (N = 13, 11 males, 2 females, age: 17-70, M: 38.85 SD: 15.02) consisted of representative emergency forces of the Austrian volunteer fire brigade of Gumpoldskirchen (n = 7) as well as paramedics of the Johanniter organization Vienna (n = 6). The study plan aimed to induce equivalent strain in both, real environment (RC) and VR-based operation scenarios (VR). The- refore, two identical training procedures were developed, each of which was in turn divided into a (i) cognitive-emotional and (ii) a physical strain block. Both blocks were repeated twice in each procedure. Each participant carried out both training procedures (RC and VR). Figure 3 depicts a summarizing schematic overview of the complete study design. To avoid order effects, a crossover design was used, where participants were allocated to two groups, whereas group A received the first training procedure in RC and the second training procedure in VR, and group B the other way around. The study took ~ 160 minutes for each participant to complete. Both scenarios, RC and VC, consisted of two task blocks. Each block started with a 5-minute inten- sive running session in operational clothing and addition of a backpack of 20 kg payload, followed by a cognitive-emotionally challenging phase. This phase included the LEDVV situation report with (i) the presentation of an emergency scenario with critical events (ES; mediated either by smart TV or by VR headset), (ii) giving the request by the team leader followed by a 30-second anticipation time (AT), and (iii) the presentation of the LEDVV situation report by the first responder (SR).
At the very beginning of the study, each participant completed a 3-minute resting period in a seated position without external disturbance. This phase served as a baseline for later bio-signal evaluations. Finally yet importantly, participants attended psychological tests (PT) that were implemented on a touchscreen tablet, consisting of (i) a 3-minute psychomotor vigilance task test (PVT, following Dinges & Powell, 1985 as well as Basner et al., 2011) as well as (ii) a 3-minute lasting Stroop test (following Golden et al. 1978; with the parametrization of Fennell & Ratcliff, 2019). The first responders attended these tests three times: the first time after the resting phase (‘pre’), the second time after the first scenario (‘inter’) and the third time after the second scenario (‘post’).
Biosensors and Data Analytics The bio-signal information channels included EDA- (electro dermal activity), ECG- (electrocardiogram), and respiratory-based signals. The signals were
First Responder Situation Reporting in Virtual Reality Training 79
Figure 3: Study design with preparation phase (baseline measurements), real and VR- based conditions of the field study, and applications of psychological testing (pre, inter, post).
sampled at 1,000 Hz and recorded using a BIOPAC BioNomadix® system including MP160 multichannel data acquisition station and dedicated wea- rable transmitter/receiver units for each signal channel. Furthermore, eye movements were tracked at 90 Hz during virtual reality tasks by means of a modern virtual reality head mounted display with integrated eye tracker, i.e., the HTC Vive Pro Eye system.
Descriptive Statistics The results of EDA analysis with respect to SCRs/min are depicted in Figure 4, for all participants, i.e., firefighters and paramedics. Since the num- ber of SCRs/min varies dependent on the person, we have normalized it to the respective personal range observed during all sections of interest. In Reality Condition (RC), the medians of SCRs/min are 0.00% for the resting phase, 58.40% (ES), 73.44% (RQ) and 83.99% for the situation report phase (SR). The medians in VR Condition (VC) are 70.77% (ES), 81.83% (RQ), and 75.85% (SR). Note, that we did not evaluate SCRs during running workout (immediately before the ES), because during high physical activity, the physi- cal stress component is dominant, moreover, the EDA signal contains many artefacts. The results of normalized heart rate variability measures based on SDNN are depicted in Figure 5. In Reality Condition (RC), the medians of SDNN are 88.12 % for the resting phase, 40.52% (ES), 0.27% (RQ) and 4.80 % for the situation report phase (SR). The medians in VR Condition (VC) are 36.43% (ES), 5.48% (RQ), and 14.57% (SR).
Inferential statistics We applied the Wilcoxon signed-rank test (Wilcoxon, 1945) as a non- parametric statistical hypothesis test to analyze whether comparable distribu- tions on RC and VC scenarios would be significantly different. The two-sided test compares the locations of two populations using a set of matched samples. It does not assume that the differences between paired samples
Figure 4: EDA-based stress analytics depicted by SCR per minute in % of individual range evaluated on all study phases of interest. It is noted that the baseline section- of-interest (left) starts with lowest stress, monitoring (ES) provokes substantial stress, and anticipating as well as reporting induced a maximum level of stress allover.Figure 5: Cardiovascular data-based stress analytics depicted by heart rate variability (measured by SDNN) in % of individual range evaluated on all study phases of interest. The figures are consistent with the ordering perceived in Figure 4.
are normally distributed which is the case for the requested data. Eventu- ally, the results of applying the test on the referred EDA-based distributions of the corresponding RC and VC conditions, respectively, did not approve the hypothesis to represent two different distributions with sufficient significance level by p=.53 (ES), p=.31 (RQ) and p=.48 (SR).
CONCLUSION
Advanced training of first responders and emergency staff with typical ope- ration scenarios is one of the upcoming challenges of the near future. This work applied a commercially available VR-based training with the purpose
First Responder Situation Reporting in Virtual Reality Training 81
to investigate the cognitive-emotional stress levels in the context of situa- tion reporting under rather realistically simulated mission conditions. We engaged emergency forces of the Austrian volunteer fire brigade and parame- dics of the Johanniter organization in an exploratory pilot study. Each first responder was subject to a test programme that tested a formalized repor- ting schema (LEDVV), inducing equivalent strain in both, real environment training scenarios and VR-based training scenarios.
In summary, the results indicate that LEDVV-based situation reporting both in Reality as well as in Virtual Condition achieved a rather high level of cognitive-emotional stress and should be thoroughly trained. The distri- butions of the EDA-based measurements of stress did not significantly differ between RC and VC conditions. Therefore these results motivate the use of VR environments for the training of stress-resilient decision-making behavior of emergency forces.
ACKNOWLEDGMENT
This work was partly funded from project VR-Responder under grant No 873522 of the Austrian FFG KIRAS program, co-financed by the Austrian Federal Ministry for Climate Action, Environment, Energy, Mobility, Inno- vation and Technology (BMK), and from the project SIXTHSENSE under grant No 883315 of the Horizon 2020 research and innovation program of the European Commission.
REFERENCES
AFAC (2016). “Use of Lookouts, Awareness, Communications, Escape Routes, Safety Zones (LACES) System for Wildfire Firefighters Safety on the Fireground”, Australasian Fire and Emergency Services Authorities Council, AFAC Publication No. 2013, East Melbourne, Vic: Australia. AFAC Ltd. Alexander, M.E.; Thorburn, W.R. (2015). “LACES: Adding an ‘A’ for Anchor point(s)
to the LCES wildland firefighter safety system”, Chapter 4, 121–144, in Leblon, B., Alexander, M.E., eds., Current International Perspectives on Wildland Fires, Mankind and the Environment. Hauppauge, NY: Nova Science Publishers, Inc. Basner M, Rubinstein J. (2011). “Fitness for duty: a 3-minute version of the Psychomotor Vigilance Test predicts fatigue-related declines in luggage- screening performance”, J Occup Environ Med. 2011 Oct; 53(10):1146–54. doi: 10.1097/JOM.0b013e31822b8356. Boucsein, W., Fowles, D.C., Grimnes, S., Ben-Shakhar, G., Roth, W.T., Dawson, M.E., and Filion, D.L. (2012). “Publication recommendations for electrodermal measurements”, Psychophysiology, 49, 1017–1034. Braithwaite, J. J., Watson, D.G., Jones, R., Rowe, M. (2015). “A Guide for Analysing
Electrodermal Activity (EDA) & Skin Conductance Responses (SCRs)”, Techni- cal Report, 2nd version, Selective Attention & Awareness Laboratory (SAAL), Behavioural Brain Sciences Centre, University of Birmingham, UK. Cacioppo, J.T., Tassinary, L.G., Berntson, G.G., eds., (2007). “Handbook of Psychophysiology”, 3rd edition, Cambridge University Press. Cone, D.C., Serra, J., Kurland, L. (2011). “Comparison of the SALT and Smart triage
systems using a virtual reality simulator with paramedic students”, Eur J Emerg Med 18(6):314–21, https://doi.org/10.1097/MEJ.0b013 e3283 45d6f d.
Dinges D.F., Powell J.W. (1985). “Microcomputer analysis of performance on a portable, simple visual RT task during sustained operations”, Behaviour Research Methods, Instruments & Computers, 17, 652–655. Fennell, A., & Ratcliff, R. (2019). “Does Response Modality Influence Con-
flict? Modelling Vocal and Manual Response Stroop Interference”, Journal of Experimental Psychology: Learning, Memory, and Cognition. Advance online publication. http://dx.doi.org/10.1037/xlm0000689. Freeman, K. M., Thompson, S. F., Allely, E. B., Sobel, A. L., Stansfield, S. A., Pugh, W. M. (2001). “A virtual reality patient simulation system for teaching emergency response skills to US Navy medical providers”, Prehospital and Disaster Medicine, 16(1), 3–8. Frye, L.M., Wearing, A.J. (2014). “What were they thinking? A model of metaco-
gnition for bushfire fighters”, in Owen, C., ed., Human Factors in Emergency Management, 57–58. Burlington, VT: Ashgate. Golden, C. J. (1978). “Stroop Color and Word Test: A Manual for Clinical and Experimental Uses”, Wood Dale, Illinois: Stoelting Company. Kirschbaum, C., Pirke, K.M., & Hellhammer, D.H. (1993). âŁ~The ‘Trier Social
Stress Test’–a tool for investigating psychobiological stress responses in a labora- tory setting”, Neuropsychobiology, 28(1-2), 76–81. Koutitas, G., Smith, S., Lawrence, G. (2020). “Performance evaluation of AR/VR training technologies for EMS first responders”, Virtual Reality, https://doi.org/ 10.1007/s10055-020-00436--8. Kurenov, S.N., Cance, W.W., Noel, B., Mozingo, D.W. (2009). “Game-based mass
casualty burn training”, Studies Health Technol Inf 142:142–144. Mills, B., Dykstra, P., Hansen, S., Miles, A., Rankin, T., Hopper, L., Brook, L., Bartlett, D. (2019). “Virtual reality triage training can provide comparable simulation efficacy for paramedicine students compared to live simulation-based scenarios”, Prehospital Emerg Care. https://doi.org/10.1080/10903127.2019.1676345. Mossel, A., Schoenauer, C., Froeschl, M., Peer, A., Goellner, J., Kaufmann, H. (2021).
“Immersive training of first responder squad leaders in untethered virtual reality”, Virtual Reality, 25:745–759. ÖBFV (2016). “Heft 122 – Der Feuerwehreinsatz”, Österreichischer Bundesfeuerwehrverband, https://www.bundesfeuerwehrverband.at/heft-122/. Putman, T. (1995). “The collapse of decision making and organizational structure
on Storm King Mountain”, Wildfire, 4(2), 40–45. Salahuddin, L., Cho, J., Jeong, M.G., Kim, D. (2007). “Ultra short term analysis of heart rate variability for monitoring mental stress in mobile settings”, Conf Proc IEEE Eng Med Biol Soc 2007:4656–9. Shaffer, F., & Ginsberg, J.P. (2017). “An Overview of Heart Rate Variability Metrics
and Norms”, Front. Public Health 5:258 doi: 10.3389/fpubh.2017.00258. Wilcoxon, F. (1945). “Individual Comparisons by Ranking Methods”, in: Biometrics Bulletin, 1, 1945, 80–83, JSTOR 3001968.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Cognitive Computing and Internet of Things
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-19-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional