Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Performance of Fire Fighters in a Multiple-Sensory VR Pump Panel Training Simulation

Department of Systems Engineering Colorado State University, Fort Collins, CO, 80523, USA ↗ ↗
Publication year
Digital Object Identifier10.54941/ahfe1005684

Abstract

Virtual reality (VR) simulations have emerged as valuable tools for training and education, allowing learners to engage with realistic virtual environments. This study investigates a novel VR training simulation that integrates visual, audio, haptic, and olfactory cues. Participants (N=54) learned a specific sequence of interactions for operating the Super Soaker 5000 pump panel and were subsequently tested on their ability to replicate these steps. Demographic data and sense of presence questionnaires were collected. The research reveals intriguing findings. While the combined sensory conditions enhanced participants’ sense of presence, they did not significantly impact final pump panel test performance. However, an essential factor emerged: the amount of time spent during the learning phases significantly affected final test outcomes. These results prompt further exploration into optimizing training duration and refining the integration of sensory modalities in VR simulations. This paper discusses the study’s results, the implications for training effectiveness, and the need for future research. By understanding how sensory cues influence learning outcomes, educators and practitioners can enhance VR training experiences and better prepare learners for real-world scenarios.

INTRODUCTION

Virtual reality (VR) simulations have become increasingly popular as a training and education tool. These simulations allow learners to interact with virtual environments in a realistic way, providing a unique opportunity to practice skills and gain knowledge in a safe and controlled setting. VR simulations can train individuals in various fields, including healthcare, military, and aviation.

One of the critical benefits of VR simulations is the ability to provide immediate feedback and assistance to learners. This can be achieved using sensors and tracking devices that allow the simulation to respond to the learner’s actions in real-time. Additionally, VR simulations can provide expert modeling and coaching, helping learners develop their skills and improve their performance.

© 2024. Published by AHFE Open Access. All rights reserved. 1167

1168 Ledgerwood and Gallegos

Another important aspect of VR simulations is the ability to integrate multiple sensory modalities, allowing learners to experience the virtual environment in a more immersive and realistic way. This can be achieved through head-mounted displays, olfactory devices, and haptic feedback systems. By providing a more immersive and realistic training experience, VR simulations can help learners retain information better and develop their skills to succeed in their field (Boller, Ouellet, and Belleville, 2021; Duan et al., 2023; Moulaei et al., 2024).

Research has shown that multimodal integration can enhance the effectiveness of VR training (Doumanis, 2019). For instance, integrating haptic feedback in VR has greatly improved learners’ learning performance and efficiency (Zhai et al., 2022). Similarly, olfactory feedback in VR environments has been suggested to enhance the realism and immersion of the training experience (Richard et al., 2006).

While the benefits of individual modalities like haptics and olfaction in VR training have been studied, there is a lack of comprehensive research examining the combined effects of these modalities. Research combining haptics and olfaction in VR training could contribute significantly to the field. It could lead to developing more effective and immersive VR training simulations, potentially enhancing short-term memory performance and overall learning outcomes. This could have wide-ranging applications in education, rehabilitation, and professional training.

To understand these effects, we conducted a study from August 31 to November 3, 2023 with 61 firefighters. The reference condition, hereafter referred to as baseline, included visual and audio elements. The test conditions used a multimodal approach integrating the baseline with haptic feedback, olfactory, and a combination of all modalities.

The study aimed to determine if integrating multiple senses into a VR simulation effected training outcomes and sense of presence. The sensory integrations were not tied directly to a specific interaction or learning outcome but used as a persistent element experienced throughout the training and testing scenarios. The results could be used to design and develop future VR training experiences with sensory integration.

RELATED WORK

In VR, using multiple senses has been found to be important for improving learning and task performance. Research has shown that incorporating visual, auditory, and haptic feedback can make the VR experience more realistic and immersive, helping users better understand and execute tasks. For example, a recent study by Marucci et al. (2021) found that multisensory feedback improved users’ response time compared to haptics or visual feedback alone. Other research has shown that olfactory cues combined with force feedback improved memory 2 hours after receiving the VR training (Xie et al., 2021). However, multisensory integration’s effectiveness depends on the task’s nature and the individual’s sensory preferences, so further research is needed to optimize the design of VR environments for task learning.

Several factors have been identified in the literature to make VR training effective, including integrating multiple senses (e.g., haptics and olfactory

Performance of Fire Fighters in a Multiple-Sensory VR Pump Panel Training Simulation 1169

feedback), using props, interaction between trainer and trainee, and social interactions. For example, a study by Durai, Arjunan, and Manivannan (2019) found that integrating audio, visual, and haptic feedback in VR improved cardiopulmonary resuscitation (CPR) training performance.

Olfactory in VR The use of scent in VR has been around for many years. Delivery methods for odor displays have varied depending on the application and included airflow, vortex ring, natural diffusion, or direct injection (Yanagida, 2012). The introduction of commercial wearable devices that can mount to a VR head- mounted display (HMD) has made olfactory integration easier to deploy across applications.

The use of scent in VR training has shown some promise, with a study conducted by Dozio et al. (2021) showing that scent cues can improve performance in visual search tasks, Similarly, research by Halabi et al. (2013) found that the addition of scent in a surgical training application improved memory enhancement.

Haptics in VR There are many different types of tactile haptic feedback devices used in VR simulations, including vibrotactile, electrotactile, thermal, and force feedback. The integration of haptics into VR training has shown improved skill transfer and performance across basic interactions such as object translation, camera navigation, and even more complex tasks like laparoscopic surgery. devices have been shown to be of the learning process (Gani et al., 2022). A systematic review by Rangarajan, Davis, and Pucher (2020) found of the eight VR surgical trainings comparing haptics vs. non-haptics, six showed a reduced learning curve and faster time to completion when, while the other two showed no difference. The role of haptics highlights the potential in VR training but inconsistencies across studies leave room for continued research on the appropriate use case and application.

Summary As noted in multiple experiments discussed above, the addition of haptics or olfactory to an audiovisual VR experience has improved performance. However, limited research has looked at combining both haptics and olfactory to determine their multisensory effects on the immediate performance outcome in training and sense of presence. In addition, the studies that did combine olfactory and haptics had few subjects. To address these gaps, our study compares the different sensory combinations and the impact they have on the ability of a participant to learn a task sequence and their sense of presence in a VR scene.

EXPERIMENT A VR training simulation that integrated visual, audio, haptic, and scent was developed for this study. The training was set up such that a user would

1170 Ledgerwood and Gallegos

learn a particular sequence of interactions for operating the Super Soaker 5000 pump panel and then test the user’s ability to repeat that sequence of steps. Finally, the users responded to demographic and sense of presence questionnaires. The study used a four-factorial between-subjects design with sensory combination as the independent variable. Additional demographics were collected for age and gender.

Data collection was conducted at two different fire departments. Each station provided a private room for the participants. Participants stood in a designated area while using the VR HMD so that the controls on the Super Soaker 5000 Pump Panel were placed near their waist level. The study had approval from the Colorado State University IRB (protocol #4786).

Pump Panel The sequence of steps for correctly using the Super Soaker 5000 pump panel included eight interactions. The panel interactions included flipping levers, rotating knobs, pressing buttons, and moving sliders. The number of steps used for the sequence was based on research by Miller (1956) showing short-term memory capacity for people ranging from five to nine interactions.

To remove potential bias from prior training or operating experience from the firefighters, the components of the pump panel were assigned arbitrary names that intentionally did not align with a traditional fire engine pump panel. These interaction tasks included statements like “Step 2: Press the ‘Gork Button’ to engage the Gork mode”or “Step 6: Adjust the ‘Move Ratio’ control to achieve the desired ratio setting of 1:10.”

Each interaction was coded with a numerical value on the backend. As the participants interacted with the panel, the associated number for that knob, switch, lever, slider, or button was appended to a sequence variable in the data log. That sequence was compared to the correct sequence for the final test.

Apparatus A Meta Quest 2 was used for the VR HMD. The Meta Quest 2 controllers were programmed to provide haptic feedback (i.e., vibrations) for the different interactions with the pump panel. The controller vibrations were only enabled for the conditions that contained haptics. A wearable odor display was used to present a scent to the users (see Figure 1). The device used a water-based scent solution containing up to 5% fragrance, 1% of Benzisothiazolinone, and a surfactant. While the scent cartridge could hold up to nine fragrances, a unit was developed specifically for this study which contained only the body odor scent. This scent was previously tested in our lab to determine the intensity and frequency levels for detection with 34 users (Ledgerwood, Gallegos, and Marie, 2024). The scent configuration for this experiment set intensity at 600ms bursts, which was identified in prior testing to be detectable at least 60% of the time.

Performance of Fire Fighters in a Multiple-Sensory VR Pump Panel Training Simulation 1171

Figure 1: VR HMD with wearable odor display.
Original PDF Figure 1:

Experimental Design Data analysis was conducted to predict [dependent variables] pump panel performance and sense of presence. Pump panel performance was based on the sequence entered during the final test by the participant. An exact match for the pump panel performance was coded as a binary variable of success (1) or failure (0). The sense of presence felt by the participant was measured using the igroup presence questionnaire (IPQ). The IPQ contains 14 questions aligned to presence, realism, spatial-realism, and involvement variables (Schubert, Friedmann, and Regenbrecht, 2001).

The study used a four-factorial between-subjects design with sensory combination as the independent variable.

  • B: Baseline condition of the training simulation with written and audio instructions that informed the user which lever, knobs, sliders, and buttons to press. Upon successful interaction with a button on the Super Soaker 5000, a green light illuminated on the panel. Text labels were provided for all panel components with a visual indicator indicating “On”, “90
  • ”, “10”, or other values for the targeted success case.
  • B+H: Baseline (B) + haptics, the Meta Quest 2 controllers were configured to provide vibrotactile haptic feedback (i.e., vibrations) when the user interacted with the pump panel components.
  • B+S: Baseline (B) + scent, the wearable olfactory display mounted on the HMD disbursed a 1-second burst of scent every 30 seconds.
  • B+H+S: Baseline (B) + haptic feedback from controllers (H) + scent disbursements from wearable olfactory display (S).

The users were randomly assigned to one of the four conditions. The researcher launched the appropriate VR pump panel simulation based on the condition assigned. For the conditions containing scent, the wearable olfactory display was mounted onto the base of the Meta Quest 2 headset. The number of participants across the conditions is shown in Table 1.

1172 Ledgerwood and Gallegos

Table 1. Participant count by condition.

Table available in the original PDF paper.

Condition Number of Participants Base 13 Base + Haptic 14 Base + Scent 13 Base + Haptic + Scent 14

Additionally, the analysis accounted for these additional variables for their effects on pump panel performance and sense of presence:

  • Total Practice Time: total duration (seconds) that the participant spent across the three training rounds.
  • Total Test Time: total duration (seconds) that the participant spent on the final test.
  • Age: categorized as 26–35, 36–45, or 46–55.

Participants There were 61 firefighter participants recruited for this study. Participants were recruited through emails from local fire departments in the greater Denver, Colorado area. Exclusion criteria for the study excluded participants under 18, over 55 due to diminished ability to detect odor stimuli, and pregnant women due to potentially heightened sense of smell (Doty et al., 1984). Of the 61 participants, 14 records were removed from the study due to participant withdrawal, lack of consent, or missing information. A total of 54 participants were included in the analysis, including 3 females and 51 males. The participants’ ages are shown in Table 2.

Table 2. Participant age.

Table available in the original PDF paper.

Age Number of Participants Proportion 26 - 35 16 30% 36 - 45 27 50% 46 - 55 11 20%

Procedure Users were informed of risks associated with the study, including cybersickness, vertigo, and collisions with objects while wearing the virtual reality headset. These risks were mitigated by conducting the study in a controlled environment, using a stationary boundary (e.g., not allowing users to move outside a 1-meter radius) and by ensuring the researcher was present to prevent walking into objects. The participants were also informed that they may stop participating at any time, and if they experienced cybersickness or discomfort, they should take off the HMD.

The users were verbally instructed on how to use the controller to interact with the initial questions and the levers, knobs, sliders and buttons on the

Performance of Fire Fighters in a Multiple-Sensory VR Pump Panel Training Simulation 1173

pump panel. After completing three rounds of instruction, participants were prompted to take the final test. The test requested the participants to complete the sequence of interactions for operating the Super Soaker 5000 Pump Panel and then hit the “Engage” button. The final test was performed without the assistance of the written or audio guidance. If participants mistakenly performed an interaction (e.g., flipping the “Power Tilt” lever to “On”) they could undo that interaction in the next step (e.g., flipping to “Off”) without being penalized and proceed forward with the sequence.

Questionnaires on demographics and sense of presence were presented inside the VR simulation at the end of the study. All data was captured and transmitted to a Google firebase database. The data contained a random participant ID, timestamps for training time and test time, the sequence numbers for the final test, and the response to the questions. No personally identifiable information was collected or transmitted during the experiment. Participation took approximately 15-minutes for the experiment and 5-minutes for reviewing the informed consent and learning how to use the Meta Quest 2 device.

Within the VR space, participants were in an open area with the Super Soaker 5000 Pump Panel located in front of them with a menu screen located above (see Figure 2). The menu screen informed the user about the study and prompted them to click “Next” to begin. A sequence of menus then guided the user through the initial onboarding, where they practiced the different interactions, responded to the consent form, and then began the three rounds of instruction. If the user did not agree to the consent form the data was not used in the analysis. No identifiers were linked to the users and all data was anonymous.

Figure 2: Super Soaker 5000 pump panel.
Original PDF Figure 2:

After the test was completed, the participants were asked to provide their gender and age. The users then completed the IPQ within VR simulation to assess if the different sensory combinations affected their sense of immersion, presence, or the realism of the VR simulation.

1174 Ledgerwood and Gallegos

The IPQ is a measurement scale constructed using a large pool of items and 2 survey waves with approximately 500 participants. The IPQ consists of three subscales (i.e., spatial presence, involvement, and realism) and one additional general item not belonging to a subscale (i.e., presence). The three subscales emerged from principal component analyses and can be regarded as independent factors. The IPQ was selected based on being a validated instrument used to measure presence in different virtual environments, including video games, virtual reality, and telepresence systems variables (Schubert, Friedmann, and Regenbrecht, 2001).

Results

In this section, we present the statistical analysis for the pump panel test performance and the sense of presence based on the different sensory conditions. The R statistical software program (version 4.3.1) was used for data cleaning and analysis. Statistical significance was assessed at α < 0.05.

The variable gender was not included in the analysis due to only three participants selecting female. The imbalance in the sample could present a bias and would not be generalizable to a larger group.

Pump Panel Performance The exact sequence match on the final test was modeled using a generalized linear model with the logit link function (i.e., binary logistic regression) with no responses omitted. Results are shown in Table 3. There were significant effects on total practice time (p = .013) and participants aged 36–45 (p = .040). None of the multisensory conditions had a significant effect on the final test.

Table 3. Summary of model output for predicting a successful test.

Table available in the original PDF paper.

Variable Odds Ratio SE z-Value p-Value Intercept 0.013 2.353 –1.832 .067 (ns) B+H (ref. Baseline) 0.580 1.058 –0.514 .607 (ns) B+H+S (ref. Baseline) 0.765 1.118 –0.239 .811 (ns) B+S (ref. Baseline) 0.280 1.228 –1.036 .300 (ns) Total Test Time 0.966 0.028 –1.243 .241 (ns) Total Practice Time 1.026 0.010 2.487 .013 Age 36–45 (ref. 26–35) 0.130 0.993 –2.056 .040 Age 46–55 (ref. 26–35) 0.152 1.136 –1.657 .097 (ns)

The variable age was further evaluated; specifically, Figure 3 shows the effect of a change in age group using an estimated marginal means plot, where a significant drop in likelihood to get an exact match on the pump panel test can be seen from the youngest age group to the middle age group.

Performance of Fire Fighters in a Multiple-Sensory VR Pump Panel Training Simulation 1175

Figure 3: Estimated marginal means plot for age.
Original PDF Figure 3:

Sense of Presence Measures of IPQ were evaluated using repeated multivariate measures analysis of variance (MANOVA) and individual ANOVAs to assess if the different modalities or their combinations had an effect on the level of presence felt by the participant. Tukey HSD post hoc tests were used to further assess significant factors.

The dependent variables were realism, involvement, presence, and spatial- realism assessed by the IPQ. The independent variables included exact match, condition, total practice time, age, and gender. The MANOVA showed a significant effect on the exact sequence match F(4, 42) = 2.774, p = .039.

The ANOVAs were run for each of the dependent variables with the same independent variables as the MANOVA. There were significant effects for presence on exact match F(1, 45) = 9.878, p = .00296 and realism for age F(2, 45) = 3.498, p = .039. The IPQ variables involvement and spatial-realism did not have any significant effects.

DISCUSSION

The results of this study show that there were variables that effected participant performance on the pump panel training task. However, the specific conditions (i.e., B, B+H, B+S, and B+H+S) for multisensory did not have significant effects. The performance on the pump panel was effected by the amount of time the participant spent when learning the sequence across the three training rounds (p = .013) and if they were in the age group 36– 45 (p = .040). The result was a 1.026 higher odds ratio for an increase of 1-second spent in the training modes and a 0.87 lower odds ratio for a change to the age group 36–45 from 26–35.

The sense of presence based on the IPQ also did not yield significant effects based on the sensory conditions but did so for a participant’s ability to successfully complete the final test with the correct sequence (p = .039). The users who felt a higher sense of presence, regardless of the condition, performed better on the final pump panel test. Looking at the individual subsets of the IPQ, we also saw realism (p = .00296) and an exact match on the final test with presence (p = .003).

1176 Ledgerwood and Gallegos

Limitations A limitation of the commercially available, wearable olfactory display was the types of scents available. Further, a previous study identified intensity levels for the detection of scent above certain thresholds, but that scent was incongruent with the task of performing the pump panel operation. Future work could integrate scents that are more congruent with the primary task to determine if the performance impacts would be different.

The use of VR controllers allowed for a seamless integration of haptics to the training simulation, however, the haptics were limited to vibrations. The use of a more elaborate haptic system, such as a force feedback device, for the types of interactions used in this study may have resulted in a more pronounced effect.

This study was built around first responders to understand what effects multisensory VR had on their sense of presence and performance in training. While 61 firefighters participated, the data was skewed from a gender perspective, with only three participants reporting female. Future studies could benefit from a greater distribution across genders to see if there was an impact on the effects.

CONCLUSION

In this paper, we presented a user study comparing firefighters’ sense of presence and ability to learn a task under different multisensory conditions in a VR training simulation. The study included four different conditions: (B) baseline audiovisual representation, (B+H) baseline plus haptic feedback through Meta Quest 2 controllers mapped to pump panel interactions, (B+S) baseline plus olfactory where a background scent was disbursed every 30 seconds through a wearable olfactory display, and (B+H+S) all conditions combined.

The results from this study showed no difference in the multisensory integrations on training and sense of presence outcomes. However, we did see that there were significant effects based on the amount of time a participant spent in the learning stages of the VR pump panel trainer. Based on these findings, there may be value for educators and trainers to prolong the amount of time users spend under different training conditions.

ACKNOWLEDGMENTS

The authors want to thank Boulder Fire Rescue, Longmont Fire Department, and Mountain View Fire Rescue for their support.

REFERENCES

Boller, B., Ouellet, E. and Belleville, S. (2021) ‘Using virtual reality to assess and promote transfer of memory training in older adults with memory complaints: A randomized controlled trial’, Frontiers in Psychology, 12. Doty, R. L., Shaman, P., Applebaum, S. L., Giberson, R., Siksorski, L. and

Rosenberg, L. (1984) ‘Smell identification ability: Changes with age’, Science, 226, pp. 1441–1443.

Performance of Fire Fighters in a Multiple-Sensory VR Pump Panel Training Simulation 1177

Doumanis, I., Economou, D., Sim, G. R. and Porter, S. (2019) ‘The impact of multimodal collaborative virtual environments on learning: A gamified online debate’, Computers & Education, 130, pp. 121–138. Dozio, N., Maggioni, E., Pittera, D., Gallace, A. and Obrist, M. (2021) ‘May I smell

your attention: Exploration of smell and sound for visuospatial attention in virtual reality’, Frontiers in Psychology, 12. Duan, D., Wu, Z., Zhou, Y., Wan, X. and Wen, D. (2023) ‘Working memory training and evaluation based on brain-computer interface and virtual reality: Our opinion’, Frontiers in Human Neuroscience, 17. Durai, V. S. I., Arjunan, R. and Manivannan, M. (2019) ‘The effect of audio and

visual modality based CPR skill training with haptics feedback in VR’, IEEE Conference on Virtual Reality and 3D User Interfaces, pp. 910–911. Halabi, O., Al-Mesaifri, F., Al-Ansari, M., Al-Shaabi, R., Barki, H. and Foufou, S.

(2013) ‘Incorporating haptic and olfactory into surgical simulation’, International Conference on Cyberworlds, pp. 52–55. Gani, A., Pickering, O., Ellis, C., Sabri, O. and Pucher, P. (2022) ‘Impact of haptic feedback on surgical training outcomes: A Randomised Controlled Trial of haptic versus non-haptic immersive virtual reality training’, Annals of Medicine and Surgery. Ledgerwood, S., Gallegos, E. and Vans, M. (2024) ‘Measuring detection and

habituation of olfactory stimuli in virtual reality for improved immersion’, AHFE International Conference, 137. Marucci, M., Di Flumeri, G., Borghini, G., Sciaraffa, N., Scandola, M., Pavone, E. F., Babiloni, F., Betti, V. and Arico, P. (2021) ‘The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence’, Scientific Reports, 11. Miller, G. (1956) ‘The magical number seven, plus or minus two: Some limits on our

capacity for processing information’, The Psychological Review, 63, pp. 81–97. Moulaei, K., Sharifi, H., Bahaadinbeigy, K. and Dinari, F. (2024) ‘Efficacy of virtual reality-based training programs and games on the improvement of cognitive disorders in patients: A systematic review and meta-analysis’, BMC Psychiatry, 24(1). Rangarajan, K., Davis, H. and Pucher, P. H. (2020) ‘Systematic Review of Virtual

Haptics in Surgical Simulation: A Valid Educational Tool?’, Journal of Surgical Education, 77(2), pp. 337–347. Richard, E., Tijou, A., Richard, P. and Ferrier, J. L. (2006) ‘Multi-modal virtual environments for education with haptic and olfactory feedback’, Virtual Reality, 10(3-4), pp. 207–225. Schubert, T., Friedmann, F. and Regenbrecht, H. (2001) ‘The experience of presence:

Factor analytic insights’, Presence: Teleoperators and Virtual Environments, 10(3), pp. 266–281. Xie, B., Liu, H., Alghofaili, R., Zhang, Y., Jiang, Y., Lobo, F. D., Li, C., Li, W., Huang, H., Akdere, M., Mousas, C. and Yu, L.-F. (2021) ‘A review on virtual reality skill training applications’, Frontiers in Virtual Reality, 2. Yanagida, Y. (2012) ‘A survey of olfactory displays: Making and delivering scents’,

IEEE SENSORS, pp. 1–4. Zhai, X., Sun, Y., Wang, M., Asmi, F., Cai, W. and Chu, X. (2022) ‘Exploring the effect of virtual reality with haptics on educational research: A meta-analysis from 2010 to 2020’, 8th International Conference of the Immersive Learning Research Network (iLRN), pp. 1–7.

Cover of the proceedings volume Human Factors in Design, Engineering, and Computing

Explore the complete digital volume

Human Factors in Design, Engineering, and Computing

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-35-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →