Emre Eraslan, Avinash Gupta (2025). Toward Human-Centered Swarm Control: A VR-based UAV Simulator for Training and Cognitive Evaluation. In: Human Factors in Design, Engineering, and Computing. AHFE Open Access, vol. 199, pp. 915–923. https://doi.org/10.54941/ahfe1006904
This paper introduces a virtual reality (VR)-based unmanned aerial vehicle (UAV) simulator designed to support immersive training and early-stage human factors evaluations in rescue and emergency mission contexts. The system simulates a leader-follower UAV configuration, where a human operator controls a lead drone through a VR headset and joystick, while autonomous drones maintain any geometric formations using onboard sensing and dynamic obstacle avoidance [1]. This swarm-based coordination reflects real-world search and rescue scenarios, where rapid decision-making, spatial awareness, and teamwork are essential.The simulator is developed with a focus on human-in-the-loop design, providing an immersive teleoperation experience that places the user in high-pressure environments with dynamic spatial constraints. Visual and auditory feedback, along with the ability to switch between multiple drone perspectives, is intended to support situational awareness, mission control, and error recovery in complex terrain.From a human factors perspective, the simulator serves as a flexible testbed for evaluating cognitive and ergonomic variables in safety-critical tasks. It allows the assessment of operator workload, interface usability, attention allocation, and situation awareness under realistic but controlled conditions. Subjective evaluation tools such as the NASA Task Load Index (NASA-TLX) and the Situation Awareness Rating Technique (SART) [2] can be embedded directly within the VR experience. At the same time, objective data from head and hand movement, control inputs, and UAV performance are recorded to gain insight into operator behavior.The simulator integrates high-fidelity UAV dynamics using closed-loop reference model adaptive controllers [3], and can be equipped with tools such as eye tracking, physiological sensors for workload estimation, and visual attention assessment methods like ATTENDO [4]. These additions support deeper analysis of how operators acquire information, prioritize tasks, and shift attention during mission-critical events.Overall, the VR-based UAV simulator addresses a growing need to evaluate and enhance human performance in multi-agent control systems used in emergency operations. It offers a scalable, immersive environment that supports both training and cognitive engineering. Its modular structure allows for future integration of real UAVs, digital mapping capabilities, and adaptive interfaces, making it a valuable platform for advancing human-centered UAV system design.References [1] Saunders, J., Call, B., Curtis, A., Beard, R., & McLain, T. (2005). Static and dynamic obstacle avoidance in miniature air vehicles. In Infotech@ Aerospace (p. 6950).[2] Braarud, P. Ø. (2021). Investigating the validity of subjective workload rating (NASA TLX) and subjective situation awareness rating (SART) for cognitively complex human–machine work. International Journal of Industrial Ergonomics, 86, 103233.[3] Eraslan, E., & Yildiz, Y. (2021, December). Modeling and adaptive control of flexible quadrotor uavs. In 2021 60th IEEE Conference on Decision and Control (CDC) (pp. 1783-1788). IEEE.[4] Oberhauser, M., & Dreyer, D. (2017). A virtual reality flight simulator for human factors engineering. Cognition, Technology & Work, 19, 263-277.
Unmanned Aerial Vehicles (UAVs) have become increasingly important in rescue and emergency missions due to their ability to access hazardous environments, provide real-time situational data, and support time-critical decision-making (Murphy and Stover, 2008). Their flexibility, speed, and aerial perspective make them valuable assets in disaster response, search and rescue, and remote reconnaissance operations (Floreano and Wood, 2015). As mission complexity increases, the demand for skilled UAV operators who can manage not only single-drone control, but also multi-agent coordination becomes essential.
Despite the widespread adoption of UAVs, existing training tools often lack realism, immersive feedback, or the ability to simulate complex swarm behaviors (Khan, Siddique and Lee, 2021). Many platforms rely on simplified
physics models, two-dimensional user interfaces, or pre-scripted scenarios that fail to capture the dynamic conditions and human decision-making demands found in real rescue operations. These limitations restrict the development of critical cognitive skills such as spatial awareness, attention management, and adaptive control under pressure (McCarley and Wickens, 2005). To address these gaps, immersive simulation environments are emerging as promising tools for both training and human factors research (Cummings, 2014). Virtual reality (VR) systems offer the ability to simulate high-stakes scenarios with rich sensory feedback and support for naturalistic operator input. When combined with principles of human-centered design, such systems can improve the evaluation of workload, usability, and performance in controlled but realistic settings (Oberhauser and Dreyer, 2017).
This paper presents the design of a VR-based UAV simulator developed specifically to support human-in-the-loop operation of swarm-based UAV systems. The simulator enables a user to control a lead drone within a multi- agent formation, with autonomous follower drones capable of maintaining geometric patterns and dynamically avoiding obstacles. The system integrates high-fidelity control models with immersive 3D visualization and supports real-time interaction through a VR headset and joystick. The simulator is intended to serve as both a training platform and a research testbed for evaluating cognitive performance, decision-making, and interface design in mission-critical UAV operations.
BACKGROUND AND RELATED WORK Operating UAVs requires sustained attention, spatial reasoning, and rapid decision-making in dynamic environments. As UAV systems grow more autonomous and complex, human operators are increasingly expected to supervise multiple agents while maintaining situational awareness, minimizing error, and adapting to task demands. Research has shown that excessive workload, interface complexity, and poor feedback design can significantly impair operator performance in remote UAV missions (Parasuraman, Cosenzo and De Visser, 2009; Wright, Decrements and Cummings, 2013). Factors such as delayed feedback, lack of sensory immersion, and limited control transparency contribute to reduced situational awareness, which in turn increases the risk of failure during critical mission phases (Endsley, 1995).
VR platforms have gained attention in the human factors community as tools for simulating cognitively demanding tasks in immersive and controllable environments. VR has been applied successfully in domains such as aviation, driving, and emergency response to evaluate training effectiveness, mental workload, and interface usability (Jerald, 2015; Stone, 2017). These systems offer high ecological validity and allow researchers to manipulate task complexity, environmental hazards, and interface characteristics while collecting detailed behavioral and physiological data.
Several UAV simulation tools have emerged to support operator training and system prototyping. However, many of these platforms are limited to
Toward Human-Centered Swarm Control: A VR-Based UAV Simulator 917
desktop interfaces, non-immersive displays, or simplified mission logic. While commercial flight simulators (e.g., X-Plane, DJI Flight Simulator) provide basic piloting experience, they often lack support for swarm behaviors, adaptive control models, or human factors instrumentation. Research- focused simulators have begun to incorporate collaborative control and scenario-based training but typically focus on individual drone operation or autonomous navigation rather than team-based mission execution (Zhang et al., 2020).
Human factors researchers have long relied on standardized tools to assess mental workload and system usability in simulation studies. Common subjective instruments include the NASA Task Load Index (Hart and Staveland, 1988) and the Situation Awareness Rating Technique (Taylor, 1990). These are often complemented by objective metrics such as task completion time, error rate, and eye or motion tracking to evaluate attention allocation and cognitive strategy (Salas et al., 2008).
Despite these advances, few VR-based UAV simulators have been designed from the ground up to support both immersive training and systematic evaluation of operator performance. There remains a need for flexible, modular systems that integrate high-fidelity UAV models with human-in-the- loop interfaces and embedded measurement tools. This project addresses that gap by developing a virtual simulator that bridges cognitive evaluation and swarm-based UAV control in a unified manner.
SYSTEM DESCRIPTION AND INTERACTION DESIGN This virtual UAV simulator was developed to support immersive operator training and human performance evaluation in swarm-based rescue missions. The system is designed with a human-in-the-loop approach, emphasizing realistic task conditions and interaction under pressure.
Users control a lead UAV through a VR headset and joystick, while autonomous follower UAVs maintain formation and avoid obstacles. The interface supports real-time audiovisual cues and dynamic perspective switching to enhance situational awareness and control transparency. The simulator prioritizes intuitive interaction, mission realism, and modularity. It is intended both as a training tool and a flexible testbed for studying workload, attention, and usability in complex UAV operations. In what follows, we will investigate the system architecture, human-machine interface design, and autonomous swarm control behavior.
System Architecture The simulator integrates Unity and MATLAB to combine immersive 3D visualization with high-fidelity UAV control. Figure 1 represents the system architecture in a detailed manner. MATLAB is responsible for all control logic and dynamic modeling, including a nonlinear UAV model and a closed- loop reference model (CRM) adaptive controller. CRM adaptive control enables stable and robust tracking performance in the presence of system uncertainties and disturbances by shaping the transient response of the adaptive system (Annaswamy and Lavretsky, 2013; Lavretsky and Wise,
918 Eraslan and Gupta
2013). This approach has been successfully applied in previous UAV control research and serves as the foundation for the adaptive element in this simulator (Eraslan and Yildiz, 2021; Eraslan and Yildiz, 2024).
Unity serves as the immersive 3D visualization environment, rendering UAV positions, formation behavior, and terrain features in real time. A continuous data link between MATLAB and Unity ensures synchronized simulation between control computation and visual feedback, while avoiding the limitations of black-box physics engines.
Figure 1: System architecture of the VR-based UAV simulator: the human operator has a VR headset on and utilizes a pilot joystick to give commands to the leader UAV.
The system is operated using a Meta Quest 2 VR headset and a Logitech Extreme 3D Pro joystick. This hardware setup allows users to control the lead UAV through intuitive manual inputs and receive immersive, head-tracked visual feedback during simulation.
Human-Machine Interface Design The simulator’s interface was developed to support intuitive control and real-time situational awareness in complex mission settings. Visual feedback includes spatial positioning, drone orientation, and proximity to environmental obstacles, rendered in an immersive 3D environment. Color- coded status indicators and formation markers help the operator track swarm behavior and detect deviations.
Toward Human-Centered Swarm Control: A VR-Based UAV Simulator 919
Auditory feedback complements visual cues by providing spatial alerts and status changes. These cues are designed to offload visual demand and maintain awareness of critical system states, such as obstacle encounters or formation re-entry events.
To support perception across the swarm, users can switch between multiple onboard drone perspectives. Each UAV is equipped with a virtual camera feed, allowing the operator to alternate between first-person and third- person views of the lead and follower drones. This multi-angle feedback enables better understanding of swarm motion and obstacle dynamics. The VR headset incorporates head-tracked control, allowing users to naturally scan the environment while receiving synchronized visual updates. Together with the joystick, this setup creates a closed-loop feedback system in which the operator’s physical orientation directly affects awareness and task performance.
Autonomous Swarm Control Behavior The simulator models a leader-follower swarm configuration consisting of three UAVs: one human-controlled leader and two autonomous followers. The operator pilots the lead UAV in real time using a joystick and VR headset, while the follower UAVs maintain a fixed formation, such as a triangle, based on the leader’s position and heading.
Figure 2: An example of the formation flight and obstacle avoidance. The follower UAVs (pink and green) are equipped with an obstacle avoidance algorithm. In the vicinity of on obstacle, the follower UAVs momentarily break out of the formation, and the obstacle avoidance algorithm is activated.
920 Eraslan and Gupta
When the operator-controlled UAV encounters an obstacle, the user is responsible for avoiding it manually. In parallel, the autonomous follower UAVs are equipped with onboard obstacle avoidance algorithms that allow them to detect and respond to hazards independently. If a follower detects a nearby obstacle, it temporarily exits the formation, navigates around the obstruction, and returns to its assigned position once the path is clear. This behavior is consistent with principles seen in multi-sensor collision avoidance systems used in autonomous aerial platforms (Fasano et al., 2008). See Figure 2 for an example of dynamic obstacle avoidance and reformation behavior.
This swarm coordination structure allows the human to remain focused on strategic navigation while the autonomous agents manage local-level responsiveness. The system supports realistic training and evaluation of operator decision-making, swarm behavior, and shared control in dynamic environments.
HUMAN FACTORS INTEGRATION The simulator was developed with human factors principles integrated into every stage of its design. Key performance-shaping factors such as situation awareness, workload management, and interface usability guided the selection of feedback modalities, interaction methods, and swarm behavior logic. The interface aims to support continuous spatial orientation, reduce cognitive overload, and allow the user to monitor system state without excessive manual input (Wickens, 2008; Endsley, 1995).
The system is built to maintain a human-in-the-loop architecture, allowing the operator to remain actively engaged in high-level control while delegating low-level responsiveness to the autonomous followers. This balance enables the study of shared autonomy, attention allocation, and cognitive effort during mission execution.
To support evaluation, the simulator enables the collection of both subjective and objective human performance data. Subjective measures such as the NASA Task Load Index (Hart and Staveland, 1988) and the Situation Awareness Rating Technique (Taylor, 1990) will be embedded directly within the VR environment for post-task assessment. Objective metrics, including head and hand movement, control latency, and trajectory deviations, will be logged in real time to support unobtrusive behavioral analysis.
This architecture provides a flexible testbed for controlled experiments on teleoperation, swarm coordination, and adaptive interface development. Future iterations will also incorporate physiological sensing and eye tracking to further extend its capabilities for cognitive workload analysis and interface evaluation.
TRAINING AND EVALUATION APPLICATIONS The simulator is designed to support skill development for UAV operators performing complex, multi-agent tasks in rescue and emergency mission contexts. By immersing users in realistic spatial and temporal conditions,
Toward Human-Centered Swarm Control: A VR-Based UAV Simulator 921
the system provides a safe, repeatable environment for practicing navigation, coordination, and formation control under cognitive load.
Scenarios can be configured to vary in complexity, enabling progressive training modules that increase demands on perception, decision-making, and manual control. Trainees may begin with basic flight and formation maintenance before progressing to missions involving obstacle avoidance, dynamic terrain, or anomaly recovery. This structure mirrors real-world mission escalation and allows targeted development of cognitive and motor skills relevant to UAV swarm control.
The simulator’s architecture also supports its use as an experimental platform. Task parameters, interface features, and autonomy levels can be manipulated to study human performance under different conditions. Real- time data collection and embedded evaluation tools make it well-suited for controlled human-subject research on teleoperation, adaptive automation, and team coordination.
Looking forward, the system is designed to integrate with real UAV platforms. Planned extensions include support for digital mapping, computer vision-based terrain interpretation, and direct operator control of physical UAV swarms via the same VR interface. These capabilities position the simulator as both a research tool and a bridge toward future field deployment.
CONCLUSION AND FUTURE WORK This paper presented the design of a VR-based UAV simulator developed to support immersive operator training and human factors research in swarm- based rescue missions. The system combines high-fidelity UAV dynamics with a human-in-the-loop control interface, enabling users to command a leader UAV while autonomous followers maintain formation and avoid obstacles in real time.
The simulator was developed with a strong emphasis on human factors, incorporating design elements that support situational awareness, cognitive workload management, and interface usability. Features such as real-time audiovisual feedback, multi-angle drone perspectives, and modular scenario design create a rich environment for both training and evaluation.
Next steps include pilot testing the system with users to assess its usability, cognitive demands, and responsiveness. These early evaluations will inform iterative improvements to the interface, feedback mechanisms, and swarm behaviors. Performance metrics and user feedback will guide the refinement of the system for both instructional and experimental use.
Long-term applications include integration with live UAV platforms, support for autonomous mapping, and deployment in field training environments. The simulator also offers potential for advancing shared autonomy frameworks and studying human-machine teaming in time- critical, safety-sensitive operations.
922 Eraslan and Gupta
REFERENCES
Annaswamy, A. M. and Lavretsky, E. (2013) ‘Adaptive control: Stability, robustness, and applications’, in Lavretsky, E. and Wise, K. A. (eds.) Robust and Adaptive Control: With Aerospace Applications. London: Springer, pp. 259–296. Cummings, M. L. (2014) ‘Man versus machine or man + machine?’, IEEE Intelligent
Systems, 29(5), pp. 62–69. Endsley, M. R. (1995) ‘Toward a theory of situation awareness in dynamic systems’, Human Factors, 37(1), pp. 32–64. Eraslan, E. and Yildiz, Y. (2021) ‘Modeling and adaptive control of flexible
quadrotor UAVs’, 2021 60th IEEE Conference on Decision and Control (CDC), Austin, TX, USA, pp. 1783–1788. doi: 10.1109/CDC45484.2021.9683103. Eraslan, E. and Yildiz, Y. (2024) ‘Flexible quadrotor unmanned aerial vehicles: Spatially distributed modeling and delay-resistant control’, Journal of Guidance, Control, and Dynamics, 47(6), pp. 1109–1122. Fasano, G., Accardo, D., Moccia, A., Carbone, C., Ciniglio, U., Corraro, F.
and Luongo, S. (2008) ‘Multi-sensor-based fully autonomous non-cooperative collision avoidance system for unmanned air vehicles’, Journal of Aerospace Computing, Information, and Communication, 5(10), pp. 338–360. Floreano, D. and Wood, R. J. (2015) ‘Science, technology and the future of small autonomous drones’, Nature, 521(7553), pp. 460–466. Hart, S. G. and Staveland, L. E. (1988) ‘Development of NASA-TLX (Task Load
Index): Results of empirical and theoretical research’, Advances in Psychology, 52, pp. 139–183. Ioannou, P. A. and Sun, J. (2012) Robust Adaptive Control. Mineola, NY: Dover Publications. Jerald, J. (2015) The VR Book: Human-Centered Design for Virtual Reality. San
Rafael, CA: Morgan & Claypool. Khan, A., Siddique, A. and Lee, Y. (2021) ‘UAV simulator technologies for training and education: A review’, IEEE Access, 9, pp. 65554–65575. Lavretsky, E. and Wise, K. A. (2013) Robust and Adaptive Control: With Aerospace
Applications. London: Springer. McCarley, J. S. and Wickens, C. D. (2005) Human factors implications of UAVs in the national airspace. DOT/FAA Technical Report. Murphy, R. R. and Stover, S. L. (2008) ‘Rescue robots for mudslides: A descriptive
study of the 2005 La Conchita landslide response’, Journal of Field Robotics, 25(1), pp. 3–16. Oberhauser, M. and Dreyer, D. (2017) ‘A virtual reality flight simulator for human factors engineering’, Cognition, Technology & Work, 19(2), pp. 263–277. Parasuraman, R., Cosenzo, K. A. and De Visser, E. J. (2009) ‘Adaptive automation
for human supervision of multiple uninhabited vehicles: Effects on change detection, situation awareness, and mental workload’, Military Psychology, 21(2), pp. 270–297. Salas, E., Rosen, M. A., DiazGranados, D. and Klein, C. (2008) ‘Expertise-based intuition and decision making in organizations’, Journal of Management, 34(5), pp. 941–973. Stone, R. J. (2017) ‘Virtual reality for training: Opportunities and challenges’,
Ergonomics, 60(5), pp. 660–666. Taylor, R. M. (1990) ‘Situational awareness rating technique (SART): The development of a tool for aircrew systems design’, in Situational Awareness in Aerospace Operations (AGARD-CP-478). Neuilly Sur Seine, France: NATO- AGARD.
Toward Human-Centered Swarm Control: A VR-Based UAV Simulator 923
Wickens, C. D. (2008) ‘Multiple resources and mental workload’, Human Factors, 50(3), pp. 449–455. Wright, J., Decrements, C. and Cummings, M. L. (2013) ‘The impact of interface
design on UAV supervisory control and situation awareness’, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 57(1), pp. 1111–1115. Zhang, B., Mehta, A., Khasawneh, A. and Gramopadhye, A. K. (2020) ‘A review of human–UAV interaction research for application in public health emergency response’, Human Factors and Ergonomics in Manufacturing & Service Industries, 30(5), pp. 311–325.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Design, Engineering, and Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-75-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional