Daniela Doroftei, Geert De Cubber, Hans De Smet (2022). Assessing Human Factors for Drone Operations in a Simulation Environment. In: Human Factors in Robots, Drones and Unmanned Systems. AHFE Open Access, vol. 57, pp. 122–129. https://doi.org/10.54941/ahfe1002319
The number of small drones and drone operations is expanding and proliferating tremendously. However, there is a problem. Drones crash. Often. When they do, international studies show that over 70% of the drone crashes can be related to human factors. Combining these two facts, it is clear that - if we want to avoid a massive number of drone incidents in the future – it is required to develop a strategy to incorporate human factors in the drone deployment process and the training of drone pilots. Pilots for regular aircraft or for larger (typically military) drones generally follow extensive simulator training before engaging in any real flight. However, for small rotorcraft, this is much less the case, because it is very difficult to convey a realistic representation to the human sensory system. Both for fixed wing and rotary wing drones, the main problem with current simulator-based pilot training programs is that they are limited to simplistic scenarios (typically flying predefined patterns and practicing take-off and landing operations), without providing much qualitative feedback to the trainee or the supervising entity.In response to these identified shortcomings, we present in this paper a drone operator performance assessment tool, which uses a realistic environment and realistic operational conditions to measure the performance of the drone operator, both in a qualitative and quantitative manner. These metrics can then be used by training responsibles to adapt / adjust the theoretical and practical training courses for drone pilots, such that the curriculum (both the practical and the theoretical courses) can be iteratively optimized to best fit the needs. An important aspect of any qualification assessment procedure is the definition of the test methodologies and of the test scenarios. Within the subject of drone pilot training, these test scenarios are currently most often very limited to simple take-off & landing operations and of following simple patterns in the air. For pilots working in the security sector (military, police, firefighters, civil protection, ...) in tough operating conditions, these highly simplistic scenarios are hardly relevant. Therefore, we also propose a set of standard test methods specifically geared towards the training of drone operators in the security sector.In this paper, we present the architecture of the developed assessment tool, which runs inside a simulation environment, enabling repetitive (statistically relevant) testing in a controlled environment. The simulation engine is based upon the AirSim simulator and uses the UnReal game engine for realistic environmental rendering. Together with the end-users, twenty-two scenarios have been defined within two main mission environments: a mountainous and an urban environment. These scenarios are especially conceived to cover most of the hazards and environmental challenges that end users wanted to test for, the pilot performance capabilities that they wanted to see measured and the human factors that they identified as potentially important influencing factors for pilot performance.
As the amount of drone operations rises each year, so does the number of incidents with these systems (Chow et al., 2016). Research has shown that the large majority of these drone incidents are caused by human error (Shi- vely, 2015). In order to avoid all these incidents, it is therefore required to investigate what are the root causes for these errors, with the aim to learn from these mistakes for improving the training procedures of future drone pilots.
However, assessing the human factors that play a role in the occurrence of incidents is not a trivial problem. In general, two types of approaches can be distinguished. On the one hand, a detailed incident report analysis can be performed in order to investigate the causes of the incident. However, for operations with small drones, incident reporting is very often missing, mea- ning that the input data required for performing the analysis is missing or not abundant enough to allow for statistically relevant processing. On the other hand, there are the approaches that make use of simulation environments to observe pilots in a controlled environment. This is a common practice within manned aviation, where pilots for regular aircraft or for larger (typically mili- tary) drones generally follow extensive simulator training before engaging in
Assessing Human Factors for Drone Operations in a Simulation Environment 123
any real flight. However, for operations with small drones, this is much less the case, mainly because it is very difficult to convey a realistic representation to the human sensory system for these kinds of operations. Moreover, both for rotary wing and fixed wing systems, a major problem with present-day simulators is that they are limited to simplistic scenarios (typically executing a predefined pattern or practicing landing or take-off maneuvers). The out- put of such a simulation system does not providing a high-quality feedback to the trainee or the mentor.
To tackle the issues identified above, we propose in this paper a drone operator performance assessment tool, which measures the performance of drone operators in both a qualitative and a quantitative manner. The ultimate goal is then that these metrics are used by the trainers to iteratively optimize the training curricula to guarantee a maximum safety level.
For any simulation system, the level of realism is a key aspect that must be carefully considered in order to achieve the desired result. For the proposed simulation framework, we make use of a highly realistic environment, inclu- ding realistic operational conditions (wind, weather effects, etc.). Another important aspect of any qualification assessment procedure is furthermore the definition of the test methods and scenarios. Present-day simulators gea- red at drone pilot training are very often very limited with respect to the type of scenarios covered. Most often, they only provide the possibility to train some simple take-off & landing operations or to practice executing simple patterns in the air. However, many of the human factors related to opera- ting drones only arise in much more complex scenarios, such as working in adverse environmental or operational conditions. Pilots operating drones in tough operating conditions (e.g., military, police, firefighters, civil protection units encounter such complex scenarios, etc.). For these users, the simplistic scenarios are hardly relevant. Therefore, this paper also introduces a stan- dard test methodology which is specifically oriented towards the assessment of the performance of drone operators working in the security sector.
PREVIOUS WORK & MAIN CONTRIBUTIONS The modelling of drone operator human performance is not a new subject. Already in 2006, the US Air Force developed a human performance model (Deutsch, 2006). However, as this research focused on operations with large and remotely operated military drones, these operator performance mode- ling approaches focus solely on operator workload analysis, with the aim of optimizing the (remote) crew composition. For smaller drone systems, this is not very relevant, as the ‘crew’ for these systems is in general very limited.
On the other hand, (Bertuccelli et al, 2010) proposed a new formulation for a single operator performing a search mission with multiple drones in a time-constrained environment. (Wu et al., 2016) expanded on this idea by proposing a multi-operator multi-drone operator model.
A major criticism with respect to the methods that are discussed above, is that they all rely heavily on aspects such as attention and fatigue modeling and neglect other aspects that are paramount for operations in the security
sector, such as mission stress, enemy counter-measures, varying operator skill levels, etc.
To tackle these issues, this paper presents a methodology to develop a drone operator performance model, which is specifically geared to operators of drones in the security sector.
The development of quantitative evaluation methodologies for the evalu- ation of the performance of drones and their operators is something that has already been researched before by the National Institute of Standards and Technology (NIST), specifically in the framework of urban search and rescue (USAR) operations (Jacoff et al., 2017). Within this research, NIST develops standardized test methods and test infrastructure to quantitatively evaluate the performance of drone operators. The resulting standard test methods ena- ble any user to generate statistically relevant performance data to evaluate airworthiness, maneuvering, sensing, payload functionality, etc. While extre- mely valuable, these standard test methodologies developed by NIST heavily focus on urban search and rescue operations and are not generically useable for all types of security operations.
Within our previous work (Doroftei et al., 2020), we therefore initia- ted a concept of using a set of standardized test methodologies for human performance modelling in the domain of security operations, based upon the existing NIST framework for urban search and rescue operations. In this paper, we build further upon this work, by comparing its performance to a commercial state–of–the–art solutions. Before making this compari- son, the following section introduces the overall concept of the proposed methodology.
CONCEPTUAL OVERVIEW OF THE APPROACH To assess the relationship between human factors and the human operator performance, we follow a user-centered design approach (Doroftei et al., 2017) to come to the methodology which is graphically depicted in Figure 1, and which can be summarized as follows:
1. In a first step, we identify which human factors could potentially influ- ence the performance of drone pilots, via a set of interviews with expert drone operators. From this set of interviews, it appears that the most rele- vant human factors to include are: Task Difficulty, Pilot Position, Stress, Fatigue and Pressure. Each of these identified parameters is re-identified with the test subjects during an intake questionnaire to assess the state of the pilot when she or he starts the simulation exercise. 2. Via a set of interviews with expert drone operators working in the secu- rity sector, we identify which operational scenarios and environmental conditions potentially affect the performance of drone pilots. From this set of interviews, a set of 22 standard operational scenarios were compi- led that cater to the needs of as many end-users (i.e., drone operatives in the security sector) as possible. These scenarios consider complex target detection, tracking, observation & identification missions in urban and rural environments.
Assessing Human Factors for Drone Operations in a Simulation Environment 125
Figure 1: Schematic overview of the test procedure where the pilots are subjected to. After taking an intake survey, the pilots have to perform a complex mission in a simu- lation environment. While doing this, their performance parameters and physiological state are assessed. After completing the mission, they perform an outtake survey.
3. We developed and extensively documented a highly realistic simulation environment, where drone operators can perform complex drone opera- tions (Doroftei et al., 2020). In summary, it is based upon the Microsoft AirSim simulation engine (Shah et al., 2017), which is an open-source simulator for drones, built on the Unreal Engine (Epic Games, 2022). This simulation environment is completely open and customizable, which enables us to incorporate the standard test scenarios, to multiple customi- zable drones and to quantitatively measure the performance of the pilots on-line while executing the mission. Within this simulator, 22 standard operational scenarios are defined within two environments (urban/rural). In these scenarios, the operators need to deal with large-scale dynamic environments, changing environmental conditions and time pressure in order to deliver quality data in a minimal amount of time. These are all factors that can lead to human errors and can have an impact on the performance of the operator. 4. When having finished their scenario within the simulation environment, the physiological state of the drone operators is assessed by means of a questionnaire. This questionnaire also serves to assess whether there are changes in the physiological state of the drone operators with respect to the moment of performing the intake survey. A typical example for this would be the pressure level, which is typically perceived as quite low before a trial (as it concerns ‘only’ a simulation trial), but is generally perceived much higher due to the time pressure induced in the trial scenarios.
Figure 2: Comparison of the features of the proposed methodology related to the training of drone operators to existing commercial solutions.
5. A repeated execution of the 22 test scenarios over enough drone ope- rators as test subjects entails that statistically relevant data is obtained related to the physiological state prior to beginning the mission, during the mission, and after completing the mission. Furthermore, the simula- tion engine quantitatively measures human performance data, like e.g. the time required to perform mission objectives.
Using this data, a mathematical model is built up between on the one hand the human factors and the human physiological state and on the other hand the human performance. This model enables us to predict human performa- nce given a certain input state and it can be used for drone pilot performance assessment.
VALIDATION & COMPARISON A direct comparison to state of the art systems is not straightforward, as there are not many existing tools for the assessment of the human factors for drone operations. Therefore, in this section, we concentrate on the valida- tion of the drone pilot training tool, for which there are multiple competing possible solutions. Referring back to the conceptual scheme of the proposed architecture (section 3), the validation focuses on steps one to three, while discarding steps four and five.
Figure 2 shows a comparison of the features of the proposed architecture related to drone pilot training with existing commercial solutions in the same domain. Here we will discuss each of these features:
Assessing Human Factors for Drone Operations in a Simulation Environment 127
In terms of graphics, the presented tool relies on the UnReal rendering engine (Epic Games, 2022), which is capable of recreating highly rea- listic environments. Most of the existing commercial solutions also use some form of the UnReal engine (or the competing Unity engine) under the hood. However, the advantage of our method is that it is possible to add own models and thereby build up the level of realism.
The world size is important, as real complexity can only be achieved in feature-rich large-scale open-world environments. Only our method and the DJI Drone Simulator provide this capability.
The compatibility with remote controllers is also a key aspect for our rese- arch, as the remote controller is the human-machine interface. Optimizing its choice is one of the underlying goals of the program. The propo- sed methodology is somewhat limited with this respect, because it relies on AirSim for making the link to the remote controller and the list of AirSim-compatible remote controllers is somewhat limited. In contrast, the Zephyr and RealFlight solutions are compatible with a very wide range of remote controllers.
Human errors mostly appear when executing complex scenarios. It is the- refore very surprising to notice that most commercial solutions really focus on scenarios with a relatively low degree of complexity, in contrast to the method we use.
The number of drone models that are incorporated into the simulator determines the versatility and modularity of the tool. Most of the solu- tions have a very good offering in this domain. Our solution relies on AirSim, which enables to freely incorporate drone models. However, this does imply that the drone model needs to be known.
Dealing with non-standard environmental effects like rain, wind, snow, etc., is a typical cause for human error. Most of the solutions therefore also incorporate these features as training models.
In order to use the training data effectively, it is required to log the ope- rator performance during flights. Interestingly, only our solution and the Zephyr simulator provide this functionality.
Next to the operator performance, the status and performance of the drone needs to be logged during flight, in order to be able to correlate the effect of human pilot action with the drone performance. To our knowledge, our approach is the only one which does this.
Going beyond individual scenarios towards full-scale application trai- ning is a feature that very few simulators provide. The DJI simulator offers application training for search & rescue and for powerline inspe- ction applications. Our solution provides application training for security applications (detection, tracking, identification of persons) in multiple scenarios.
Overall, it can be concluded that, while the DJI and Zephyr drone simula- tor also provide some very good features & concepts, our proposed solution provides capabilities beyond the feature set of state of the art commercial solutions in the domain.
CONCLUSION & FUTURE WORK This paper presented a methodology for the qualitative and quantitative assessment of the performance of drone pilots. The methodology is based upon a set of standard test methods and a virtual training environment. The proposed methodology enables the development of a comprehensive human performance drone model, which helps us to make a link between the human factors and the physiological state on the one hand and the human per- formance on the other hand. This will not only help us to understand the relationship between these parameters, it also would, in a later stage, support completely pilot-agnostic qualitative (Doroftei et al., 2015) and quantitative (De Cubber et al., 2017) evaluation of drones and drone pilots.
However, before this can be accomplished, it is required to achieve a num- ber of research goals. In a first phase, the proposed methodology will be validation-tested by real expert military drone pilots. This will not only allow us to qualitative and qualitative assess the performance of these pilots, but also to build up the human performance model. In a second phase, this human performance model will then be used, on the one hand as a reference for drone pilot performance testing and on the other hand for assisting with the accre- ditation of new drone designs, as it would allow to eliminate the human pilot from the test process.
REFERENCES
Bertuccelli, L.F., Beckers, N.W.M., Cummings, M.L. (2010). Developing operator models for UAV search scheduling. In Proc. of AIAA Guidance, Navigation, and Control Conference, Toronto, Canada. Chow, E., Cuadra, A., Whitlock, C. (2016). Hazard Above: Drone Crash Database -
Fallen from the skies. The Washington Post. De Cubber, G., Doroftei, D., Balta, H., Matos, A., Silva, E., Serrano, D., Govindaraj, S., Roda, R., Lobo, V., Marques, M., Wagemans, R. (2017). Operational Valida- tion of Search and Rescue Robots. In: De Cubber, G., Doroftei, D. (eds.) Search and Rescue Robotics - From Theory to Practice. Deutsch, S. (2006). UAV Operator Human Performance Models. BBN Report 8460. Doroftei, D., De Cubber, G., De Smet, H. (2020) “Reducing drone incidents by incor-
porating human factors in the drone and drone pilot accreditation process,” in: Advances in Hu-man Factors in Robots, Drones and Unmanned Systems, San Diego, USA, p. 71–77. Doroftei, D., De Cubber, G., Wagemans, R., Matos, A., Silva, E., Lobo, V. Cardoso, G., Chintamani, K., Govindaraj, S., Gancet, J., Serrano, D. (2017). User-Centered Design. In: De Cubber, G., Doroftei, D. (eds.) Search and Rescue Robotics - From Theory to Practice. InTech. Doroftei, D., Matos, A., Silva, E., Lobo, V., Wagemans, R. De Cubber, G. (2015).
Operational validation of robots for risky environments. In: Proc. 8th IARP Workshop on Robotics for Risky Environments. Lisbon, Portugal. Epic Games, Inc., 2022, accessed 13 February 2022, Website: https://www.unrealen gine.com/ Jacoff, A., Saidi, K. (2017). Measuring and Comparing Small Unmanned Aircraft
System Capabilities and Pilot Proficiency Using Standard Test Methods. NIST report.
Assessing Human Factors for Drone Operations in a Simulation Environment 129
Shah, S., Dey, D., Lovett, C., Kapoor, A. (2017). AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. Field and Service Robotics. Shively, J. (2015). Human Performance Issues in Remotely Piloted Aircraft Systems.
in: ICAO Conference on Remotely piloted or piloted: sharing one aerospace system. Wu, Y., Huang, Z., Li, Y., Wang, Z. (2016). Modeling Multioperator Multi-UAV Operator Attention Allocation Problem Based on Maximizing the Global Reward. In: Mathematical Problems in Engineering.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Robots, Drones and Unmanned Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-33-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional