Sophie Hart, Vicky Banks, Seth Bullock, Jan Noyes (2022). Understanding human decision-making when controlling UAVs in a search and rescue application. In: Human Interaction & Emerging Technologies (IHIET 2022): Artificial Intelligence & Future Applications. AHFE Open Access, vol. 68, pp. 485–494. https://doi.org/10.54941/ahfe1002768
The introduction of Unmanned Aerial Vehicles (UAVs) within search and rescue missions provides human operators with ‘eyes in the sky’. However, little attention has previously been paid to the implications of this type of technology on the decision-making procedures of search and rescue teams; in particular, the human operator responsible for extracting and analysing UAV sensor data in order to achieve the mission objective, typically to locate missing people.Within the field of Human Factors, theoretical decision models have been used to identify user requirements for interfaces, training protocols, workstation layouts, and decision aids. We propose that decision models can be applied to the study of Human-Robot Interaction. Thus, the current paper presents a literature review of decision models used within Human Factors. The provision of a UAV within a Search and Rescue operation is used to case study the utility of these decision models for capturing the aspects of decision-making exercised by the sensor operator.
Rapid advancements in sensor technology and artificial intelligence have enabled the application of robotics within sociotechnical systems to pro- vide additional information sources that support the compilation of situation awareness (SA) (Riley and Endsley, 2005). For instance, Unmanned Aerial Vehicles (UAVs) provide the human operator with access to multiple perspe- ctives on their environment using onboard payload sensors (e.g., cameras) (Cahillane et al., 2012). In that sense, UAVs enhance SA by providing opera- tors with ‘eyes in the sky’. However, introducing a non-human agent within the control-feedback loop is not a straightforward process (Banks and Stan- ton, 2016). The introduction of automation has contributed to several fatal incidents, such as the Tesla crash (Banks et al., 2018) and two Boeing 373 Airmax crashes (Endsley, 2019) due to emergent interaction issues and misun- derstandings concerning the capabilities of the non-human agent. To that end, it is crucial to establish effective human-robot partnerships so that human operator(s) and robots can effectively cooperate as a team.
The study of human-robot partnerships falls within the human-robot inte- raction (HRI) research domain. This has been formally defined as “a field of study dedicated to understanding, designing, and evaluating robotic systems for use by or with humans” (Goodrich and Schultz, 2008, p.1). Human Factors (HF), namely the study of human interaction with other humans and system elements (Dul et al., 2012), is a prominent research area within HRI which has identified numerous relevant issues. For example, commonly cited challenges surround the topics of situation awareness (Riley and Ends- ley, 2005), operator workload (Hooey et al., 2018) and human-automation interaction issues (de Visser and Parasuraman, 2011). All of these factors con- tribute to an individual’s ability to process information and make decisions. To address these issues, one avenue of investigation involves using theoreti- cal models to understand where current ways of working can be improved to best support operator decision-making (e.g., Parnell et al., 2021a, b). Model- ling the HRI decision-making process is challenging (Sheridan, 2016) but can deliver many benefits. Previously, decision models have been used to inform the design of user interfaces, training protocols, workstation layouts, and decision aids (Jenkins, 2012; Banks et al., 2021; Parnell et al., 2021b). The current paper discusses the use of HF decision models within HRI. Through- out the paper, the use of UAVs in a search and rescue scenario is considered as a case study through which to explore the suitability and benefits of these different approaches to decision modelling within HRI.
DECISION MODELLING There are many different models of human decision-making. Typically, rese- arch within HRI uses computational approaches to map the shared mental models held between human operators and robotics (Lebiere et al., 2013). Conversely, HF research adopts a theoretical approach to decision model- ling. The following section provides an overview of decision models used within the HRI and HF literature.
Decision Modelling Within HRI Rasmussen (1996) organised research on decision-making into four catego- ries: (i) decision-making paradigms developed from normative models by Subject Experts; (ii) guidance on decision support tools for situations requi- ring knowledge-based problem solving; (iii) naturalistic decision making (NDM; Klein et al., 1989) models that describe actual decision-making beha- viour; (iv) cognitive-based models that conceptualise the cognitive processes involved in a decision-making task (Lebiere et al., 2013). When working within a human-robot partnership, decision-making represents a dynamic process guided by the goals of the non-human and human agents, sha- red knowledge, and contextual information accessed from the world (Bicho et al., 2011; Goodrich and Schultz, 2008). Approaches to modelling decision- making processes often use cognitive, computational models (Schmerling et al., 2018). This approach can involve process modelling principles of human behaviour so that non-human agents understand and emulate human decision-making processes.
Understanding Human Decision-Making when Controlling UAVs 487
In terms of HF research within HRI, previous work used phases of Cogni- tive Work Analysis (Stanton et al., 2017) to elicit user requirements that support the maintenance of SA (Adams, 2005; Adams et al., 2009). However, the uptake of this approach has slowed in recent years. Indeed, computational approaches have dominated the literature on modelling human decision- making within HRI. Consequently, it is likely that the benefits of Rasmussen’s (1996) research strands (particularly NDM) have not been realised fully within HRI.
Decision Modelling Within Human Factors The following sections provide an overview of decision-making models across two categories. The first category, NDM, describes expert decision-making processes in real-world applications (Klein et al., 1989; Klein, 2008). Conver- sely, formative models address decisions that could be made within the work domain rather than what actually happened from the operator’s viewpoint (Jenkins et al., 2010).
Naturalistic Decision-Making The field of NDM provides one approach to studying decision-making pro- cesses in terms of “how people make decisions in real-world contexts that are meaningful and familiar to them” (Lipshitz, 2001, p.32). The NDM appro- ach gave rise to numerous decision-making models although the debate over the optimal model is ongoing (Lipshitz, 1993; Lintern, 2010). Interestingly, only a subset of models have been applied to sociotechnical systems (e.g., Parnell et al., 2021b; Parnell et al., 2021a; Plant and Stanton, 2012; Banks et al., 2018). These models include the Recognition Primed Decision Model (RPDM; Klein et al., 1989) and the Perceptual Cycle Model (PCM; Neis- ser, 1976). First, the RPDM emphasises the decision-makers’ experience and recognition of the situation as critical components of the decision-making process (Klein et al., 1989; Klein, 2008). Once a state of recognition is rea- ched, the decision-maker sequentially simulates potential courses of action until a satisfactory option is identified. However, the model has attracted some criticism. The RPDM represents mental simulation as belonging solely ‘in the head’ of the decision-maker (Plant and Stanton, 2015). This means that dynamic changes within the environment, which may present new infor- mation to the decision-maker, are not represented in the decision-making process despite being a defining factor of naturalistic environments (Orasanu and Connolly, 1993). While this model may suit environments where dyna- mic change is less important for decision-making processes, the PCM may be more appropriate when changing environmental information is central for guiding decision-making (Banks et al., 2021; Parnell et al., 2021b).
The PCM comprises three central components: “Schema”, “World”, and “Action”, all of which interact in a perceptual, cyclical manner (Neisser, 1976). Schemata refer to the decision-maker’s internally held knowledge stru- ctures derived from previous experience, which become activated in response to familiar environmental stimuli (Lieberman, 2012). The inclusion of sch- ema within the PCM sets it apart from the RPDM, and for that reason, the
model has been described as advantageous as it incorporates the decision- maker’s psychological processes and interactions within the environment (Banks et al., 2021; Parnell et al., 2021a, b).
Formative Modelling Whilst NDM models can describe and explain the decision-making processes of human operators, the focus on the decision-maker’s cognitive processing can neglect aspects of the decision process, such as the information nodes used to guide decision-making (Banks et al., 2020). In that sense, formative models may provide a broader understanding of the work domain’s tasks, goals, and constraints (Jenkins et al., 2010).
Several formative models are used within HF to map decision-making processes. For instance, one widely used technique is Cognitive Work Analy- sis (CWA; Stanton et al., 2017). The second CWA phase, Cognitive Task Analysis, is specifically geared to elicit insight on decision-making processes (McIlroy and Stanton, 2011). Indeed, previous research has demonstrated the utility of CTA, using Decision Ladders (Rasmussen, 1974) to model current ways of working and identify requirements for automated systems (Asmayawati and Nixon, 2020). Decision Ladders provide a template for mapping “the set of generic sub-tasks involved in decision making”(Rasmus- sen, 1994). The sub-tasks documented within the model are not attributed to a human or non-human agent. Therefore, the entirety of the decision- making process within the sociotechnical system is represented (Rasmussen, 1974; Banks et al., 2020; McIlroy and Stanton, 2011). The decision lad- der is also theoretically underpinned by Rasmussen’s 1983 skill-, rule-, and knowledge-based framework to identify where knowledge-based activities could be supported by improvements to a system’s design, such as automated decision aids (Banks et al., 2020; Parnell et al., 2021b).
UAV IN A SEARCH AND RESCUE APPLICATION The interdependence between social and technical agents during a rescue mission defines the nature of the SAR domain as a sociotechnical system (Adams, 2005; Harrington et al., 2018). As such, the integration of UAV technology within an already complex sociotechnical system requires a syste- mic approach that considers the role and needs of the end-user (Adams, 2005; Norman, 1986). Introducing a UAV to these operations presents a new, separate subsystem, that is, the UAV technical team. A UAV technical team typically comprises three operators: a pilot (navigating the UAV); a sensor operator (analysing payload data, e.g., aerial imagery); and the supervisor (managing the UAV team) (Adams et al., 2009). There has been much rese- arch on UAV navigation due to the inherent complexities of piloting a UAV from a remote location (e.g., Lathan and Tracey, 2002). However, the role of the sensor operator has received less attention despite the criticality of sen- sor processing for constraining the search space and the challenges of sensor analysis on workload and SA (Prewett et al., 2010; Riley and Endsley, 2005).
Understanding Human Decision-Making when Controlling UAVs 489
Table 1. Decision-making aspects of the UAV operators in an S&R scenario.
Table available in the original PDF paper.
Decision-Making Aspect Definition Psychological processes of the sensor operator The psychological processes that occur ‘in the head’ of the sensor operator when monitoring UAV sensor output (Parnell et al., 2021b). Technical processes of the UAV
The technical processes carried out by the UAV (e.g., automatic object detection; Goodrich and Schultz, 2008). Perceptual processes of the sensor operator
The perceptual processes employed when feature-searching a 2D environment for information (Morison et al., 2015). Decisions must be justifiable
The outputted decision – informed by information elicited from the UAV (Mouloua et al., 2003) – is represented as the best outcome for the context of use (Parnell et al., 2021b). Accounts for experience of the sensor operator
The experience held by the sensor operator is represented as a factor effecting decision-making (Mouloua et al., 2003). Options are generated in parallel and compared
The options available to the sensor operator are considered simultaneously and subject to comparative evaluation (Parnell et al., 2021b). Accounts for the interaction with other actors
Interactions with other actors in the sociotechnical system (e.g., human-human, human-UAV) are captured (Adams et al., 2009). Decision-making starts with environmental event
An event occurring within the external environment initiates the decision-making process (Parnell et al., 2021b). For a sensor operator, this process would begin when the decision to use a UAV has been announced by the Incident Commander (Harrington et al., 2018). Dynamic nature of decision-making
External events (e.g., information collected by the UAV) can change the initial decision of the human operator (Adams et al., 2009).
Modelling the Decision-Making Processes of UAV Sensor Operators The following section assesses the applicability of decision models to under- stand the decision processes of the sensor operator. In order to support this evaluation, it is important to understand the extent to which decision models capture the different aspects of decision-making. Parnell et al. (2021b) identified several aspects of decision-making, including decision justifiabi- lity, option generation, cognitive processes, experience, system interactions, and decision-making triggers. Here, we make use of these aspects as a basis to define the emergent decision-making facets that apply when interacting with UAV technology and define several decision-making aspects that emerge when reviewing the literature on HRI (see Table 1).
These decision-making aspects were used to identify which elements of decision-making could be captured by the aforementioned models to elicit insight into the processes of the sensor operator (see Table 2). Three decision
Table 2. Summary of decision-making aspects captured by decision models.
models were evaluated: RPDM, PCM, and Decision Ladders. This asses- sment determined that the PCM encapsulates all aspects of decision-making. Applying the PCM to the decision-making processes of the sensor operator could identify requirements for an image classification decision aid to better support the identification of missing individuals.
Although Zelnio and Fendley (2019) have assessed support systems for an image classification task, the design of the decision aid was seemingly informed by the experimenter rather than the end-user. This can pose chal- lenges when integrating the decision tool into real-world applications due to the disparity between the system designer’s and end-user’s expectations (Nor- man, 1986). In that sense, the PCM would look to close this gap by taking a user-centred design approach.
While it may seem rational rely on a PCM to model the decision-making processes of the sensor operator given its coverage of all of the decision- making aspects, the application of multiple decision models in tandem has been recommended due to the complementary nature of each model’s out- put (Lintern, 2010; Parnell et al., 2021b). Where the PCM delineates when and what information is required to update the operator’s mental schema, the decision ladder identifies the information processing activities primed for automated support, and the RPDM provides a detailed account of an ope- rator’s situation assessment and goal prioritisation (Parnell et al., 2021b). The different insights provided by each model can therefore provide a more holistic overview of design requirements for an image classification decision aid.
DISCUSSION
Reviewing the contrasting qualities of multiple types of decision model has clarified their potential to each contribute to the field of HRI. Although it was not feasible to include the entirety of HF decision models in the current review, multiple alternative models were considered in order to represent the broad range of approaches available within HF and encourage the uptake of the entirety of Rasmussen’s (1996) decision-making research strands. Whilst there is much debate over the optimal decision-making model (Lipshitz, 1993), we argue that this is a redundant dispute. Instead, we should ask
Understanding Human Decision-Making when Controlling UAVs 491
(i) what models are best suited to the context and (ii) which set of models are best used in tandem to achieve the systems aims.
Our work was exemplified using the case study of UAVs within an S&R application. The insights gathered on decision-making processes could be used to manage workload by ensuring that an operator can access and act on relevant information at the appropriate time (Hancock et al., 2007). As the use of singular UAVs evolves towards the use of multi-robotic teaming (Chen and Barnes, 2014) and swarm robotics (Cardona and Calderon, 2019) it is important that the human operator can effectively interpret multiple streams of sensor data to best inform their decision-making process (Mou- loua et al., 2003). The application of decision models in HRI has thus far focused on computational modelling and some usage of CWA (Adams et al., 2009; Jenkins, 2012; Nowroozi et al., 2012). Nevertheless, HF research on decision-making within HRI is currently very limited, particularly for UAVs (Mouloua et al., 2018). Future work should seek to implement these methods at the earliest stage of the design lifecycle to support the design of interfa- ces, decision support systems, training protocols and identify automation requirements (Adams et al., 2009; Jenkins, 2012).
ACKNOWLEDGMENT
This material was funded and delivered in partnership between the Thales Group and the University of Bristol and with the support of the UK Engi- neering and Physical Sciences Research Council Grant Award EP/R004757/1 entitled Thales-Bristol Partnership in Hybrid Autonomous Systems Enginee- ring (T-B PHASE).
REFERENCES
Adams JA. (2005) Human-Robot Interaction Design: Understanding User Needs and Requirements. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 49(3), pp. 447–451. Adams, J.A., Humphrey, C.M., Goodrich, M.A., Cooper, J.L., Morse, B.S., Engh,
C. and Rasmussen, N., (2009) Cognitive task analysis for developing unmanned aerial vehicle wilderness search support. Journal of Cognitive Engineering and Decision Making, 3(1), pp. 1–26. Asmayawati, S. and Nixon, J. (2020) Modelling and supporting flight crew decisionmaking during aircraft engine malfunctions: developing design recommendations from cognitive work analysis. Applied Ergonomics, 82 Banks VA, Allison CK, Plant KL, et al. (2021) Using the Perceptual Cycle Model
and Schema World Action Research Method to generate design requirements for new avionic systems. Human Factors and Ergonomics in Manufacturing & Service Industries 31(1), pp. 66–75. Banks, V.A., Plant, K.L. and Stanton, N.A. (2018) Driver error or designer error: Using the Perceptual Cycle Model to explore the circumstances surrounding the fatal Tesla crash on 7th May 2016. Safety Science, 108, pp. 278–285. Banks, V.A., Plant, K.L. and Stanton, N.A (2020). Leaps and shunts: designing
pilot decision aids on the flight deck using Rasmussen’s ladder. Contemporary Ergonomics and Human Factors.
Banks, V.A. and Stanton, N.A., (2016) Keep the driver in control: Automating automobiles of the future. Applied Ergonomics, 53, pp. 389–395. Bicho, E., Erlhagen, W., Louro, L. and e Silva, E.C. (2011) Neuro-cognitive mechani-
sms of decision making in joint action: A human–robot interaction study. Human Movement Science, 30(5), pp. 846–868. Cahillane, M., Baber, C. and Morin, C. (2012) Human factors in UAV. Sense and Avoid in UAS: Research and Applications, 61, p. 119. Cardona, G.A. and Calderon, J.M. (2019) Robot swarm navigation and victim
detection using rendezvous consensus in search and rescue operations. Applied Sciences, 9(8). Chen, J.Y. and Barnes, M.J. (2014) Human–agent teaming for multirobot control: A review of human factors issues. IEEE Transactions on Human-Machine Systems, 44(1), pp. 13–29. de Visser, E. and Parasuraman, R. (2011) Adaptive aiding of human-robot teaming:
Effects of imperfect automation on performance, trust, and workload. Journal of Cognitive Engineering and Decision Making, 5(2), pp. 209–231. Dul, J., Bruder, R., Buckle, P., Carayon, P., Falzon, P., Marras, W.S., Wilson, J.R. and van der Doelen, B. (2012) A strategy for human factors/ergonomics: developing the discipline and profession. Ergonomics, 55(4), pp. 377–395. Endsley, M. (2019) Human Factors & Aviation Safety, Testimony to the United States
House of Representatives Hearing on Boeing 737-Max8 Crashes. Human Factors and Ergonomics Society, December 11. Goodrich, M.A. and Schultz, A.C. (2008) Human-Robot Interaction: A Survey. Now Publishers Inc. Hancock, P.A., Mouloua, M., Gilson, R., Szalma, J. and Oron-Gilad, T (2007). Pro-
vocation: Is the UAV control ratio the right question? Ergonomics in Design,15(1), p.7. Harrington, K., Brown, M., Pinchin, J. and Sharples, S., 2018. Decision-making within missing person search. Cognition, Technology & Work, 20(4). Hooey, B.L., Kaber, D.B., Adams, J.A., Fong, T.W. and Gore, B.F. (2017) The
underpinnings of workload in unmanned vehicle systems. IEEE Transactions on Human-Machine Systems, 48(5), pp . 452–467. Jenkins D.P. (2012) Using cognitive work analysis to describe the role of UAVs in military operations. Theoretical Issues in Ergonomics Science 13(3), pp. 335–357. Jenkins, D.P., Stanton, N.A., Salmon, P.M., Walker, G.H. and Rafferty, L., 2010.
Using the decision-ladder to add a formative element to naturalistic decision- making research. Intl. Journal of Human–Computer Interaction, 26(2-3), pp. 132– 146. Klein, G. (2008) Naturalistic Decision Making. Human Factors 50(3), pp. 456–460. Klein, G.A, Calderwood, R. and Macgregor, D. (1989) Critical decision method for eliciting knowledge. IEEE Transactions on Systems, Man, and Cybernetics 19(3), pp. 462–472. Lathan, C.E. and Tracey, M. (2002) The Effects of Operator Spatial Perception and
Sensory Feedback on Human-Robot Teleoperation Performance. Presence 11(4), pp. 368–377. Lebiere, C., Jentsch, F. and Ososky, S. (2013) Cognitive models of decision making processes for human-robot interaction. International Conference on Virtual, Augmented and Mixed Reality. Springer, pp. 285–294. Lieberman, D.A. (2012) Human Learning and Memory. New York, NY, US:
Cambridge University Press.
Understanding Human Decision-Making when Controlling UAVs 493
Lintern, G. (2010) A comparison of the decision ladder and the recognition-primed decision model. Journal of Cognitive Engineering and Decision Making 4(4), pp. 304–327. Lipshitz, R., Klein, G., Orasanu, J. and Salas, E., (2001) Taking stock of naturalistic
decision making. Journal of Behavioral Decision Mmaking, 14(5), pp. 331–352. McIlroy, R.C. and Stanton, N.A. (2011) Getting past first base: Going all the way with Cognitive Work Analysis. Applied Ergonomics 42(2), pp. 358–370. Morison, A.M., Woods, D.D. and Murphy, T. (2015) 42 Human-Robot Interaction
as Extending Human Perception to New Scales. Mouloua, M., Gilson, R., and Hancock, P. (2003) Human-centered design of unmanned aerial vehicles. Ergonomics in Design 11(1), pp. 6–11. Mouloua, S. A. et al. (2018) ‘Trend Analysis of Unmanned Aerial Vehicles (UAVs)
Research Published in the HFES Proceedings’, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 62(1), pp. 1067–1071 Neisser, U. (1976) Cognition and Reality WH Freeman. New York. Norman, D.A. and Draper, S. W. (1986) User Centred System Design: New Perspectives on Human-Computer Interaction. Erlbaum Associates. Nowroozi, A., Shiri, M.E., Aslanian, A., et al. (2012) A general computational reco-
gnition primed decision model with multi-agent rescue simulation benchmark. Information Sciences 187, pp. 52–71. Orasanu, J. and Connolly, T. (1993) The reinvention of decision making. Decision Making in Action: Models and Methods 1, pp. 3–20. Parnell KJ, Wynne RA, Griffin TG, et al. (2021a) Generating Design Requirements
for Flight Deck Applications: Applying the Perceptual Cycle Model to Engine Fai- lures on Take-off. International Journal of Human–Computer Interaction 37(7), pp. 611–629. Parnell, K.J., Wynne, R.A., Plant, K.L., et al. (2021b) Pilot decision-making during a dual engine failure on take-off: Insights from three different decision-making models. Human Factors and Ergonomics in Manufacturing & Service Industries. Plant, K.L. and Stanton, N.A. (2012) Why did the pilots shut down the wrong
engine? Explaining errors in context using Schema Theory and the Perceptual Cycle Model. Safety Science 50(2), pp. 300–315. Plant, K.L. and Stanton, N.A. (2015) The process of processing: exploring the validity of Neisser’s perceptual cycle model with accounts from critical decision-making in the cockpit. Ergonomics 58(6), pp. 909–923. Prewett, M.S., Johnson, R.C., Saboe, K.N., et al. (2010) Managing workload in
human–robot interaction: A review of empirical studies. Computers in Human Behavior 26(5), pp. 840–856. Rasmussen, J. (1974). The human data processor as a system component. Bits and pieces of a model. Riso-M-1722. National Laboratory. Rasmussen. J. (1983) Skills, rules, and knowledge; signals, signs, and symbols, and
other distinctions in human performance models. IEEE Transactions on Systems, Man, and Cybernetics SMC-13(3), pp. 257–266. Rasmussen, J. (1996) Merging paradigms: decision making, management, and cognitive control. Third International NDM Conference: Decision-Making Under Stress. Ashgate, pp. 67–85. Rasmussen, J., Petjersen, A.M., and Goodstein, L. P. (1994) Cognitive Systems
Engineering. Wiley. Riley, J.M and Endsley, M.R. (2005) Situation Awareness in Hri with Collaborating Remotely Piloted Vehicles. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 49(3), pp. 407–411.
Schmerling, E., Leung, K., Vollprecht, W., et al. (2018) Multimodal probabilistic model-based planning for human-robot interaction. 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 3399–3406. Sheridan, T.B. (2016) Human–Robot Interaction: Status and Challenges. Human
Factors 58(4), pp. 525–532. Stanton, N.A., Salmon, P.M., Walker, G.H., et al. (2017) Cognitive Work Analysis: Applications, Extensions and Future Directions. CRC Press. Zelnio, H., and Fendley, M. (2019) Automated Aiding of Image Analysts as a
Function of Sensor Type. In NATO-HFM300 Symposium on Human Autonomy Teaming.
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-44-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional