Rune Stensrud, Aleksander Simonsen, Olav Rune Nummedal, Sigmund Valaker (2024). Situation awareness training as a prerequisite for handling complexity in Human-autonomy teaming: Demonstration and experiment proposal. In: Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems. AHFE Open Access, vol. 119, pp. 496–506. https://doi.org/10.54941/ahfe1004514
Human-autonomy teaming (HAT) is characterized by high degrees of interdependence between humans and machine (Lyons, 2021). This underscore the need for human-autonomy teams (HATs) defined as “at least one human working cooperatively with at least one autonomous agent” (McNeese et al., 2018, p.262). However, this interdependence may vary, for example according to how well the human (s) and machine (s) may solve subtasks autonomously. Drawing on the extant literature on human decision making, the ability to project future events is essential to prioritize and use both human and machine resources in ways that accomplish tasks (Endsley, 2000). The question arises as to how humans and machines can be enabled to make such projections together. We here focus on the human’s part of this information processing and decision making and the needs for for adjusting the mode of collaboration due to the changes in the environment (Lundberg & Johansson, 2021; Stensrud, Mikkelsen & Valaker, 2023). The human may or may not take the initiative to change the mode of collaboration, such as in engaging in more detailed collaboration. What may ensure that the human is enabled and do take the initiative to change the mode of collaboration? Both cognitive, emotional as well as issues such as task load may influence the degree to which the human changes its mode from loose to tight control and/or input or vice versa (Endsley & Garland, 2000) what we may also call nuances to configurations of the team architecture (O’Neill et al., 2023). Recent reviews indicate that maintaining awareness is critical, yet can be impaired over prolonged time (Casner & Hutchins, 2019). Specifically we concentrate on the situation awareness (SA) level 3, projecting future state of elements in the environment and the switching from one way of collaborating to another. In short, in our example this concerns the ability to foresee a change from a relatively stable environment with easily observable entities to one that has more complexity regarding the entities to observe and their interrelations. Our reasoning is that if the human are able to form predictions of changes in the environment it can also be enabled to change its way of collaborating. Given the detrimental effects of time-pressure and task-load and fatigue etc. that may impede the forming of sound predictions (Endsley & Garland, 2000), we propose preparation that reduce the risk of such impediments and that empower the human to make predictions.Automation can [traditionally] be defined as that in which “the system functions with no/little human operator involvement: however, the system performance is limited to the specific actions it has been designed to do”. Endsley (2015, “Autonomous Horizons” (p. 3)). Autonomy is often characterized in terms of the degree to which the system has the capability to achieve mission goals independently, performing well under significant uncertainties, for extended periods of time, with limited or non-existent communication, and with the ability to compensate for system failures, all without external intervention. Autonomy can be thought of as a significant extension of automation in which very high-level mission-oriented commands will be successfully executed under a variety of possibly not fully anticipated circumstances […] given adequate independence and task execution authority. Autonomy can be considered as well designed and highly capable automation.[..] over the next 30 years […] we will see a gradual evolution of system control, with intermediate levels of autonomy being applied to various functions. As the autonomy developed becomes more capable over time, can handle a greater range of functions, and can handle greater ranges of variability in the environment, systems will slowly evolve to more autonomous operations for longer periods of time. (Endsley (2015), «Autonomous Horizons» (p. 4)).Having mapped the interdependencies we indicate ways of formally testing the influence of SA level 3 training on human adaptivity. Firstly we suggest an inductive approach whereby a group of humanoperators are followed through their learning and familiarization with the system. Included are time spent on making plans for and contingencies plans for the employment of the autonomous system.During phases of teaming with the system we will record the input the humans make to their system interface. We will categorize what type of “orders” and other input the humans make. We willcategorize them according to the delegating supervisor approach (Levels of automation) versus more collaborative (mixed initiative and coactive design) types of approaches. This mapping willenable us to describe potential learning cycles in the familiarization and make hypothesis about the role of SA level 3 training.Our future research endeavors will embark on this and report the empirical results in the near future.
Situation Awareness Training as a Prerequisite for Handling Complexity 497
INTRODUCTION
Drawing on the extant literature on human decision making the ability to project future events is essential to prioritize and use both human and machine resources in ways that accomplish tasks (Endsley & Garland, 2000). For example in a human-human team by projecting (feedforward) the needs of patients time-critical help (e.g. for intensive trauma patients) can be pro- vided. This relies, in a team and social setting, on updating the shared awareness and integrating the information and then make projections (Burke et al., 2006). Likewise it is essential to provide projections that can guide machines, something that an autopilot may (within confines) provide and automatically adjust its course, but the human may need to intervene in some critical situations. Recent reviews indicate that maintaining awareness is crit- ical, yet can be impaired over prolonged time (Casner & Hutchins, 2019). The question arises as to how humans and machines can be enabled to make such projections together. We here focus on the human’s part of this infor- mation processing and decision making in the collaboration with unmanned surveillance drones in the military.
Underlying the need for adjusting the mode of collaboration is the changes in the environment (Valaker et al., 2022). We focus here on the complexity of the environment, i.e. the magnitude of elements and their interrelations, as well as the dynamics, i.e. the rate of changes in the elements in the environ- ment. These characteristics of the environment may fluctuate. In situations of low complexity the autonomous machine entities may go on with their work to a large degree without any collaboration. This is typical of low interdependence. However interdependencies may change, e.g. increase, as the complexity and dynamics increases (Van de Merwe et al., 2024). One example from our military context is the degree of unknown and/or civil- ian entities in the vicinity of military targets. The more the unknown and/or civilian entities the more the complexity of the environment are changing, and the rate of this change may not be linear. In particular with respect to use of kinetic force. Such a situation may necessitate much more intertwined collaboration between the human and the machines we specifically discuss, both from an efficiency standpoint (where should the drones be placed to achieve success) and not the least from the point of meaningful human con- trol so as to conduct operations in accordance with the law of armed conflict. The human may or may not take the initiative in engaging in more detailed collaboration. The initiative may be sound or it may be flawed, depending on the correctness of the situation awareness of the human operator.
What may ensure that the human is enabled and do take the initiative to change the mode of collaboration? Both cognitive, emotional as well as issues such as task load may influence the degree to which the human changes its mode from hands-off to hands-on or vice versa (Endsley & Garland, 2000). Specifically we focus on the role of situation awareness level 3, projecting future state of elements in the environment and the switching from one way of collaborating to another. In short in our example this concerns the ability to foresee a change from a relatively stable environment with easily observ- able entities to one that has more complexity regarding the entities to observe
and their interrelations. Our reasoning is that if the human are able to form predictions of changes in the environment it can also be enabled to change its way of collaborating. Given the detrimental effects of time-pressure and task-load fatigue etc. that may impede the forming of sound predictions (End- sley & Garland, 2000), we propose preparation that reduce the risk of such impediments and that empower the human to make predictions.
Endsley and Garland (2000) point to contingency planning as a key foun- dation for being able to project. We therefore propose that not only passively receiving instructions on the autonomous system (i.e. its tasks and how to communicate tasks, its capabilities, how it moves, range, data and informa- tion processing) is enough to enable projecting and adjusting during actual HAT task resolution. Rather going through vignettes that vary in complexity and dynamics and that require the humans to do some contingency planning we see as critical. In recent work Endsley (2023) indicate that both extensive and ongoing training as well as explicit ways of communicating information between human and AI systems are important to develop the humans’ men- tal model of the AI. These mental models may steer the type of collaboration chosen. We build on prior discussion (Stensrud et al., 2023) of how some human-autonomy teaming (HAT) design approaches (mechanisms for coor- dination), specifically levels of automation (LOA), mixed-initiative (MI), and coactive design (COAD) (Johnson et al., 2011 and 2018) could be combined and how they are triggered by how the human projects the situation.
Our hypothesis is that one can benefit from making structured prepa- rations (in teams) involving CoA training for operators (i.e. contingency training (Endsley & Garland, 2000) and team training (Myers et al., 2018)). Specifically a training where the human operator collaborate with the AI in simulated environments that vary the environmental complexity and dynam- ics, we see as enhancing the operators’ ability to make sound predictions. Again this knowledge will enable the operator to intervene, or not intervene, at appropriate points in the task resolution.
We are presenting theory on autonomy basics in chapter 2. Further, intro- duce an inductive method in chapter 3, on how to design an “out check” routine on UxS systems. Besides the practical “out check”, we are present- ing a functional analysis of the demonstrator in chapter 4, followed by a discussion on practical implications and future research in chapter 5.
THEORY ON AUTONOMY BASICS Automation can [traditionally] be defined as that in which “the system functions with no/little human operator involvement: however, the system performance is limited to the specific actions it has been designed to do” Endsley (2015, p. 3). Autonomy is often characterized in terms of the degree to which the system has the capability to achieve mission goals indepen- dently, performing well under significant uncertainties, for extended periods of time, with limited or non-existent communication, and with the ability to compensate for system failures, all without external intervention.
Autonomy can be thought of as a significant extension of automation in which very high-level mission-oriented commands will be successfully
Situation Awareness Training as a Prerequisite for Handling Complexity 499
Figure 1: Situation awareness training (Adapted from Sarvesh Sawant et al., 2023).
executed under a variety of possibly not fully anticipated circumstances […] given adequate independence and task execution authority. Auton- omy can be considered as well designed and highly capable automation (Endsley, 2015, p. 4).
Adaptive Autonomy in Human-Robotic Teams Bakken et al (2023) argue that automation and autonomy represent two inde- pendent, but related, concepts that characterize usage of Extended knowledge (AI) to support decision-making (Table 1). Conceptually the two dimensions may be thought of as continuous, but in the framework model they are both dichotomized as high and low. Since the dimensions are independent, they may take on any of the 2 × 2 combinations: low-low, high-high, low-high, and high-low (Endsley, 2015 [p. 8, fig. 3]; Bakken et al (2023)[p. 6, table 1] propose the following labels for these combinations:
Low automation – low autonomy: Consultancy
High automation – low autonomy: Adaptation
Low automation – high autonomy: Integration
High automation – high autonomy: Supremacy
Table 1. Automation and autonomy represent two independent, but related, concepts that characterize usage of Extended knowledge to support decision-making (Endsley, 2015).
Table available in the original PDF paper.
Autonomy/authority (Effectiveness) Low High
Automation (Efficiency) Low Consultancy Integration High Adaption Supremacy
The aim of efficiency using AI – that is, why we automate for task per- formance – is to achieve consistency, precision, and accuracy, at speeds
and volumes that humans cannot match, and at a lesser cost. The aim of effectiveness through autonomy is the belief (or rather hope) that AI may produce even better decisions without the supervision or intervention of a human, to the end that AI will by itself define and pursue “greater goals” and higher-valued end-states when decisions are left to the machines – in part or altogether (Endsley, 2015 [p. 20 i.e. mixed-initiative team training]; Bakken et al., 2023 [p. 5]).
SITUATION AWARENESS TRAINING Training in general can be defined as the systematic acquisition of attitudes, concepts, knowledge, rules, or skills that should result in improved perfor- mance (Salas et al., 2009). In the case of HAT the human operator specifically develop, to a more or less refined degree, mental models of the autonomous agent(s). Mental models are “a consistent understanding and representation of how systems work” (Endsley, 2023), thus incorporating concepts, knowl- edge and rules that apply to the autonomous system. We now elaborate on how the understanding and representation could be supported by training, which again may transfer to real life situations.
The autonomous system are always developed to be able to perform within certain contextual boundaries (Lyons et al., 2021). Although some systems may evolve throughout time, these boundaries are crucial to understand the capability of the system, i.e. how it works. Van de Merwe et al. (2024) point to several contextual variables relevant in their ship navigation case. In more generic terms context or environment may be defined according to the com- plexity and dynamics of a situation. We suggest three crucial steps in how transparency and explain ability of the system in its environment are pro- vided to build the human operators mental model. We generically present the training model in Figure 1. The suggestion on training model is inspired by Sarvesh Sawant et al. (2023).
In more detail, the training must include basic familiarization with the sys- tem to obtain a sound level 1, perceive and level 2 understand in different levels of complexity and dynamics (phase 1). Secondly we suggest to make vivid to the human operator how the system works at varying changing lev- els of complexity and dynamics (Endsley, 2023) (phase 2). Thirdly the human operator we believe would learn more deeply how the system works by both planning and executing missions that varies in time their environmental con- straints (Endsley & Garland, 2000; Salas et al., 2009). This involves enacting the specific collaboration process needed. This familiarizes with the fact that not all situations increase in complexity and dynamics, but may revert to low complexity and dynamics. Crucially in this last phase the human operator can plan the needed collaboration process (Endsley, 2023) and then execute a spe- cific simulated collaboration process. Here the operator makes a connection between the knowledge of system, how it works in different situations, and the chosen collaboration process. Feedback will be provided on the success of using the specific collaboration process which may reinforce or dampen the use of specific collaboration processes in new simulation trials, which can provide the human an opportunity to adapt its collaboration process. It is
Situation Awareness Training as a Prerequisite for Handling Complexity 501
thus assumed that the operator may learn from failure, i.e. by wrongly using a specific collaboration process. In the Table 2 below we sketch broadly how one may proceed in this training with respect to the logic of gaining knowl- edge of different and changing environmental constraints and their impact on system performance.
There are of course several trajectories with respect to changing environ- mental constraints, but these are some of the more generic that one may follow. This is just an example. Importantly the specific number of tasks a system may do autonomously varies according to the systems capabilities, and may change over time (Lyons et al., 2021). We thus suggest that for each system separate analysis of the specific content of the complexity and dynamics, and type of tasks undertaken need to be made concretized.
Table 2. SA training with respect to the logic of gaining knowledge of different and changing environmental constraints and their impact on system performance.
Table available in the original PDF paper.
Environmental constraints
Phase 1: Basic familiarization
Select collaboration process
Phase 2: Intermediate familiarization
Phase 3: Advanced familiarization
Environmental constraints changing (examples)
A) Low complexity and low dynamics
use LOA to define what the machine does autonomusly
The autonomous system may perform task 1,2 and 3
From A to B Changes in the amount of tasks being performed by the system
The human should make decisions on what tasks the system can do
B) High complexity and low dynamics
The human should engage in MI
use MI to jointly collaborate on some tasks
The autonomous system may perform task 1 and 2
From B to C Changes in the human assistance needed by the system
C) Low complexity and high dynamics
change from MI to COAD on some tasks
The human should engage in COAD
From C to D Changes from some to extensive needed from the system
The autonomous system may perform task 1 and 2, but need some human assistance to perform task 2 D) High complexity and high dynamics
change from COAD to LOA.
The human should switch from COAD to LOA
From D to A Changes from some tasks performed and extensive assistance needed back to fully capable system
The autonomous system may perform task 1 but need extensive human assistance to perform task 2
TEST BED The software structure in Valkyrie is based on a distributed architecture in which most of the software is running on a set of on-board computers carried by each individual UxV. The software architecture is illustrated in Figure 1,
and in line with the high level reference architecture of Swarm-centric Systems for ISR (SS4ISR) guideline that address operational and system recommenda- tion of Swarm-centric systems (NATO-STO-TR-SET-263, 2022, p. 1–1). An uncrewed system (UxS) is defined as a system capable of exerting its power to perform designated missions with no human aboard (modified NIST 2004 definition) (Huang, H.-M.,2004). Examples include uncrewed aerial vehicles (UAV), uncrewed ground vehicles (UGV), uncrewed surface vessels (USV), and uncrewed underwater vehicles (UUV), botnets (cyber), tethered systems, and control systems or mission planning systems with machine learning or AI enabled data analysis. System in UxS is both the vehicle/machine and the payloads” (O’Neill et al., 2023, [footnote 1 on pg. 2]).
Figure 2: Valkyrie (UxVs) test bed (Adapted from Nummedal, 2021; Mathiassen, K. & Nummedal, O. R., 2022).
The Valkyrie (UxVs) are uncrewed aerial vehicles (UAV) illustrated in Figure 3. The Valkyrie (UxS) systems application, with (UxS) common configuration options i.e. demonstrating remotely operated vehicles, of autonomous systems, of swarming, and of teaming, for each configura- tion option we develop sense-understand-control-plan-sensor coordination mission treads.
Each UxV in the system implements all the necessary sub modules on- board and can act autonomously and independently. There is no hard constraints on the maximum number of UxVs that can operate concurrently, either in the on-board software, or from the perspective of the Ground Con- trol Station (GCS). The GCS supports interfaces for assigning missions to the system and to visualize the system status like positions and state of the
Situation Awareness Training as a Prerequisite for Handling Complexity 503
individual UxVs. These missions are typically composed of abstract tasks like “map area”, “search along axis”and can be assigned to either one individual, a group, or all UxVs currently connected to the system. When mutiple UxVs are connected to the system, the basic layout of the GCS interface remains the same and the system can be operated with the same number of terminals or monitors (one or two depending on the setup). When the mission is received on the UxV side, the “Decision making” autonomy component will decode the missions into a set of tasks which will recursively be decomposed further into simpler tasks, like for instance decomposing a “search along axis” task into individual movement and sensor control steps that can be executed by the “Behavior” components which control the actual autopilot and sensor pay- load trough the platform interfaces. The Mission manager on-board the UxV will report system status and mission progress back to the operator station, as well as coordinating with other UxVs. In addition to executing the mis- sion, the UxVs decision autonomy also monitors the status of all sub systems continuously and executes fail-safe behaviors like reacting to a loss-of-link situation, low battery, UxV being outside of the designated operation area etc. The reaction to these conditions can be mission-specific. When assigning a mission to the Valkyrie system, the following information is provided through the command-and-control protocol: An arbitrary list of the UxVs assigned to this mission, a geographic specifier like a location, an axis, or an area, a task- descriptor field specifying what the task is, i.e., searching, patrolling or simply moving, and a set of constraints like maximum speed, altitude and collision avoidance policy. It is up to the onboard autonomy to actually implement the behaviors and the low-level control details which makes the system interop- erable with respect to different platforms with different control and sensor layouts.
CONCLUSION
Bakken et al. (2023) proposes that automation and autonomy represent two independent, but related, concepts. To explore empirically the framework presented (in Table 1). We ought to take calibration of trust in autonomy as a critical prerequisite (Endsley, 2015; Kaber, 2018; O’Neill et al., 2022). Our suggestion to extend the framework of Bakken et al. (2023) is to take the human factor into consideration that characterize usage of extended knowl- edge to support proper decision-making (e.g. historical mapping of sensor data applicable for a future enhanced exploitation station). We will invite our sponsors to a follow-up on these subjects, and will propose to continue the experimental program proposed in this paper, to explore this need of preparation of operators. We propose to conduct experiments with (UxS) operators and decision makers, in a context with a backdrop of relevant military scenarios. We suggest manipulating both automation and auton- omy – the independent variables – of an extended knowledge tool to support decision making, in a 2 × 2 experimental design. As dependent variables we measure transparency and reliability on (UxS) system, as well as perceptions of accountability. We will also operationalize and measure perceived align- ment and level 3 (SA). Performance and goal attainment will also be measured
(Mouloua et al., 2020). As potential moderators we suggest using cognitive style, personality factors, expertise and experience of the operators.
Practically, we will instrument the investigation with a set of system engi- neering methods to inform the system technicians with a system overview based on activity modelling. A mission engineering approach, will be explored when we are developing the UxS Reference Architecture. The pro- posed base line for this activity is an indicative system reference architecture (Valkyrie) (Figure 2).
An evaluation based on the principles of Endsley (2015; 2023) applied on the UxS Valkyrie could be done.
As we suggested each new iteration of the system separate analysis of the specific content of the complexity and dynamics, and type of tasks under- taken need to be made concrete. This may evolve according to the system capabilities.
ACKNOWLEDGMENT
The authors would like to acknowledge the Valkyrie (UxVs) support team.
REFERENCES
Bakken, B.; Lund-Kordahl, I.; Bjurström, E. AI in Future C2 – Who’s in Command When AI Takes Control?; Track 4; IDA: Maryland, U. S, 2023; Vol. paper 057, p. 8. Burke, C. S.; Stagl, K. C.; Salas, E.; Pierce, L.; Kendall, D. Understanding Team
Adaptation: A Conceptual Analysis and Model. Journal of Applied Psychology 2006, 91 (6), 1189–1207. https://doi.org/10.1037/0021-9010.91.6.1189. Casner, S. M.; Hutchins, E. L. What Do We Tell the Drivers? Toward Minimum Driver Training Standards for Partially Automated Cars. Journal of Cognitive Engineering and Decision Making 2019, 13 (2), 55–66. https://doi.org/10.1177/ 1555343419830901. Endsley, M. R.; Garland, D. J. Pilot Situation Awareness Training in General Aviation.
Proceedings of the Human Factors and Ergonomics Society Annual Meeting 2000, 44 (11), 357–360. https://doi.org/10.1177/154193120004401107. Endsley, M. R. Autonomous Horizons: System Autonomy in the Air Force – A Path to the Future. Volume I: Human-AutonomyTeaming. 2015. https://doi.org/10. 13140/RG.2.1.1164.2003. Endsley, M. R. Supporting Human-AI Teams: Transparency, Explainability, and Sit-
uation Awareness. Computers in Human Behavior 2023, 140, 107574. https: //doi.org/10.1016/j.chb.2022.107574. Huang, H.-M. (2004). Terminology for specifying the autonomy levels for unmanned systems :: version 1.0 (NIST SP 1011; p. NIST SP 1011). National Institute of Standards and Technology. https://doi.org/10.6028/NIST. SP.1011 Johnson, M.; Bradshaw, J. M.; Feltovich, P. J.; Jonker, C. M.; van Riemsdijk, B.;
Sierhuis, M. The Fundamental Principle of Coactive Design: Interdependence Must Shape Autonomy. In Coordination, Organizations, Institutions, and Norms in Agent Systems VI; De Vos, M., Fornara, N., Pitt, J. V., Vouros, G., Eds.; Springer Berlin Heidelberg: Berlin, Heidelberg, 2011; Vol. 6541, pp 172–191. https://doi.org/10.1007/978-3-642-21268-0_10. Johnson, M.; Bradshaw, J. M.; Feltovich, P. J. Tomorrow’s Human–Machine Design Tools: From Levels of Automation to Interdependencies. Journal of Cognitive
Situation Awareness Training as a Prerequisite for Handling Complexity 505
Engineering and Decision Making 2018, 12 (1), 77–82. https://doi.org/10.1177/ 1555343417736462. Kaber, D. B. A Conceptual Framework of Autonomous and Automated Agents. Theoretical Issues in Ergonomics Science 2018, 19 (4), 406–430. https://doi.org/10. 1080/1463922X.2017.1363314. Lundberg, J.; Johansson, B. J. E. A Framework for Describing Interaction between
Human Operators and Autonomous, Automated, and Manual Control Systems. Cogn Tech Work 2021, 23 (3), 381–401. https://doi.org/10.1007/s10111-020- 00637-w. Lyons, J. B.; Sycara, K.; Lewis, M.; Capiola, A. Human–Autonomy Teaming: Definitions, Debates, and Directions. Front. Psychol. 2021, 12, 589585. https: //doi.org/10.3389/fpsyg.2021.589585. Mathiassen, K., & Nummedal, O. R. (2022). LandX20 Experiment Report – exper-
iment for future land warfare capabilities with focus on increased situational awareness and unmanned systems (FFI-Report 22/00274). FFI; 978-82-464- 3390-5. https://www.ffi.no/publikasjoner/arkiv/landx20-experiment-report-expe riment-for-future-land-warfare-capabilities-with-focus-on-increased-situationa l-awareness-and-unmanned-systems McNeese, N. J.; Demir, M.; Cooke, N. J.; Myers, C. Teaming With a Synthetic Teammate: Insights into Human-Autonomy Teaming. Hum Factors 2018, 60 (2), 262–273. https://doi.org/10.1177/0018720817743223. Human Performance in Automated and Autonomous Systems. Current Theory and
Methods; Mouloua, M., Hancock, P. A., Ferraro, J., Eds.; CRC Press/Taylor & Francis Group: Boca Raton, 2020. Myers, C.; Ball, J.; Cooke, N.; Freiman, M.; Caisse, M.; Rodgers, S.; Demir, M.; McNeese, N. Autonomous Intelligent Agents for Team Training. IEEE Intell. Syst. 2019, 34 (2), 3–14. https://doi.org/10.1109/MIS.2018.2886670. NATO-STO. Swarm System for Intelligence Surveillance and Reconnaissance;
Technical Report RDP STO-TR-SET-263; STO, 2022. https://www.google.com/ url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwiJn_ToiP2BAx VuVvEDHYPWD98QFnoECBkQAQ&url=https%3A%2F%2Fwww.sto.nato.in t%2Fpublications%2FSTO%2520Technical%2520Reports%2FSTO-TR-SET- 263%2F%24TR-SET-263-ES.pdf&usg=AOvVaw2SZKMAbgu3G2N5om3FQm 7f&opi=89978449 (accessed 2022-08-26). Nummedal, O. R. Flamingo - a UAV for Autonomy Research; FFI-Report 21/00318; FFI: Kjeller, 2021. https://www.ffi.no/publikasjoner/arkiv/flamingo-a-uav-for-au tonomy-research. O’Neill, T.; McNeese, N.; Barron, A.; Schelble, B. Human–Autonomy Teaming: A
Review and Analysis of the Empirical Literature. Hum Factors 2022, 64 (5), 904– 938. https://doi.org/10.1177/0018720820960865. O’Neill, J., McDonald, J., Heaton, B., Thotahewa, K., Augur, C., & Brown, K.
(2023). Applying a Mission-Engineering Thread Approach using Sense-Decide- Effect to the Problem of a Common C2 System for UxS. ICCRTS, Baltimore. Salas, E.; Wildman, J. L.; Piccolo, R. F. Using Simulation-Based Training to Enhance Management Education. Academy of Management Learning & Education 2009, 8 (4), 559–573. https://doi.org/10.5465/AMLE.2009.47785474. Sawant, S., Brady, C., & Mallick, R. (2023). Balancing the Scales of Explainable and
Transparent AI Agents within Human-Agent Teams. POS168. Stensrud, Rune, Sigmund Valaker, and Torgar Haugen. “Interdependence as an Element of the Design of a Federated Work Process.” In 2020 IEEE International Conference on Human-Machine Systems (ICHMS), 1–6. Rome, Italy: IEEE, 2020. https://doi.org/10.1109/ICHMS49158.2020.9209423.
Stensrud, Rune, Sigmund Valaker, and Bjorn Mikkelsen. “Orchestrating Humans and Teammates to Counter Security Threats: Human-Autonomy Team- ing in High and Low Environmental Complexity and Dynamism,” 2023. https://doi.org/10.54941/ahfe1002830. Van De Merwe, K.; Mallam, S.; Nazir, S.; Engelhardtsen, Ø. Supporting Human
Supervision in Autonomous Collision Avoidance through Agent Transparency. Safety Science 2024, 169, 106329. https://doi.org/10.1016/j.ssci.2023.106329. Valaker, Sigmund, Rune Stensrud, Reidar Haugen, and Bjørnar Mikkelsen. “Possible Consequences for Command and Control of Shifting Coordination Form: Navigating the Murky Waters of Adverse Situations and Complex Missions,” 2021.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-95-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional