Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Experimental Evaluation of Mission-Planning Support in Multi-User Manned-Unmanned Teaming Applications with Shared Unmanned Systems

University of the Bundeswehr Munich, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1002309

Abstract

In this study, we evaluate our work towards sharing unmanned systems among multiple distributed users. We propose a hierarchical approach in which the use of unmanned systems can be requested by secondary users. To support the primary user in managing these requests, we developed a prototype of a planning agent that can generate and evaluate potential solutions that incorporate the request into the mission plan. These solutions can then be provided to support the primary user’s decision-making process. We implemented the planning agent in a helicopter research simulator and validated it with military pilots. Within an experiment, we investigated different configurations that supported the participants on different levels of automation. Results indicate that a higher level of automation caused a positive effect on performance but is accompanied with a loss in situational awareness.

INTRODUCTION

The ongoing development of unmanned systems increases the potential to improve future operations regarding speed, efficiency, safety, and afforda- bility. However, dynamic and complex missions will still require human coordination and intervention in terms of planning, decision-making, and problem solving (Schulte and Donath, 2019). As a hybrid approach, Manned- Unmanned Teaming (MUM-T) combines the operation of manned and unmanned systems to enhance the team’s capabilities and mission performa- nce (OSD, 2011).

In single-User MUM-T missions, one manned command vehicle is used to manage a number of unmanned systems to successfully and efficiently con- duct missions. We investigate the extension of this concept to multiple users, who work within a common mission space and share unmanned systems. This can, on the one hand, increase the efficient use of unmanned systems because they are not tied to a single user but can be deployed across multi- ple missions. Unused capacities can thus be made available to others who

© 2022. Published by AHFE Open Access. All rights reserved. 46

Experimental Evaluation of Mission-Planning Support 47

otherwise would not have access to an unmanned system. On the other hand, when several users share a resource that they need for their respective work processes, they may interfere with each other. Especially overlapping demand and uncoordinated access can result in delays in the work process, inefficient resource use or unnecessary increases in mission costs. Howe- ver, sharing a resource with others could also lead to planning uncertainty, making it difficult to coordinate tasks well and may encourage users to accept inefficiencies.

MISSION-PLANNING SUPPORT Our application investigates MUM-T scenarios in which a manned helico- pter is teamed with three unmanned aerial vehicles (UAVs) to accomplish dynamic and complex military transport missions. The UAVs contribute to mission execution particularly with their automated reconnaissance capabili- ties. Additional users who require the UAVs for their missions are represented by ground-based units, such as forward air controllers or convoys.

In our hierarchical approach (Roth and Schulte, 2020) to shared UAV use, there is a primary user (host) who mainly uses the UAVs for own purpose but also makes them available to additional secondary users (clients). Therefore, a client is authorized to request the conduction of specific tasks as well as temporary access to the UAVs. Consequently, in addition to own mission con- duction, the host is furthermore tasked with coordinating the cross-user UAV deployment and managing UAV absence with own mission requirements. Potential problems that may emerge from this include increased task load, reduced mission performance and negative impact on situational awareness. To compensate for these issues, our research focuses on assisting the host in processing these requests and the associated mission-planning.

Our approach pursues to pre-process a request and to provide the host with feasible solutions and appropriate courses of action. Therefore, we developed a planning agent that is capable of merging external plan elements received from the client into the host’s original mission plan to identify solutions that are in accordance with the host’s mission requirements. Additionally, the agent is enabled to take advantage of possible optimizations such as task restructurings to counteract increases in mission costs that are accompanied with integrating the request. Identified solutions are evaluated based on sele- cted criteria and rated against each other. Thus, our approach is based on generating and evaluating multiple potential solutions that incorporate the request into the original mission plan. These solutions can then be used to provide decision support on different levels of automation.

INTERACTION To investigate individual effects on workload, situational awareness and per- formance, we scaled the agent’s interaction according to the following levels of automation (Parasuraman, Sheridan and Wickens, 2000; Ruff et al, 2002): manual, interaction by alternatives, interaction by selection, and interaction by proposal.

48 Roth and Schulte

Manual: On this level, the host is not offered any support by the planning agent and is therefore solely responsible for deciding on a request and its integration into the mission plan.

Interaction by alternatives: On the next automation level, the agent pro- vides various solutions but without an evaluation of individual options or a comparison between them. Assistance includes providing a variety of feasible solutions, visually supporting decision-making, and simplifying the imple- mentation of a solution. Otherwise, the solution set is not restricted or selected. For this, the planning agent identifies solutions for integrating requ- ested UAV use, under the condition that the mission remains feasible and all mission dependencies are met.

A visual representation of these solutions should provide support in eva- luating the effects of the respective solution. This includes the direct impact, such as UAV transition costs associated with a request and the duration of the requested use. Together, they represent the time period in which the UAV cannot be used for own mission execution. Furthermore, the solutions should visualize effects that occur as an indirect consequence, such as delays to other planned tasks or transition costs after finishing the requested use. Both can negatively impact the remainder of the mission execution and should likewise be considered when incorporating a request.

By confirming a solution, it is automatically implemented by the planning agent. This includes the assignment of the request to a UAV as well as the rescheduling of already existing tasks or their transfer to another UAV.

Interaction by selection: Evaluating the quality of individual solutions allows the planning agent to pre-select implementations that appear more desirable than others. It is thereby possible to intervene with a reduced set of solutions. However, we tried to avoid that the set consists of similar solutions. The human decision-making process is usually oriented towards finding an acceptable solution instead of optimizing it in detail (Todd and Gigerenzer, 2000). Consequently, if a solution is rejected, that is probably not because it is not the absolute optimum, but because the user dislikes something about the solution. In this case, it would be disadvantageous if alternatives were highly similar to the original solution, increasing the probability that the problem is still existent. Instead, our approach was to provide an optimized solution for each available UAV. With three UAVs, this results in a set of three proposed solutions that each assign the request to a different UAV.

Interaction by proposal: Support at this level involves the proposal of a specific solution for incorporating a request. While the other interaction modes required the user to choose between several offered solutions, this mode provides the solution with the best rating. In this way, the decision pro- cess is supported in that this optimized solution is already pre-selected and can eventually be directly confirmed. This potentially reduces both task load and mental resources required to finding a solution. However, this saving is strongly dependent on whether the suggested solution corresponds to the user’s expectations and intension. Otherwise, the user has to handle the pro- blem manually and can only derive reduced benefit from the assistance. In order to still provide support in such a case, we enabled the user to manually specifying aspects that are to be included in the solution.

Experimental Evaluation of Mission-Planning Support 49

Therefore, the user was allowed to make specifications on two levels to modify the solution. First, it can be restricted to which UAV the request should be assigned. The proposal is then modified to the best solution given that the selected UAV adopts the request. By this, the user is enabled to access the solution set that provides an optimized solution for each UAV (intera- ction by selection). Secondly, the positioning of the requested plan element can be specified. This allows for individually selecting a solution from the entire space of feasible solutions (interaction by alternatives). This two-stage selection option therefore allows to access solutions of lower automation levels.

EXPERIMENTAL EVALUATION The planning agent was implemented into our MUM-T research helicopter simulator that allows for the simulation of dynamic transport missions in a multi-user environment. We furthermore designed a human-in-the-loop expe- riment that was to be conducted with military helicopter pilots. To reduce training time and to benefit from the experience and expertise of subject matter experts, our experimental group was to consist of pilots who had experience in coordinating subordinate aircraft. As this significantly limited the group of available participants, we designed our experiment according to a within-subjects design.

We investigated three different experimental conditions that varied in the interaction-level between the participant and the planning agent. It ranged between manual, interaction by alternatives and interaction by proposal, whereby the manual mode served as a baseline in which no assistance was provided. Due to limited experimental time, the configuration interaction by selection was not part of the experiment.

Before the experiment started, the participants underwent a training, which included the visualization and manual processing of requests as well as the interaction with the planning agent’s decision support. Subsequently, the participants performed six short helicopter transport missions in which they were teamed with three UAVs. Each mission lasted approximately 10 to 15 minutes. Their task was to plan and execute the UAV deployment according to a given mission goal and a given helicopter flight path. During mission execution, the subjects received requests for external UAV use which they were to integrate into their own mission plan. Decision support provided by the agent was dependent on the investigated condition. The three conditions were equally distributed on the six missions. To counteract learning effects and combinatorial effects, the order of both the conditions and the missions were counterbalanced.

Within each mission, there was an unannounced freeze probe that inter- rupted the simulation to assess situational awareness. Within a SAGAT questionnaire (Endsley, 1988), participants had to recall the exact locations of the three UAVs on an empty representation of the mission area. Around each position, they were asked to draw an area of uncertainty, in which they expected the respective UAV to be. On completion of each mission, subje- cts were asked to self-rate their perceived workload that was caused by the

50 Roth and Schulte

Mathematical equation reproduced from the original PDF
Equation reproduced from the original PDF.

RESULTS

A total of 10 military helicopter pilots (all male) between the ages of 25 and 51 years (mean 42.1; SD 8.2) and an average of 2486 flight hours participated in the study. Although results are not significant, noteworthy trends in the data are discussed below.

Workload: The assessment of the workload, that particularly resulted from the integration of requests (see Figure 1), indicates an increase between the configurations of manual (M: 2.05, SD: .97) and interaction by alternatives (M: 2.30, SD: 1.00). We attribute this to an increase in visual and coordina- tion effort accompanied with the selection of an alternative. A possible reason for this could be that the provision of potential solutions increased the num- ber of integrations that participants might have evaluated for themselves and consequently also increased the complexity of finding a solution. Inspecting and examining a selected alternative also places an additional burden. Visu- ally perceiving and comprehending the presented solution imposes an effort that is not present in the manual integration. Likewise, verifying whether the presented solution is the desired one requires additional mental effort by the user. Frequently changing the solution by switching through the different alternatives, as it was often observed in the experiment, further intensifies this effort. This is because the representation on the tactical map changes with each change of the solution, which again has to be visually perceived and mentally processed.

Integrating a request while being supported with a proposed course of action shows a positive effect compared to both other configurations. Betw- een the two configurations interaction by alternatives (M: 2.30, SD: 1.00) and interaction by proposal (M: 1.80, SD: 1.25), there is a considerable decrease in the subjective workload rating. We attribute this to the proposed course of action decreasing task complexity. The better the suggested solution matched the user’s intentions, the less often it was required to change and compare solutions with each other.

Also, in comparison to manual (M: 2.05, SD: .97), the workload was rated to be lower. We concluded from this that proposing a course of action sim- plified the task of integrating a request and compensated for the increase in complexity that was observed in the configuration interaction by alternatives.

Situational Awareness: The results in Figure 2 indicate that the errors betw- een the reported and the actual UAV positions were significantly lower for manual (M: 1.91, SD: .72) and interaction by alternatives (M: 1.86, SD: .70) than for the interaction by proposal (M: 2.39, SD: 1.16). Consequently, subje- cts were worse at reflecting the actual positions of the UAVs when provided with a suggested solution for integrating a request. This suggests that the process of manually finding a solution, as it is required in both manual and interaction by alternatives, is beneficial to situational awareness. We suppose that through the decision-making process, both the solution and the reasons that contribute to the decision are memorized. If this process is abbreviated

Experimental Evaluation of Mission-Planning Support 51

Figure 1: Workload results.
Original PDF Figure 1:
Figure 2: Situational awareness results. position error (left) and uncertainty area (right).
Original PDF Figure 2:

or simplified by presenting a proposed course of action, characteristics of a solution are less likely to be embedded into the user’s situational awareness. Consequently, it may prove more difficult to recall or reconstruct the decision and to derive the UAV positions from that in a later phase.

Likewise, the possible area of stay that the subjects were asked to draw for the individual UAVs was larger when they were supported with a pro- posed course of action (M: 3.23, SD: 2.20) compared to manual (M: 2.48, SD: 1.47) and interaction by alternatives (M: 1.97, SD: 1.22). This suggests that the subjects’ uncertainty about the actual position of a UAV was greater when a solution was proposed. We suspect that the same reason as for the increased position error is responsible for this effect. Despite the fact that the assessed data is purely subjective, the result is consistent with the objective results derived from the position determination. Thus, the test subjects see- mingly were aware to have a reduced situational awareness with regard to the UAVs’ whereabouts. This thesis can be supported by individual statements of the pilots in which the perception of a reduced situational awareness was indicated.

Performance: Performance measures showed that response times for manual (M: 25.15, SD: 17.55) were relatively high in comparison to the response times for interaction by alternatives (M: 17.93, SD: 6.36) and inte- raction by proposal (M: 16.67, SD: 6.10). Accordingly, with the latter two configurations the subjects were able to process the request significantly faster, from which we infer an increase in performance (see Figure 3).

52 Roth and Schulte

Figure 3: Performance results.
Original PDF Figure 3:

We attribute this improvement on the one hand to a reduced mental effort, given that the expected effects of a solution were determined by the agent and visualized to the user. Accordingly, the users were not required to identify these effects themselves and were thus able to decide more quickly. Ano- ther possible reason could be a reduced physical effort to integrate a request. While manual integration required the subjects to operate buttons that were relatively far apart on the interface, the other two configurations allowed to integrate a request using a single dialog with closely located buttons. This may also have resulted in a reduction in response time.

The difference in response times between interaction by alternatives and interaction by proposal appears to be less pronounced. Nevertheless, the time to process a request was reduced once again with a proposed course of action. Similar to the reduction in Situational Awareness, a facilitated decision-making process through the provision of a recommended course of action could be a possible reason for this. The better the proposed solution corresponds to the user’s idea the less time is required to select a solution and compare it to others.

CONCLUSION

Within this study, we presented our approach to support UAV deployment across multiple decentralized users in MUM-T environments. We developed a prototype of a planning agent capable of pre-processing multi-user resou- rce demands and providing decision-making support. Interaction between the agent and the user was scaled according to different levels of automa- tion, ranging from no support over sets of alternatives to proposing a distinct course of action. The described approach was implemented into a research helicopter simulator and evaluated in an experiment with 10 military heli- copter pilots. The results imply that providing a set of alternatives increased the workload associated with integrating a request but improved performa- nce in terms of response time. Proposing a course of action decreased the perceived workload and also improved performance but might be accompa- nied by negative effects on situational awareness. This suggests a trade-off between workload and situational awareness, possibly due to an abbreviated decision-making process. Further research should investigate the application of adaptive interaction that is scaled according to the presumed workload of

Experimental Evaluation of Mission-Planning Support 53

the user and the criticality of the request. Another aspect to study is whe- ther diminished situational awareness is linked to foreign UAV use. As users mainly concentrate on own mission conduction, it would be reasonable that attention towards UAVs that conduct externally requested tasks is reduced. This may also result in dependencies between levels of automation and situ- ational awareness that are different from acquainted investigations without third-party UAV use.

REFERENCES

Endsley, M. R. (1988). Situation awareness global assessment technique (SAGAT).

In Proceedings of the IEEE 1988 national aerospace and electronics conference, 789–795. Office of the Secretary of Defense (2011). Unmanned Systems Integrated Roadmap, 2011–2036. DoD. Washington, DC. Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and

levels of human interaction with automation. IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans, 30(3), 286–297. Roth, G., and Schulte, A. (2020). A concept on the shared use of unmanned assets by multiple users in a manned-unmanned-teaming application. In International Conference on Human-Computer Interaction, 189–202. Ruff, H. A., Narayanan, S., & Draper, M. H. (2002). Human interaction with levels

of automation and decision-aid fidelity in the supervisory control of multiple simulated unmanned air vehicles. Presence, 11(4), 335–351. Schulte, A., and Donath, D. (2019). Cognitive engineering approach to humanautonomy teaming (HAT). In 20th International Symposium on Aviation Psych- ology, Dayton. Todd, P. M., & Gigerenzer, G. (2000). Précis of simple heuristics that make us smart.

Behavioral and brain sciences, 23(5), 727–741.

Cover of the proceedings volume Human Factors in Robots, Drones and Unmanned Systems

Explore the complete digital volume

Human Factors in Robots, Drones and Unmanned Systems

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-33-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →