Johannes Maria Ernst, Birte Thomas- Friedrich, Max Friedrich (2026). Mission management in human autonomy teams – an HMI design concept for managing multiple uncrewed aerial systems from a fighter cockpit. In: Intelligent Human Systems Integration (IHSI 2026): Disruptive and Innovative Technologies. AHFE Open Access, vol. 200, pp. 227–237. https://doi.org/10.54941/ahfe1007082
Future air combat will be a complex interaction of crewed and uncrewed aerial vehicles. While many systems will act highly automated or even autonomously, human operators will remain essential for defining mission objectives, setting priorities, and making decisions that require human judgement. Effective collaboration between the human and the highly automated planning systems appears to be a crucial requirement for the success of such a human autonomy team. We present our human-machine interface (HMI) concept for the collaborative management of multiple drones in a future fighter cockpit – resulting from a user-centered development approach with several fighter pilots. We first used an Abstraction Hierarchy to identify the properties that are relevant to describe a mission plan proposed by the highly automated system. This hierarchical structure was then applied to the HMI design so as to tailor the level of detail of the displayed information precisely to the pilot’s situation-specific needs. This includes that information from the complex planning algorithms is presented to the pilots in a transparent and comprehensible way. Finally, an interaction concept was developed for the pilots to be able to easily input their preferences into the planning system.
Modern warfare will be characterized by the joint and coordinated use of crewed and uncrewed aircraft. This so-called human autonomy teaming promises to increase mission success and efficiency while reducing the risk for the crews. Uncrewed aerial vehicles (UAV) equipped with sensors and effectors will preferably carry out the high-risk tasks, while the next-gener ation fighter crews are monitoring and managing them from a safe distance; they will only enter high-risk situations themselves if no other option exists (Patti, 2021). Thus, a human autonomy team is expected to accomplish mis sions that would be too complex or dangerous for a package of conventional crewed fighter aircraft alone.
The described scenario implicates a fundamental role shift for future fighter pilots: Planning and managing the tasks of the escorting UAVs will become a main function, whereas the classic piloting duties become less Received September 18, 2025; Revised November 28, 2025; Accepted December 13, 2025; Available online February 1, 2026
relevant – not least due to the advancing automation in the cockpit. This new role requires a new human-machine interface (HMI) which enables the pilots to effectively manage all UAVs assigned to them during an ongoing mission (Johnson et al., 2007).
The literature offers various approaches to the design of HMIs for UAV management with different automation levels and tasking paradigms like task-based or goal-based guidance (e.g., Roth et al., 2020). Hocraffer & Nam (2017) performed a meta-analysis on 27 papers in order to understand the status quo of UAV swarm management HMI research. They conclude that user and mission-specific customization as well as an increase of the auto mation level of the swarm seem to have a positive effect on the operator’s cognitive workload and situation awareness (SA). Furthermore, they recom mend that the operation of the swarm should not be done manually but at least partially automated. Also, frequent task switching should be avoided to preserve SA.
Later, Huttner & Friedrich (2023) identified 25 relevant publications in their systematic literature review of mission management user inter faces for UAVs. They summarized current challenges of such mission planning systems and strongly recommend the adoption of a user-centric development approach so as to enhance usability, satisfaction, and overall effectiveness.
Our research follows the recommendations by Huttner & Friedrich (2023) and Hocraffer & Nam (2017): We utilize a user-centered develop ment approach where several fighter pilots were involved at different stages of the process. Further, we consider a use case with high automation level of the swarm management.
ROLE OF THE HUMAN IN OUR CONCEPT The effective task management of multiple UAVs in time-critical situations is a complex problem with many variables, where performance can be lim ited by human capabilities. Thus, intelligent mission planning systems that support human pilots in solving these optimization problems are currently under development (e.g., Özcan et al., 2025; Peng et al., 2021).
This paper examines a use case in which the pilots are assisted by an intel ligent mission planning system that:
gathers relevant information and
computes optimized mission plan proposals for all involved assets.
While many systems will act highly automated or autonomously, human operators will remain essential for:
defining mission objectives,
setting priorities,
monitoring mission progress, and
taking decisions that require human judgement.
Figure 1: Role share between pilot and intelligent planning system.
Effective collaboration and task load management between the human and the highly automated planning system appears to be a crucial require ment for the success of such a human autonomy team (Ichaso, 2022). This implies that information from the complex planning algorithms is pre sented to the pilots in a transparent and comprehensible way. Vice versa, the pilots must be able to easily input their preferences into the planning system.
HMI-DESIGN APPROACH Our goal was to develop an HMI concept for the collaborative manage ment of multiple UAVs in a future fighter cockpit – applying the role share between pilot and planning system described in the previous section. While the UAVs execute plans with high level of autonomy, the pilots’ tasks will mainly be to:
1. supervise the plan execution, 2. compare and choose between two or more alternative plans provided by the mission planning system if necessary, and 3. adapt a proposed mission plan, if required.
The focus of this paper is placed on how to present the plan proposals to the pilots. Our HMI philosophy is based on three main goals: First, the pilots should be able to quickly grasp potential problems with the execution of the current mission plan even though they conduct other tasks in parallel. Second, the pilots should be able to quickly understand the plans proposed by the mission planning system and make a decision which plan is more appropriate. Third, the pilots should be able to analyze the plans and poten tial issues more thoroughly if desired and if time is available.
To achieve these goals, we followed an HMI design concept developed at DLR. The empirically validated approach was originally devised for UAV system state monitoring (Friedrich, 2023; Friedrich & Lieb, 2019) with similar goals for the UAV operators, making it well suited for our use case. The display design concept mainly focuses on displaying critical
system states in a way that allows for quick an easy recognition through parallel search (Friedrich & Vollrath, 2022) while avoiding an overload of information (cluttering). This is achieved by first analyzing the relevant elements and parameters of a system via a work domain analysis, namely the abstraction hierarchy (Vincente & Rasmussen, 1992; Vincente, 1999). Then, the abstraction hierarchy is used to tailor the level of detailed infor mation that is displayed on the HMI. Finally, quick recognition of critical system states is achieved by using color-coded icons to represent system states. The remainder of this paper describes how we adapted this approach to develop a presentation of the plans generated by an intelligent mission planning system.
CONCEPT FOR THE MISSION PLAN PRESENTATION When designing an HMI used to represent mission plans and the status of a current mission to the pilots, we first must define the characteristics that describe such a mission plan. One could start by representing a plan with a number of parameters that describe its specific properties (e.g., mission suc cess rate or expected survivability of the assets). From a user perspective, this approach is somewhat similar to what the route planner Google Maps does: It usually proposes three alternative routes together with their properties: duration, distance, and fuel-efficiency (via a green leaf icon).
In our case, however, this approach is not comprehensive enough because a multitude of parameters is necessary to adequately describe a mission plan. This makes it impossible to represent them all on the top layer of one HMI without cluttering. Instead, the level of detail of the displayed information has to be tailored precisely to the pilot’s situation- specific needs. This means that, for instance, during monitoring phases, the HMI should show a decluttered view with few details, in which warnings and caution messages must be quickly and easily recognizable. If neces sary, the pilots must additionally be able to analyze the mission status and potential issues in depth by accessing more detailed information manually. Similarly, newly generated mission plans from the planning system must be represented with a level of detail that makes quick understanding of the plans possible.
Information Presentation Based on Abstraction Hierarchy To get the structured information representation that is required to achieve this, we applied the Abstraction Hierarchy method proposed by Rasmussen (1986). Together with military pilots as subject matter experts, we analyzed the functions and components involved in UAV mission management. This way we could identify the properties that are relevant to describe a mission plan in a hierarchical form. The resulting tree structure of the parameters also provides us with a description of the plan at different abstraction levels or different levels of detail:
Level 1 expected overall mission success rate following the mission plan, Level 2 compliance with mission objectives and level of risk, Level 3 tasking, timing, and resource efficiency as well as risk for ownship, own UAVs, friendly manned/unmanned assets, and mishap risk. Level 4 physical functions, e.g., flight performance, weapon perfor mance, et cetera* Level 5 physical elements, e.g., engine status, hydraulics status, weapon capabilities, et cetera*
It should be noted that the structure of the presented information is hier archical in the sense that the sum of the two measures on level 2 (objectives and risk) results in the overall success rate on the top level. Likewise, tasking, timing, and resource efficiency are the more detailed information presenta tion of their parent parameter “compliance with mission objectives”.
Having such a hierarchical information structure, we could now adapt the displayed level of detail by choosing the abstraction level that fits the situation. This required an accurate balancing between the highly-abstracted information presentation for the supervision on the one hand and the detailed data view required for an in-depth analysis of the mission status (or the pro posed plans) on the other hand. Here, following our HMI design concept, only information on the top three levels of the abstraction hierarchy is dis played permanently on the HMI. Information from the lower levels of the abstraction hierarchy can be accessed by the pilots on demand.
Figure 2 shows how the information on the top three levels of the abstraction hierarchy is visualized on our HMI in the plan overview area. The title “Current” denotes that the characteristics of the currently running mission plan are depicted. The expected overall mission success rate is dis played below by a “progress bar” and a percentage at the top (here: 40%). The second-level parameters are also represented by bars, percentage value, and a label. In this example, the plan’s expected compliance with mission objectives is 60% and the risk level is 80%. The characteristics on the third abstraction level are depicted by icons grouped under the second-level param eter which they belong to. The details of this color-coded icon representation are explained in the following section. When the mission planning system suggests alternative mission plans, these are represented in a similar manner, with e.g. the title denoting them as “Plan A” or “Plan B” (details below).
Color-Coded Icon Representation The purpose of the plan overview area depicted in Figure 2 is to give the pilots a summary of a mission’s expected outcome when following the respective plan. It must enable them to supervise the mission execution and to quickly recognize critical issues.
* A full list of all physical functions and elements on level 4 and 5 of the abstraction hierarchy is beyond the scope of this paper.
To achieve this, we applied an urgency-based color-coding concept: Light grey icons represent the desired state, while yellow and red symbols indi cate moderate and significant deviations. In the example in Figure 2, the risk for the own UAVs is significantly increased. Thus, the respective icon is colored red. The severity of this issue becomes even clearer as the super ordinate risk bar indicates an 80% risk rate for the overall mission plan. Together with only 60% compliance with the mission objectives caused by significant tasking issues (red icon) and a moderate resource efficiency problem (yellow icon), this results in an overall mission success rate under the current plan of only 40%. Therefore, the bar indicator on the top is also colored red.
According to a comprehensive study by Friedrich & Vollrath (2022) this coloring scheme is a suitable attentional guidance concept for such an icon- based supervisory display. The salience of the selected colors corresponds to the urgency of the function state, supporting parallel visual search. Thus, it enables operators to quickly identify problems during plan execution.
Figure 2: Icons representing the properties of a plan.
Figure available in the original PDF paper.
OVERALL HMI DESIGN The presented mission plan overview is integrated into a larger graphical user interface for the management of multiple UAVs from a fighter cockpit. It has several layouts for different phases of the pilots’ mission management. When the current plan is executed by the UAVs, the HMI offers a layout designed for the operators to supervise the correct plan execution. If the mission planning system identifies an issue and proposes alternatives to the current plan, the HMI offers a layout that can be used to compare the avail able new plans. The details of both general layouts are presented in the two following subsections.
Layout for the Supervision of Current Plan Execution
Figure 3 shows the HMI layout for the supervision of the current mission execution. It includes three main areas: a large top-down map view (1), a mission plan overview area (2), and a details panel (3).
Figure 3: HMI layout for supervision of the current plan execution.
The map view shows spatial information like the positions of targets, known assets, their intended trajectories, et cetera. Also, additional infor mation is visually coded into the map symbols. For instance, the graphical representation of an asset indicates its type, identification, status, and other parameters.
As described in the previous section, the plan overview area shows the status of the currently executed mission plan based on the top three levels of the identified abstraction hierarchy. In the depicted case all status bars and icons are white and grey, which means that there is no action required by the pilots. If they still want to check a certain aspect of the plan, they can access detailed information by clicking on an icon. This opens a corresponding view in the details panel (area 3 in Figure 3), which shows more parameters – from lower levels of the abstraction hierarchy – for the operators to investigate potential issues more thoroughly. In the illustrated example, the timing icon was used to open the UAV task scheduler: It visualizes the timelines of all tasks conducted by the own UAVs.
Layout for the Re-Planning Assisted by the Mission Planning System The HMI also provides a solution for situations in which the pilots must choose between alternative plan proposals generated by the mission planning system. The main characteristics of all alternative mission plans are shown side by side in the plan overview area. The use of the same color-coded icons and bars for each mission plan enables a quick comparison of their properties.
Assessing mission plans suggested by the system is especially important if the system cannot definitely determine one optimal solution. This could be the case in situations that require a realignment of mission priorities or a decision between solutions with disparate benefits and limitations – e.g., one plan might offer a high task success rate at the risk of low UAV survivability, whereas the other plan promises a lower risk for the assets accompanied by a lower task success rate. Further, the human operator is crucial if an ethical review is needed.
Figure 4 shows a screenshot of the developed HMI layout in an exem plary situation where it depicts the plans for a strike against an airfield. Due to a pop-up surface-to-air missile system near the runway, the initial/ current plan cannot be successfully executed anymore. This is indicated by a decreased overall mission success rate of the current mission plan, which is caused by an increased risk for the own UAVs and tasking issues (see red icons). Thus, the mission planning system proposes two alternative plans. The HMI shows the properties of the two alternatives next to the current plan in the mission plan overview area (2). The operator can directly com pare the plan parameters side-by-side. Further, two plans can be selected to be shown in parallel in the map view (1) and in the details panel (3). In this example, plan A (blue color) and plan B (brown color) are chosen to be displayed, so that the operator can directly observe in the map which UAV† is assigned to which target for the respective plan. Furthermore, the timing details panel shows side-by-side when each task is to be performed in the respective plans.
DISCUSSION
We presented an HMI for the mission management of multiple UAVs from a fighter cockpit. Overall, our goals for the HMI were (1) to enable pilots to easily monitor the current mission carried out by the UAVs under their com mand, and (2) to enable pilots to easily understand mission plans suggested by an intelligent mission planning system (with as much detail as desired and time allows). We followed a validated HMI design approach to achieve these goals (Friedrich, 2023). Further, our solution implements the recom mendations of a high level of automation in swarm management (Hocraffer & Nam, 2017) and the application of a user-centered development approach (Huttner & Friedrich, 2023).
Our solution is a blend between ecological interface design (EID) (Vincente & Rasmussen, 1992) – where all information from all levels of the abstraction hierarchy is always displayed – and the dark display design approach (Abbott,
† The eight UAVs in this example have the callsign „RC-1“ to „RC-8“.
Mission Management in Human Autonomy Teams 235 1990; Novacek, 2003), which visualizes only the deviations from the normal states. It strikes a balance between these two extreme approaches: The imple mented mission plan overview makes the functional purpose, the abstract functions, and their respective generalized functions always available, while the more detailed lower levels of the abstraction hierarchy (physical functions and form) can be accessed by the pilots whenever necessary. Furthermore, the mission plan overview is based on a proven color-coded icon visualization that makes deviations from the desired state salient and therefore quickly recognizable for the pilot. This design approach has previously been evaluated in multiple simulation studies involving both, commercial airline and drone pilots (Friedrich & Vollrath, 2021; Friedrich & Lieb, 2019; Friedrich, 2021). The results of these studies show that the display design approach effectively applies the EID approach, allowing operators to understand system states while minimizing information overload. The results show that participants accurately identified non-normal system states, and rated the displays as clear, logical, and well-organized.
1 2 3
Figure 4: HMI layout for the mission re-planning assisted by the mission planning system.
Following our user-centered approach, the next steps involve the evalu ation of the HMI with fighter pilots as subject matter experts and further development in future iterations – again in close cooperation with active fighter pilots.
CONCLUSION
In future air combat, the pilots of crewed aircraft will probably also be responsible for several escorting UAVs carrying sensors and effectors. It is assumed that their tasks will be to monitor mission execution, to define mis sion goals, set priorities, and to make decisions that require human judge ment. For this, they need an HMI that optimally supports their performance to ultimately ensure maximal mission success. This paper presented an HMI design concept for fighter cockpits which enables the pilots to supervise the current mission execution of UAV under their command, allowing for a quick recognition of emerging issues. Moreover, if replanning of a mission becomes necessary, the graphical user interface facilitates the comparison and decision between alternative plans suggested by an intelligent mission planning backend. Besides the fighter cockpit, our design approach could also be implemented for other types of user interfaces where multiple UAVs have to be managed simultaneously; for instance, in ground control stations or in larger command aircraft.
ACKNOWLEDGMENT
The authors would like to thank all the subject matter experts who took the time to participate in our interviews and workshops.
REFERENCES
Abbott, T. S. (1990). A simulation evaluation of the engine monitoring and control system display (NASA TP-2960). Baldy, I., Rossetti, L. (2022). Human-Machine Interface: An Evolutionary Necessity.
The Journal of the JAPCC, Volume 34, Summer 2022, pp. 35–41. Friedrich, M. & Lieb, J. (2019). A Novel Human Machine Interface to Support Supervision and Guidance of Multiple Highly Automated Unmanned Aircraft. Digital Avionics System Conference (DASC), San Diego, CA, USA. Friedrich, M. & Vollrath, M. (2021). Human Machine Interface Design for
Monitoring Safety Risks Associated with Operating Small Unmanned Aircraft Systems in Urban Areas. Aerospace, Volume 8, No. 3. Friedrich, M. & Vollrath, M. (2022). Urgency-Based color coding to support visual search in displays for supervisory control of multiple unmanned aircraft systems. Displays, Volume 74. Friedrich, M. (2021). Human Machine Interface Design to Support Safety Risk
Monitoring of Autonomous Small Unmanned Aircraft Systems – Results from a Mock-Up Evaluation. Digital Avionics Systems Conference (DASC), San Antonio, TX, USA. Friedrich, M. (2023). Display Design to Support Efficient Search in Safety Risk Identi-fication for Supervisory Control of Unmanned Aerial. Dissertation, TU Braunschweig.
Mission Management in Human Autonomy Teams 237 Hocraffer, A., Nam, C. S. (2017). A meta-analysis of human-system interfaces in
unmanned aerial vehicle (UAV) swarm management. Applied Ergonomics, Volume 58, pp. 66–80. Huttner, J.-P., Friedrich, M. (2023). Current Challenges in Mission Planning Systems for UAVs: A Systematic Review. Integrated Communication, Navigation and Surveillance (ICNS) Conference, Washington D.C., USA. Ichaso, R. (2022). Artificial Intelligence – Human Symbiosis in Fighter Aircraft. The
Journal of the JAPCC, Volume 34, Summer 2022, pp. 14–19. Johnson, R., Leen, M., Goldberg, D. (2007). Testing adaptive levels of automation (ALOA) for UAV supervisory control (AFRL-HE-WP-TR2007-0068). Novacek, P. (2003). Design Displays for Better Pilot Reaction. Avionics News, Oct.
2003, pp. 44–47. Patti, R. (2021). Joint operations with unmanned aircraft systems (UAS) and their future development, Combined Joint Operations from the Sea Centre of Excellence, Norfolk, Virginia, USA. Peng, Q., Wu, H., Xue, R. (2021). Review of Dynamic Task Allocation Methods
for UAV Swarms Oriented to Ground Targets. Complex System Modeling and Simulation, Volume 1, No. 3, pp. 163–175. Rasmussen, J. (1986). Information Processing and Human-Machine Interface. New York: Elsevier Science Publishing. Roth, G., Schulte, A., Schmitt, F., Brand, Y. (2020). Transparency for a work
load-adaptive cognitive agent in a manned–unmanned teaming application. IEEE Transactions on Human-Machine Systems, Volume 50, No. 3, pp. 225–233. Vicente, K. J. (1999). Cognitive work analysis: Toward safe, productive, and healthy computer-based work. Boca Raton: CRC Press. Vicente, K. J., Rasmussen, J. (1992). Ecological interface design: theoretical founda
tions. IEEE Transactions on Systems, Man, and Cybernetics, Volume 22, No. 4, pp. 589–606. Özcan, B., Friedrich, M., Huttner, J.-P., Dwinger, K. (2025). Enhancing Trust in Human-AI Interaction Through Explainable Decision Support Systems for Mission Planning of UAS Swarms. AHFE International Conference, Orlando, FL, USA.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2026): Disruptive and Innovative Technologies
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-76-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional