Caroline Kingsley, Laura Mieses, Michael Jenkins (2025). Adaptive HMI for Cyber-Physical Systems: Facilitating Multi-Level System Understanding for Rapid Response and Recovery. In: Human-Computer Interaction & Emerging Technologies. AHFE Open Access, vol. 195, pp. 209–219. https://doi.org/10.54941/ahfe1006239
In complex cyber-physical systems, especially those supporting high-stakes operational contexts, users face increasing demands to quickly comprehend and navigate intricate system data. As these systems grow in functionality, the volume and complexity of data presented on conventional interfaces can overwhelm users, particularly in scenarios requiring rapid response. During unexpected or unpredictable degradations of capability, users must transition swiftly from routine operations to diagnostic and triage actions, often under constrained timeframes. Conventional human-machine interfaces (HMIs) frequently lack the adaptive features needed to adjust not only the information presentation, but also the user interaction mechanisms based on the evolving system state. This limits user situational awareness and diagnostic efficiency. Additionally, traditional HMI designs do not adequately support multi-level system understanding, where both high-level and detailed abstractions of the system are often essential for making informed decisions under pressure. This gap between the increasing complexity of systems and the static nature of HMIs presents a critical challenge for maintaining operational resilience and effectiveness during degradation or recovery scenarios.To address these challenges, we are developing an adaptive human-machine interface (HMI) that leverages a dynamically reconfigurable display composed of digital OLED screens, each functioning as both a visual information source and an interactive control. Drawing on principles from cognitive systems engineering and ecological interface design (EID), the HMI is structured around an abstraction hierarchy, allowing users to view and interact with system states at multiple levels of abstraction, from high-level functional goals to detailed component states. This multi-layered approach enables users to access context-specific information that dynamically shifts in response to ongoing system data displayed on the primary monitor, aligning with the principles of cognitive compatibility. By presenting critical information through adaptive iconography and visual cues on the OLED screens, the system facilitates rapid perception of key operational states, significantly enhancing users’ capacity for efficient diagnostic reasoning during degraded operations.The interface’s adaptability is critical to helping users shift focus seamlessly between strategic and tactical tasks, a need particularly acute during triage and recovery phases when response time is crucial. By encoding higher-order functional information, the adaptive HMI minimizes cognitive load and reduces the risk of errors by guiding attention to actionable insights relevant to the current operational context. This design also supports “direct manipulation” of data at multiple levels, allowing users to interact with core system function information directly from the HMI, thus reinforcing the system’s transparency and intelligibility. These features make it possible to sustain a level of operational resilience even under conditions of reduced capability, as users can rapidly construct a situational model of the system’s behavior and prioritize actions accordingly. The approach’s alignment with ecological interface design principles fosters a robust user interface by emphasizing perceptual cues that map to the underlying system structure, supporting faster recognition and more effective response.This HMI prototype ultimately represents a significant advancement in facilitating cognitive work across varying levels of detail and complexity, enabling users to better understand, diagnose, and restore critical functionality in cyber-physical systems under high-stress, time-sensitive conditions.
In complex cyber-physical systems, especially those supporting high-stakes operational contexts, users face increasing demands to quickly comprehend and navigate intricate system data. As these systems grow in functionality, the volume and complexity of data presented on conventional interfaces can overwhelm users, particularly in scenarios requiring rapid response. During unexpected or unpredictable degradations of capability, users must transition swiftly from routine operations to diagnostic and triage actions, often under constrained timeframes. Conventional human-machine interfaces (HMIs) frequently lack the adaptive features needed to adjust not only the information presentation, but also the user interaction mechanisms based on the evolving system state. This limits user situational awareness and diagnostic efficiency. Additionally, traditional HMI designs
do not adequately support multi-level system understanding, where both high-level and detailed abstractions of the system are often essential for making informed decisions under pressure. This gap between the increasing complexity of systems and the static nature of HMIs presents a critical challenge for maintaining operational resilience and effectiveness during degradation or recovery scenarios.
To address these challenges, we are developing an adaptive human- machine interface (HMI) that leverages a dynamically reconfigurable display composed of digital LCD screens, each functioning as both a visual information source and an interactive control. Drawing on principles from cognitive systems engineering (CSE) and ecological interface design (EID), the HMI is structured around an abstraction hierarchy (see Figure 1), allowing users to view and interact with system states at multiple levels of abstraction, from high-level functional goals to detailed component states. This multi- layered approach enables users to access context-specific information that dynamically shifts in response to ongoing system data displayed on the primary monitor, aligning with the principles of cognitive compatibility. By presenting critical information through adaptive iconography and visual cues on the LCD screens, the system facilitates rapid perception of key operation states, significantly enhancing users’ capacity for efficient diagnostic reasoning during degraded operations.
Figure 1: Five-level abstraction hierarchy overview (adapted from Lau et al., 2008).
This paper presents an overview of applying EID principles and employing Work Domain Analysis (WDA) frameworks such as the Abstraction
Adaptive HMI for Cyber-Physical Systems: Facilitating Multi-Level System Understanding 211
Hierarchy (AH) to various complex socio-technical systems such as cyber- physical systems. The discussion includes a range of CPS applications that would benefit from EID and AH, culminating in a more detailed examination of our human-machine interface (HMI) design and prototyping efforts. Specifically, we explore a dynamically reconfigurable display that functions simultaneously as an information visualization tool and an interactive control interface.
APPLYING ECOLOGICAL INTERFACE DESIGN PRINCIPLES TO COMPLEX CYBER-PHYSICAL DOMAINS Cyber-physical systems (CPS) are increasingly integrating artificial intelligence (AI) and machine learning (ML) to enhance real-time monitoring, automation, and decision-making. These systems, which encompass both physical processes and computational intelligence, present unique challenges for human operators who must manage vast amounts of data while responding to anomalies and optimizing performance. Traditional interface designs often fail to support effective decision-making in these complex, dynamic environments. Ecological Interface Design (EID) incorporates Work Domain Analysis (WDA) as a foundational methodology, with Abstraction Hierarchy (AH) serving as one framework within WDA to structure system information in alignment with human cognitive processes. EID enhances situation awareness by structuring data across multiple abstraction levels, enabling operators to diagnose problems, anticipate failures, and take appropriate actions. The following table explores the application of EID and AH in five key CPS domains, emphasizing their potential to improve system monitoring, anomaly detection, and operator decision support.
Table 1: Application of ecological interface design (EID) and abstraction hierarchy (AH) in cyber-physical systems.
Table available in the original PDF paper.
Cyber-Physical System Context Why EID & AH? Challenges Addressed Swarm of Unmanned
Command center
Operators must manage
Systems (i.e., UAVs, Autonomous vehicles)
monitoring a swarm of UAVs for surveillance, disaster response, or search-and-rescue missions.
large-scale autonomous systems at different abstraction levels, from overall mission objectives to individual vehicle status.
Coordination of multiple autonomous agents
Anomaly detection in navigation and communications
Effective human oversight in uncertain conditions Smart Power Grid
Real-time monitoring of Requires multi-level Monitoring & Control energy distribution, integrating renewable sources and automated substations.
decision support, from system-wide energy balancing to troubleshooting individual substations.
Managing demand-response strategies
Detecting cyber threats or grid instabilities
Ensuring resilience to faults and failures Continued
Table 1: Continued Cyber-Physical System Context Why EID & AH? Challenges Addressed
Table available in the original PDF paper.
Maritime Autonomous
Fleet-wide monitoring Balances fleet-level Shipping Fleet of autonomous cargo ships operating across global trade routes.
Ensuring compliance with maritime regulations Industrial Robotics in a
AI-driven automation
Enables hierarchical
Smart Factory
with robotic arms, IoT sensors, and real-time quality control.
oversight, from production efficiency to machine-specific fault detection.
Identifying production bottlenecks
Cyber-physical interactions between AI and human workers
Preventing costly robotic failures Automated Rail
Supervising a
Supports real-time
Transport & High-Speed Rail Monitoring
high-speed railway network with autonomous trains and predictive maintenance
decision-making at system-wide (congestion, efficiency) and train-specific (sensor failures, propulsion issues) levels.
Predicting track and infrastructure failures
Optimizing train schedules dynamically
Balancing speed, energy efficiency, and safety
These examples demonstrate how EID and AH can facilitate effective monitoring and decision-making in complex CPS domains. By integrating multi-level abstraction, these design principles enable operators to maintain situational awareness, manage anomalies, and optimize system performance in increasingly automated and AI-driven environments.
ADAPTIVE HUMAN-MACHINE INTERFACES FOR RAPID RESPONSE AND RECOVERY Taking the example of command and control (C2) of a heterogenous swarm of unmanned systems (UxS) in the context of disaster response, we are exploring HMI design concepts and prototypes aligned with abstraction hierarchy principles to support the monitoring and coordination of UxS in such a scenario. By leveraging multi-layered representations, our goal is to enable operators to seamlessly transition between high-level mission objectives and detailed subsystem diagnostics. The HMI dynamically adapts to operational demands, shifting information display and interaction mechanisms based on real-time system states. This design approach incorporates context-sensitive visualization, emphasizing ecological cues that facilitate rapid comprehension of critical information, while ensuring a coherent representation of swarm autonomy levels, task assignments, and performance metrics. The ability to manipulate system function information
Adaptive HMI for Cyber-Physical Systems: Facilitating Multi-Level System Understanding 213
directly from the interface enhances operator control, fosters transparency, and reduces cognitive load, ultimately supporting effective decision-making in fast-paced, uncertain environments.
Multi-Modal HMI System Architecture As part of this effort, we have conceptualized a multi-modal HMI designed to function as a physical C2 station, enabling Operators to effectively monitor and manage UxS swarms (Figure 2). This system integrates an overhead spatial 3D display for intuitive visualization of 3D objects (e.g., CAD model of UAV) and/or multi-domain geospatial scenario (e.g., UAV and UGS in an urban environment), a curved 2D monitor serving as the primary display, and an array of configurable LCD screen buttons, dynamically driven by software to provide adaptive controls and real-time data visualization.
Figure 2: Example multi-modal HMI system for UxS swarm C2.
The section outlined in blue in Figure 2 represents the array of configurable LCD screen buttons. To meet this need, we are currently exploring Elgato brand’s product line of Stream Decks (Figure 3).
Specifically, the Stream Deck+ offers a combination of tactile dials and customizable touchscreen controls, which could be useful for fine-tuning parameters like camera zoom levels or swarm formation density. Whereas the Stream Deck XL features a significantly larger array of programmable keys, allowing for potential mapping of a larger data visualization (e.g., time series plots) allowing operators to rapidly assess and drill into mission-critical subsets of data. These devices offer a robust, software-integrated solution for dynamically controlling and visualizing UxS swarm operations in real time.
As part of our physical C2 station depicted in Figure 2, we plan to configure the LCD button array interface with two Stream Deck + models, one on either end, and five Stream Deck XLs, oriented vertically across the center (Figure 4).
Figure 4: Stream Deck + and XL configuration for physical C2 HMI.
Design Concepts for Dynamically Configurable LCD Button Arrays By employing WDA strategies and conducting knowledge elicitation (KE) with subject matter experts (SMEs) such as former UxS operators, we parsed an operational workflow into four stages: monitoring, alerting, engaging, and resolving–which are not strictly sequential. Operators may skip, revisit, or engage only select stages before resuming routine monitoring. Depending on the situation, they may briefly assess an alert and return to passive oversight without full engagement. This flexible progression aligns with the adaptive nature of real-world command and control, where responses are shaped by evolving operational demands rather than rigid procedures.
Adaptive HMI for Cyber-Physical Systems: Facilitating Multi-Level System Understanding 215
In the scenario of an operator monitoring a heterogeneous swarm of UxS in a disaster response scenario, key alerts may include:
Mission & Task Status: e.g., updates on task progress, failures, priority shifts, and uncompleted objective
Swarm Coordination: e.g., loss of formation, collision risks, and failures in multi-UxS collaboration
Subsystem Anomalies: e.g., propulsion failures, power depletion, sensor malfunctions, or communication disruptions
Operator Intervention Needs: e.g., requests for manual overrides, system failures requiring human decision-making, and AI-driven escalation points.
Focusing on an alerting use case for UxS swarm C2, for any of the above categories, the following section explores potential design concepts for a configurable LCD button array (or group of arrays), tailored to support operators across the four workflow phases.
Figure 5 represents a notional high-level layout and button mapping for the Stream Deck HMI array. Because these buttons are dynamic and software driven, it is important to note that the displayed content and functions of the individual buttons will change based on the state of the system, as well as the overall layout. For example, when the system is in a monitoring state with no active errors or anomalies detected, the buttons would simply represent an at-rest, monitoring state. However, if a high priority alert was triggered, then the display would reconfigure to prioritize that information (e.g., alert may appear on either the left most or center XL button array with corresponding actions and options displayed on the other arrays.
Monitoring This Stream Deck display section could provide real-time monitoring of UAV and UGV operations, ensuring each deployed unit executes its assigned mission effectively. The structured grid presents key performance metrics: circular status dials indicate system health and operational readiness, bar graphs display performance efficiency, and pie charts break down resource distribution. Each row corresponds to an individual unit or task group,
tracking activity levels and mission progress. The lower section logs timestamps and system data, marking synchronization points and critical mission updates. For example, in a UxS-supported disaster relief scenario, this interface could help an operator maintain situational awareness by visualizing UxS status at a high-level, including task execution monitoring, and enable seamless coordination between autonomous and human-operated teams.
Alerting This Stream Deck display section could function as an alerting interface for UAV and UGV mission oversight, ensuring system stability and rapid response to anomalies. The structured layout categorizes key operational layers, including system impacts, aerial support, ground support, and network status. Status icons, color-coded for severity, provide immediate visual feedback on system health, enabling operators to identify and address critical issues instantly. The time-series plots track performance trends, highlighting deviations that may indicate emerging problems. Dedicated controls for toggling modes and viewing full-page analytics ensure seamless navigation between alert levels and in-depth data analysis. For example, in a UxS-supported disaster relief mission, this interface would alert an Operator
Adaptive HMI for Cyber-Physical Systems: Facilitating Multi-Level System Understanding 217
to any potential or active issues across the UxS team, such as subsystem anomalies (e.g., sensor malfunctions), environmental threats (e.g., weather, jamming), or necessary operator interventions such as a system needing a manual override to effectively carry out a given task.
Engaging This Stream Deck display section is designed for real-time investigation of mission-critical failures, allowing users to expand and analyze specific UAV and UGV clusters. The top section features bar charts that provide performance breakdowns, enabling operators to identify deviations or failures at-a-glance. Below, circular status indicators highlight system health, while dedicated status icons offer a quick assessment of individual unit conditions. The interface includes selectable items, allowing users to drill down into specific units or subsystems for further investigation. A dedicated graphic display and time-series plots provide historical trends and failure patterns, aiding in root-cause analysis. During a UxS-supported disaster relief scenario, this interface could help an operator assess the root cause of an alert more quickly, such as investigating the telemetry data to determine which subsystem is involved in the alerted fault. This interface would ensure operators can quickly diagnose mission issues, isolate problem areas, and take corrective actions to maintain operational stability.
Resolving This Stream Deck display section operates in the resolving (or ‘relaying’) phase, expanding on individual UAV and UGV (UxS) failures with actionable status updates. The left-most column consists of label buttons, identifying specific units or system components. The second column displays status indicators, where graphic icons may signify resolved and/or pending issues and resolutions. The third and fourth columns present bar graphs, visualizing performance metrics and failure severity. In a UxS-supported disaster relief scenario, this interface could assist operators in tracking issue resolution, monitor system health, and relay critical updates efficiently to other stakeholders and team members the Operator would be coordinating with in the context of a given disaster response.
Adaptive HMI for Cyber-Physical Systems: Facilitating Multi-Level System Understanding 219
The adaptability of the configurable LCD button array interface is critical to helping users shift focus seamlessly between strategic and tactical tasks, a need particularly acute during triage and recovery phases when response time is crucial. By encoding higher-order functional information, the adaptive HMI minimizes cognitive load and reduces the risk of errors by guiding attention to actionable insights relevant to the current operational context. This design also supports “direct manipulation” of data at multiple levels, allowing users to interact with core system function information directly from the HMI, thus reinforcing the system’s transparency and intelligibility. These features make it possible to sustain a level of operational resilience even under conditions of reduced capability, as users can rapidly construct a situational model of the system’s behaviour and prioritize actions accordingly. The approach’s alignment with ecological interface design principles fosters a robust user interface by emphasizing perceptual cues that map to the underlying system structure, supporting faster recognition and more effective response.
CONCLUSION
This HMI prototype ultimately represents a significant advancement in facilitating cognitive work across varying levels of detail and complexity, enabling users to better understand, diagnose, and restore critical functionality in a cyber-physical systems under high-stress, time-sensitive conditions. By structuring interface design around abstraction hierarchies, we provide operators with intuitive access to system information across multiple levels, facilitating better situational awareness and more effective decision-making during high-stakes scenarios. Our prototype, which employs multi-modal interfaces including spatial 3D displays and customizable OLED screen arrays, demonstrates an applied use case and interface for real-time adaptation and user-centered interaction.
Future work will focus on expanding the design concepts into an interactive prototype and validating the interface through empirical studies with domain experts, assessing the HMI’s impact on task performance, cognitive workload, and error rates. Additionally, we plan to explore advanced machine learning techniques to further enhance the system’s adaptive capabilities, ensuring that operators receive timely, contextually relevant information to support mission success under dynamic conditions. Ultimately, the principles and design strategies presented here contribute to the ongoing evolution of human-machine interaction in critical operational domains.
REFERENCES
Hall, T. J., Rudolph, J. W., and Cao, C. G. L. (2006). Fixation and Attention Allocation in Anesthesiology Crisis Management: An Abstraction Hierarchy Perspective. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50(1), pp. 1064–1067. Lau, N., Jamieson, G. A. and Itoh, J. (2008). Ecological Interface Design in the
Nuclear Domain: An Application to the Secondary Subsystems of a Boiling Water Reactor Plant Simulator. IEEE Transactions on Nuclear Science, 55(6), pp. 3579–3596.
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-71-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional