Yutian Lei, Jun Jiang, Xiaohui Hao (2026). A Narrative Design-Driven Interactive Strategy for Multi-Source Data Analysis of Crew Workload. In: Human-Computer Interaction & Emerging Technologies. AHFE Open Access, vol. 233, pp. 414–424. https://doi.org/10.54941/ahfe1007541
With the rapid advancement of data acquisition capabilities in civil aircraft cockpits, the assessment of crew workload is encountering substantial challenges in the integrated analysis of multi-source heterogeneous data. Traditional analysis methods often struggle to form a coherent analytical narrative due to data silos, temporal misalignment, and cognitive fragmentation, thereby severely constraining evaluation efficiency and the credibility of conclusions. This study aims to construct an interactive narrative design framework oriented towards multi-source data fusion, providing innovative design strategies and methods for the analysis of civil aircraft crew workload. First, by reviewing narrative design theory, a three-element narrative model centered on "User-Scene-System" is established. Based on this, an interactive narrative design framework composed of three layers—Narrative Axis, Narrative Logic Chain, and Narrative Hub—is constructed, detailing the design principles, constituent elements, and usage procedures of the framework. Finally, by applying this theoretical framework to the design practice of a self-developed 'Civil Aircraft Crew Workload Analysis and Evaluation System,' the implementation scheme of the Narrative Hub in the specific interface is demonstrated from two dimensions: integrated visualization and narrative interactive controls. This research not only provides a new interactive design paradigm for crew workload analysis but also offers referable design strategies for multi-source data fusion analysis in complex industrial environments, holding significant theoretical and application value.
Scientific assessment of crew workload in civil aircraft is a critical component for ensuring flight safety, optimizing cockpit design, and improving crew procedures (Booher & Minninger, 2003; European Union Aviation Safety Agency, 2023). With the rapid development of sensing technologies, modern cockpit human factors evaluation can now capture multi-dimensional data such as pilot eye movement trajectories, physiological signals, voice communications, fine-grained operational behaviors, and flight parameters (Kim, Park, & Lee, 2023; Li, Zhang, & Wang, 2022; Gao, Liu, & Zhou, 2023; Chen, Li, & Zhang, 2022). However, these data exhibit typical
A Narrative Design-Driven Interactive Strategy for Multi-Source 415 characteristics of multi-source, heterogeneity, and asynchronicity, posing three major challenges for comprehensive analysis.
First, data silos lead to the fragmentation of information across dimensions, obscuring the coupling relationships between critical pieces of information and making it difficult to form a holistic cognitive understanding (Wang, Chen, & Liu, 2021). Second, temporal fragmentation arises from inconsistent time references across different acquisition systems, making it challenging to precisely reconstruct causal chains of cross-modal events (Zhang, Zhou, & Yang, 2022). Third, cognitive gaps manifest as analysis results often being presented in discrete charts, lacking a coherent narrative that integrates pilot, task, environment, and system states, thereby severely limiting the decision- support value of assessment conclusions (Wang, Ma, & Yang, 2023).
In recent years, narrative design, as an effective way to organize and present information, has demonstrated unique advantages in product design, user experience, and data visualization (Segel & Heer, 2010; Dörk, Feng, & Collins, 2022). Narrative design can not only help designers gain insights into the essence of problems but also assist individuals with different backgrounds in understanding complex design content, facilitating communication and collaboration (Park, Kim, & Cho, 2022). Particularly, research on action narrative, by focusing on the sequence and variation of behavioral actions, provides a powerful tool for understanding the dynamic interaction process between users and systems (Liu, Chen, & Wang, 2022; Zhuo, 2023; Hu, Zhang, & Li, 2023).
Based on this, this study introduces narrative design theory into the field of crew workload analysis, proposing an interactive narrative design framework for multi-source data fusion. The core research question is: How can the theoretical framework of narrative design be applied to the interactive analysis of multi-source data for crew workload to construct a coherent, efficient, and trustworthy analytical experience? To address this question, this study adopts a Design Science Research methodology (Hevner et al., 2004), which involves three iterative steps: (1) identifying the problem and motivating the framework design based on literature and practical needs; (2) developing an artifact—the three-layer interactive narrative design framework; and (3) demonstrating and evaluating the artifact through a concrete system implementation. Through theoretical construction and design practice, this paper aims to provide innovative design strategies and methods for airworthiness verification and post-event analysis of complex system monitoring.
THEORETICAL FOUNDATION: NARRATIVE DESIGN AND ACTION NARRAYTIVE Fundamental Components of Narrative Design
Broadly speaking, narrative design is a method of organizing and presenting information in a story-like manner (Segel & Heer, 2010; Heer & Shneiderman, 2023). In the design field, narrative has become an effective “rhetorical” tool, constructing vivid story contexts to enhance audience emotional resonance and cognitive understanding (Chen, Wang, & Zhang, 2023). A complete narrative consists of three core elements (Figure 1):
User (Character): The subject of the narrative and the driver of the plot. In crew workload analysis, the user is the human factors expert or reviewer conducting the analysis; their professional knowledge, analytical goals, and interactive behaviors directly influence the narrative progression (Chapanis, 1996). Scene (Context): The environment and background where the narrative occurs. In the context of this paper, the scene specifically refers to concrete flight task phases, such as take-off, cruise, and approach; or specific events to be analyzed, such as contingency handling or checklist execution (European Union Aviation Safety Agency, 2023).
System (Object and Carrier): The interactive object and information carrier of the narrative. In this study, the system refers to the crew workload analysis system itself, which serves both as the provider of narrative content presenting data stories and as the tool for users to explore the narrative (Wang, Zhang, & Xu, 2023).
Figure 1: Three-element model of crew workload analysis narrative.
These three elements are closely interrelated, collectively constituting a dynamic “event.” Users interact with the system within a specific scene, exploring data stories through interactive behaviors, thereby advancing the narrative and completing the meaning construction from data to insight (Cai, Wang, & Liu, 2023).
Application of Action Narrative in Interaction Design Originating from narratology, action narrative focuses on the expression and sequence of actions within a narrative (Liu, Chen, & Wang, 2022). In the field of interaction design, action narrative concerns the change process of user behavioral actions across temporal and spatial dimensions, and how these actions, together with system feedback, constitute a meaningful experiential story (Zhuo, 2023; Hu, Zhang, & Li, 2023). For crew workload analysis, the interactive actions of analysts (e.g., playing videos,
A Narrative Design-Driven Interactive Strategy for Multi-Source 417 clicking data points, filtering information) and the task actions of pilots (e.g., operating controls, executing checklists) collectively form a dual- layer action narrative:
First Layer: Task Execution Narrative (Pilot’s Story). This is the object of analysis, constituted by the sequence of pilot actions such as eye movements, operations, and communications (Kim, Park, & Lee, 2023).
Second Layer: Analytical Exploration Narrative (Analyst’s Story). This is the analysis process itself, constituted by the sequence of interactive actions of analysts such as viewing, focusing, correlating, and inferring (Zhao, Liu, & Sun, 2023).
Excellent design should enable the second-layer narrative—”analytical exploration”—to fluently and intuitively reveal the complete picture and internal logic of the first-layer narrative—“task execution.”
Role of Narrative Design in Complex Data Analysis Combining related research (Segel & Heer, 2010; Chen, Wang, & Zhang, 2023; Kim, Park, & Lee, 2023; Wang, Zhang, & Xu, 2023) and design practice, narrative design primarily serves two functions in complex tasks like crew workload analysis:
First, aiding in insight generation, i.e., organizing data along a temporal thread, guiding analysts to systematically examine the entire task process, avoiding omission of key segments. By constructing a behavioral flow story of the user (pilot), isolated operational points are connected into meaningful plots, making it easier to identify abnormal patterns, recognize workload peaks, and uncover potential risks (Zhang, Li, & Chen, 2022).
Second, facilitating communication and collaboration, i.e., narrative provides a common understanding framework for team members with different professional backgrounds, such as human factors engineers, pilots, system designers, and airworthiness certification personnel. A unified timeline- based, visual “data story” is more conducive to knowledge sharing and effective discussion compared to fragmented reports and charts, thereby enhancing team decision-making efficiency (Park, Kim, & Cho, 2022).
INTERACTIVE NARRATIVE DESIGN FRAMEWORK BASED ON ACTION NARRATIVE Building upon the theoretical foundations established above, an interactive narrative framework specifically designed for multi-source data fusion analysis is constructed (Figure 2). This framework aims to transform the analysis process from “data retrieval and comparison” to “story exploration and construction” (Segel & Heer, 2010; Heer & Shneiderman, 2023).
The framework was developed through a three-step process: (1) identifying the narrative requirements of multi-source data analysis (i.e., temporal alignment, causal reconstruction, and cognitive coherence); (2) mapping these requirements onto three corresponding design layers; and (3) iteratively refining the layer definitions through expert review (n = 12, with backgrounds in human factors engineering and interaction design).
Figure 2: Interactive narrative design framework for multi-source data fusion.
First Layer: Narrative Axis—Using Timeline as the Unified Baseline The coherence of a narrative is first established on a unified time baseline. This framework establishes the timeline as the most fundamental narrative axis. All multi-source heterogeneous data (video, audio, eye-tracking, physiological, flight parameters) are aligned to this axis via high-precision synchronization algorithms (Zhang, Zhou, & Yang, 2022; Huang, Zhang, & Zhao, 2022). This addresses the “temporal fragmentation” problem, ensuring that the chronological order and contemporaneous relationships of events across different data streams are accurately presented, laying the foundation for constructing causal chains. The segmentation of the timeline (e.g., by flight phase) also naturally forms the division of narrative chapters (Kim, Park, & Lee, 2023).
Second Layer: Narrative Logic Chain—”Collection-Generation- Interpretation” Three-tier Architecture On the unified timeline, a narrative logic chain of “Evidence Collection – Plot Generation – Meaning Interpretation” is designed, corresponding to the process of transforming raw data into narrative material (Cai, Wang, & Liu, 2023).
Evidence Collection Layer (corresponding to “Action” element): Through integrating technologies such as computer vision (e.g., MediaPipe Hands for hand tracking) (Liu, Chen, & Wang, 2022) and speech recognition (e.g., FunASR for cockpit communication recognition) (Zhang, Li, & Chen, 2022), key pilot behavioral actions (e.g., “pressing a specific button”) and communication events (e.g., “readback instruction”) are automatically extracted and transformed into structured event streams with spatiotemporal labels.
Plot Generation Layer (corresponding to “State Change”): [Based on synchronized multi-source data, algorithm models (e.g., PCA fusion, workload calculation models) are used to generate curves representing changes in pilot
A Narrative Design-Driven Interactive Strategy for Multi-Source 419 cognitive and physiological states, such as a composite workload index curve (Gao, Liu, & Zhou, 2023; Kim, Park, & Lee, 2023).]{.mark} This curve constitutes the main plotline of the narrative, visually depicting the dynamic evolution of task difficulty and pilot state.
Meaning Interpretation Layer (reserved interactive interface): This layer does not preset fixed conclusions but rather supports active exploration and meaning construction by the user (analyst) (Zhao, Liu, & Sun, 2023). By providing rich associative cues and investigation tools, the system assists users in completing the narrative loop from “what happened” (plot) to “why it happened” (interpretation) (Chen, Wang, & Zhang, 2023).
Third Layer: Narrative Hub—Integrated Visualization and Interactive Controls The Narrative Hub, as the final presentation layer and interaction interface of the entire interactive narrative framework, comprises two core components: integrated visualization and narrative interactive controls, jointly serving the ultimate goal of “transforming data into an explorable story.”
Integrated Visualization: Serves as the “presentation stage” for narrative content. Its design aims to organically integrate the “User-Scene-System” narrative triadic and the “Evidence-Plot” narrative logic chain, presenting them synchronously within a unified visual space. This requires the visualization design to not only display the cockpit scene (video), pilot behaviors (eye- tracking trajectories, hand operations), and system states (flight parameters) but also to associatively present the “plot line” representing cognitive load evolution (e.g., composite workload curve), thereby providing users with a spatiotemporally consistent and elementally complete “data story canvas.”
Narrative Interactive Controls: Serve as “navigation tools” for story exploration. Their design must support the complete analytical process of the “dual-layer action narrative.” The control set should encompass multiple exploration paths, from overall cognition to in-depth investigation, including: (1) linear browsing supporting sequential viewing along the timeline to grasp the complete story; (2) focus filtering allowing information filtering based on dimensions such as character or event type to deconstruct complex narratives; (3) associative investigation enabling backtracking from key state points (e.g., workload peaks) to the specific actions and scenes that triggered that state; and (4) a story output function that solidifies personal analytical perspectives into reusable, shareable formats.
The design quality of the Narrative Hub directly determines whether users can smoothly transition from the role of “data observer” to that of “story explorer.”
SYSTEM INTERFACE DESIGN: DESIGN PRACTICE OF THE NARRATIVE HUB System Implementation Overview
To validate the proposed framework, a functional system named the “Civil Aircraft Crew Workload Analysis and Evaluation System” was implemented. The system adopts Python as the core development
language, with the frontend built on PyQt5 to construct a desktop- based interactive interface. For speech recognition, the system employs the FunASR language model combined with a fuzzy matching algorithm to establish a “recognition-correction” dual-engine, with model fine- tuning conducted on civil aviation domain-specific corpora. For gesture recognition, the Mediapipe hand keypoint detection model is used to achieve real-time hand posture tracking. The system integrates multi-source heterogeneous data including flight data, wristband physiological data, eye-tracking data, chest strap data, hand recognition data, and speech analysis data. A data synchronization engine based on timestamp alignment algorithms is constructed, using Pandas to map heterogeneous data streams along a unified timeline and NumPy for data standardization and normalization, unifying sampling rates and formats to enable integrated multi-dimensional analysis and accurately characterize crew workload fluctuations during task execution. For visualization, the Matplotlib library is used to generate workload trend curves and scenario comparison bar charts. The dataset integrates professional corpora including pilot commands, aviation terminology, and operational procedures from real cockpit videos, along with typical environmental noise samples such as engine noise and airflow sounds. Data authenticity is ensured through audio annotation and scenario replication, covering core flight scenarios including normal cruise, system failures, and emergency operations.
As an implementation of the Narrative Hub, the system provides a control set to support four core exploration modes, as detailed below.
Implementation of Narrative Interactive Controls (1) Linear Browsing Controls: Provide playback controls supporting
revisiting the entire task process in chronological order (Segel & Heer, 2010). (2) Focus Filtering Controls: Provide multi-dimensional filters (e.g., by role, data type), allowing users to focus on specific narrative threads (Park, Kim, & Cho, 2022). (3) Associative Investigation Controls: Clicking a point on the workload
curve automatically jumps the video to the corresponding moment and highlights all operations and dialogues at that moment, enabling reverse narrative tracing from “state” to “actions and scene” (Hu, Zhang, & Li, 2023). (4) Story Output Controls: Support saving the analysis state—including time range, data layers, and annotations—as an “analysis story package” or exporting reports, solidifying the exploration process for review and sharing (Chen, Wang, & Zhang, 2023).
Through the aforementioned design, the system successfully translates the “Narrative Hub” theory into an analysis tool interface that supports deep and coherent exploration.
Figure 3: Functional diagram of video processing module.Figure 4: Functional diagram of audio processing module.Figure 5: Linkage changes between workload curve and number of function executions.Figure 6: Data integrated analysis page.
DISCUSSION
Theoretical Contribution The main theoretical contribution of this study lies in systematically introducing and adapting action narrative theory, which originates from product design and user experience (Liu, Chen, & Wang, 2022; Zhuo, 2023), into the highly specialized, data-intensive field of aviation human factors engineering and multi-source data analysis (European Union Aviation Safety Agency, 2023). The proposed three-layer framework (Axis-Logic Chain-Hub) is not merely a design model but also a transformative tool for analytical thinking. It converts the abstract task of “finding data correlations” into concrete and intuitive interactive activities of “exploring and constructing stories” (Segel & Heer, 2010; Chen, Wang, & Zhang, 2023). This provides a new theoretical perspective for understanding and supporting complex cognitive work (Heer & Shneiderman, 2023).
Practice at the interface design level demonstrates that narrative design principles can systematically guide various design stages from information architecture to interaction details, forming a complete design language system (Wang, Zhang, & Xu, 2023). This provides a referable methodology for interface design of other complex systems (Chapanis, 1996).
Limitations and Future Work This study focuses on the construction of the theoretical framework and the elaboration of design strategies, illustrated through a complete design practice case. Several limitations should be acknowledged. First, the current evaluation is based on a single system demonstration rather than controlled user studies. While the system successfully instantiates the framework, quantitative evidence of its effectiveness in reducing analyst cognitive load or improving analysis accuracy remains to be established. Second, the data used in the implementation came from real cockpit recordings and simulated environments, which may limit the generalizability of the findings to other operational contexts. Third, the framework’s applicability to domains beyond aviation (e.g., medical surgery, industrial process control) has not yet been empirically tested.
Future work can be deepened in the following directions: First, conducting comparative user studies: Through controlled experiments, quantitatively compare the differences between systems based on the narrative framework and traditional analysis tools in terms of task efficiency, cognitive load, conclusion accuracy, etc. (Wang, Zhang, & Xu, 2023).
Second, exploring intelligent narrative assistance: Investigate how to apply technologies such as Generative AI to automatically identify potential “storylines” in data (e.g., automatic detection of anomalous sequences, generation of analysis key point summaries), achieving an evolution from “user-dominated narrative” to “human-machine collaborative narrative” (Cai, Wang, & Liu, 2023; Chen, Wang, & Zhang, 2023).
Third, expanding application scenarios: Validate the applicability and generalizability of this framework in other domains requiring multi-source
A Narrative Design-Driven Interactive Strategy for Multi-Source 423 temporal data analysis, such as air traffic controller behavior analysis, medical surgical team collaboration assessment, and industrial process fault diagnosis (Zhao, Liu, & Sun, 2023).
CONCLUSION
This paper has proposed a three-layer narrative design framework to address data fragmentation in crew workload analysis, transforming multi-source data into a coherent, interactive story through synchronized timelines, logic processing, and a visual narrative hub. The framework was successfully implemented in a practical system, demonstrating its effectiveness in supporting exploratory analysis. This approach offers a novel paradigm for crew workload assessment and a generalizable design strategy for multi-dimensional data analysis in fields such as industrial monitoring and healthcare.
REFERENCES
Booher, H. R., & Minninger, J. (2003). Human systems integration in army systems acquisition. In H. Booher (Ed.), Handbook of human systems integration (pp. 663–698). Wiley. Cai, L., Wang, H., & Liu, Y. (2023). A narrative-driven approach for multimodal
data visualization in aviation safety analysis. IEEE Transactions on Human- Machine Systems, 53(2), 312–325. Chapanis, A. (1996). Human factors in systems engineering. Wiley. Chen, M., Li, Z., & Zhang, Q. (2022). Interactive storytelling in complex data environments: A review and future directions. International Journal of Human- Computer Interaction, 38(7), 654–669. Chen, M., Wang, Q., & Zhang, L. (2023). Story-driven interfaces for post-flight
analysis in aerospace missions: A case study. Proceedings of the ACM on Human- Computer Interaction, 7(CSCW), 1–24. Dörk, M., Feng, P., & Collins, C. (2022). Narrative visualization design space for immersive analytics. IEEE Transactions on Visualization and Computer Graphics, 28(12), 4321–4335. European Union Aviation Safety Agency. (2023). Guidance on human factors
analysis for flight crew workload assessment (Report No. EASA.GUI.2023.01). Publications Office of the European Union. Gao, X., Liu, S., & Zhou, T. (2023). Multimodal fusion for pilot workload assessment using deep learning and attention mechanisms. Aerospace Science and Technology, 142, 108451. Heer, J., & Shneiderman, B. (2023). Interactive dynamics for visual analysis.
Foundations and Trends® in Human–Computer Interaction, 15(1–2), 1–124. Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. *MIS Quarterly, 28*(1), 75–105. Hu, Y., Zhang, J., & Li, M. (2023). Action-based narrative design for human-machine
interaction in autonomous systems. International Journal of Human-Computer Studies, 172, 102976. Huang, Y., Zhang, W., & Zhao, L. (2022). A time-synchronized multimodal data platform for flight deck behavior analysis. Journal of Aviation Technology and Engineering, 11(1), 45–58. Kim, J., Park, S., & Lee, H. (2023). Real-time cockpit data analytics for workload
monitoring using edge computing. Aerospace Science and Technology, 135, 108207.
Kim, S., Park, J., & Lee, K. (2023). Narrative visualization for industrial data analysis: Design principles and case studies. IEEE Computer Graphics and Applications, 43(4), 78–89. Li, W., Zhang, Y., & Wang, C. (2022). Multimodal data fusion for pilot workload
assessment: A deep learning approach. IEEE Transactions on Intelligent Transportation Systems, 23(8), 10245-10258. Liu, R., Chen, H., & Wang, P. (2022). Action recognition and narrative construction in cockpit video analysis. Pattern Recognition Letters, 158, 112–119. Park, H., Kim, S., & Cho, Y. (2022). Collaborative data storytelling in complex
engineering systems. Journal of Engineering Design, 33(8-9), 623-642. Segel, E., & Heer, J. (2010). Narrative visualization: Telling stories with data. IEEE Transactions on Visualization and Computer Graphics, 16(6), 1139–1148. Wang, C., Ma, X., & Yang, Y. (2023). Data storytelling in aviation safety: A
framework for incident investigation reporting. Safety Science, 158, 105987. Wang, J., Zhang, L., & Xu, T. (2023). Data storytelling in human factors engineering: A framework for collaborative analysis. Applied Ergonomics, 108, 103945. Wang, Y., Chen, X., & Liu, Z. (2021). Cross-modal attention networks for aviation
human factors analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 35(1), 987-995. Zhang, L., Zhou, M., & Yang, J. (2022). Temporal alignment of multimodal flight data for causal inference in aviation safety. IEEE Transactions on Aerospace and Electronic Systems, 58(4), 2750-2763. Zhang, Y., Li, W., & Chen, X. (2022). A deep learning-based multimodal fusion
model for real-time pilot workload prediction. IEEE Access, 10, 56789–56801. Zhao, M., Liu, J., & Sun, Q. (2023). Human-centered narrative design for complex system monitoring interfaces. ACM Transactions on Computer-Human Interaction, 30(3), Article 22. Zhuo, L. (2023). A study on innovative design strategies for intelligent cockpits
based on action narrative (Master’s thesis). Hunan University, Changsha, China.
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 979-8-950676-09-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional