Tommy Langen, Gerrit Muller, Kristin Falk (2023). Conceptual modeling for Human Systems Integration in Manned-Unmanned Teaming. In: Human Interaction & Emerging Technologies (IHIET 2023): Artificial Intelligence & Future Applications. AHFE Open Access, vol. 111, pp. 91–101. https://doi.org/10.54941/ahfe1004013
Future systems are becoming increasingly complex as autonomous systems work closely with human operators. Transdisciplinary development is needed to create systems functioning in dynamic Manned-Unmanned Teaming (MUM-T) operations. This paper demonstrates the use of conceptual modeling during Human Systems Integration in the early phase of MUM-T development. We used case study research in a company with participatory action research. This paper highlights three suitable conceptual modeling approaches that provide insight into human factors and manpower distribution in the context of manned and unmanned systems in a search and rescue operation. These models include storytelling, visual ConOps, and dynamic workflow models. Through visual support, these models facilitate engaging stakeholder discussions, enhance contextual understanding, and allow for easy modifications during co-creation. They are particularly useful in preparing for workshops and eliciting knowledge from end-user meetings.
Engineers tend to be solution-oriented and interested in the technical details, designing for other engineers, which might lead to the down-prioritization of the Human Systems Integration (HSI). Systems engineers think that people are systems in the same way as machines are. People are more complex than engineered systems (Boy and Narkevicius, 2014). Thus, those who develop systems must gain insight and understanding of the whole system, which means the human system, technical system, components, and their dynamic influence from and on the environment.
In the early phase of product development, various stakeholders have dif- ferent needs for insight, and there is still high flexibility for changes to the system. The disciplines of Systems Engineering use various models and views to illustrate the system’s hierarchy, behavior, and interaction with the envi- ronment. Companies want lightweight architecture for exploration purposes (Engen et al., 2021).
This paper supports the need for engineers to apply lightweight architec- ture and modeling for HSI. Conceptual modeling is a type of lightweight
architectural method. Muller describes conceptual models as being suffi- ciently simplified to aid the modeler in understanding, reasoning, communi- cating, and making decisions with, while also being realistic enough to make sense (Muller, 2015). We, therefore, study the use of conceptual modeling to investigate human factors and manpower aspects of HSI in the development of complex systems.
We conduct two conceptual modeling iterations to prepare for the insight phase with end-users. The system of interest in its environmental context serves as a baseline for the modeling. The context intends to provide a foun- dation for the audience to comprehend the situation in which the system must operate. The contextualization approach helps ensure a shared understand- ing between developers and end-users who may speak different languages (Madni, 2015). The modeling aims to systematically understand the system of interest by establishing what the developers know and do not know. Fur- thermore, the modeling serves as a stepping stone for future development by identifying what data and information to capture and map the key factors for mission success. We conduct iterative modeling as an evolving learning process with new data and information input.
This paper’s research goal is to highlight the potential use of concep- tual modeling in early phase product development of complex systems, emphasizing Human Systems Integration into Manned-Unmanned Teaming (MUM-T). We ask the research question.
How can Human Systems Integrators use conceptual modeling to gain insight into the situational awareness of operators of MUM-T systems?
RESEARCH METHOD The research method was a case study within an industry-academic collabo- ration (Ali et al., 2022). The authors conducted two embedded units of anal- ysis following the action research cycle: Design/Plan, Test/Act, Observe, and Analyze/Reflect. The action research cycle iterated twice over 18 months. The average meeting schedule was once every month, and four workshops were held. The first author used participatory action research (Gaffney, 2008) in ongoing product development. We describe participatory action research as Company team members and academic researchers collaboratively utilizing the artifacts in a project.
The main contributors from the company side were two interaction design- ers, one project engineer, and one project manager. Additionally, software engineers, test engineers, and systems engineers were involved when relevant. The academic participation was based on the main authors’ involvement in the Company, with support from senior systems engineering researchers and four master students.
The conceptual models are evaluated according to their desirability (do the user want it), feasibility (is it organizationally achievable), and viability (does it give a return on investment) (Brown, 2008). Additionally, we measured their modeling values against impact factors explained in Engebakken et al. (2010) as the balance of these factors increases the models’ ability to assist
Conceptual Modeling for Human Systems Integration in Manned-Unmanned Teaming 93
communication. Due to confidentiality, the case presented and its artifacts are fictitious and used for illustrative purposes.
CHALLENGE IN SUFFICIENTLY OBTAINING END-USER INSIGHT The case company supplies high-tech solutions where operators must man- age multiple remote systems in demanding environments and situations. The Company experience an increased demand for remotely controlled and autonomous systems. Their systems process a large amount of real-time data from various sensors. A challenging design factor is the presentation of the essential elements for the operator to make appropriate judgments and deci- sions when tired and under stress without adding information overload. The system of interest in this study focuses on a MUM-T operation between oper- ator and machine set in a Search and Rescue scenario. However, we observe that the Company faces challenges in effectively collecting and utilizing data relevant to HSI in MUM-T.
The Company’s developers seek to understand the end-users who use their system as input for their design. The developers have expressed a need for deeper insight and knowledge regarding how the system is uti- lized. Traditional interactive design development encourages end-user testing and feedback engagement, using rapid prototyping and iterative development best practices. However, the system development in the Company has chal- lenges in accessing end-users. Observing end-users frequently during field operations and providing rapid and iterative feedback is not feasible. The developers recognize the importance of comprehending the system’s legacy state, including its functions and tasks. Therefore, their initial develop- ment focus is to gain insight into the system of interest and its interac- tions within its environment. The goal is to design a system that provides end-users sufficient situational awareness to make informed decisions. We employ conceptual modeling to conduct early lightweight architecting for end-user insight, such as storytelling, visual ConOps, and dynamic workflow models.
STORYTELLING TO COMMUNICATE COMPLEX INFORMATION We utilized storytelling to establish a contextual baseline. The objective of employing storytelling in this context was to facilitate an enhanced compre- hension of the system in its environmental context (Madni, 2015). Story- telling served as a valuable tool for engaging all stakeholders and cultivating a shared understanding, which in turn, facilitates more efficient communica- tion and reasoning around the situation. Since stakeholders possess varying domain expertise, storytelling provided less-experienced stakeholders with a more comprehensive understanding of the technical and human systems at play. We acknowledge that storytelling offers a different level of insight than direct participation during real missions, simulators, or high-level prototypes. However, we see storytelling as a low-fidelity insight tool that is effortless to create and modify, providing a quick and tangible understanding.
Figure 1: Storyboard for storytelling on strategic, tactical, and human level.
We used storytelling to communicate information through a combination of text and visuals, as seen in Figure 1. We divided the storytelling into three main sections, each with its specific purpose. The first section sets the context on a strategic level, providing a zoomed-out view of the scenario of interest. This section aims to help the audience understand the story that led up to the scenario and the environment’s characteristics that can affect the system’s per- formance and decision-making. The storyboard user can better understand the system’s potential challenges and strengths by creating a realistic scenario. The second section focuses on the tactical level, which aims to provide spe- cific details about the system’s mission. We used the five-paragraph order style to convey information effectively. This section explains the situation, mission, execution, administration, and command, a typical communication method in these scenarios. The tactical view uses visualization, such as ter- rain, weather, and vegetation, to create a degree of immersion. Additionally, it highlights the limitations of logistical support. The third and final section emphasizes the human actors in the system, here showing the Team Leader and the Rescue Team. This section aims to provide a personal connection to the system users by giving them a name, age, job title, how long they have been in the organization, and the training they possess. It also explains their situational awareness and decision-making capabilities based on their expe- rience, feelings, and information. By creating a personal connection with the users, the audience can better understand the system’s impact on the people who use it.
Our observations suggest that storytelling is a valuable tool for com- municating complex information through a mix of visualizations and text. Multiple mediums, such as reading, visual, and auditory, can enhance the audience’s understanding. The three main sections of the storytelling, each with its specific purpose, help to create a picture of the scenario, mission, and human actors involved.
VISUAL CONOPS FOR ENTITY RELATIONSHIPS IN CONTEXT Visualization effectively achieves a shared understanding of operational sit- uations (Madni, 2015), (Kjørstad et al., 2020). Using the visual Concept of Operations (ConOps), or illustrative ConOps, we can outline the most rele- vant entities in a typical operation and examine their interrelationships. The
Conceptual Modeling for Human Systems Integration in Manned-Unmanned Teaming 95
primary entities in our Visual ConOps are the System of Interest, friendly human agents and technical systems, potential threats, static structures, infrastructure, and the natural environment. We took a high-level approach for our visual representation since our primary objective was to portray our system’s operational concept in its environmental context.
The Visual ConOps phase is a continuation of the storytelling model, illus- trated in Figure 2. The initial sketch was hand-drafted, incorporating entities deemed relevant to the mission. We use maps and satellite photos to obtain insight into typical vegetation, structure distribution, and topography. As with storytelling, the initial sketch is a baseline for further exploration. High- fidelity models are not a priority at this stage. We employ static visualizations instead of animations in the first iteration to keep the workload manageable. We use labels and relationship lines to help depict the situation and convey a visual narrative. The visual ConOps is supplemented with a brief situa- tion text to describe the scene and reflective questions to emphasize with the end-user.
Figure 2: Visual ConOps of a search and rescue with manned and unmanned drones.
The operational concept was a Team Leader who receives information from multiple sources about the missing person and how to solve the task. The Team Leader orients himself, prioritizes information, and decides upon further action based on his situational awareness of the small drone and the larger unmanned drone. Various vegetation, such as trees and bushes, imposed a lack of view of danger and opportunities. Elevations in the topog- raphy could create blind spots, and the sun’s direction could reduce vision capabilities. We emphasize with the Team Leader (i.e., end-user) based on the
mentioned factors, with human factors such as if the Team Leader is tired, calm, eager, anxious, and others.
DYNAMIC WORKFLOW MODELS IN PARALLEL ACTIVITIES The operation of a system in its environment is subject to numerous simul- taneous and interdependent factors. Dynamic behavioral models facilitate a deeper understanding of the physical relationships between systems and their stakeholders, including the actions’ duration (Muller et al., 2019). Dynamic workflow models, for example, can provide situational awareness and sys- tem reactions along a timeline. Using multiple viewpoints and diagrams in a scenario and reusing internal artifacts is helpful. It provides an overview that facilitates exploration and optimization.
We perform a more detailed multi-view of the workflows to understand the operational procedures and the associated mission objectives in parallel. The conceptualization of the dynamic workflow models and how the mod- els connect to time can be seen in Figure 3. We prefer to use domain-specific frameworks for dynamic modeling that are translatable from existing oper- ations and easily understood by stakeholders. As the case is a search and rescue scenario, we adopt the OODA process as a framework for our mod- eling efforts. OODA is an abbreviation of Observe, Orient, Decide, and Act. The primary factors in dynamic models include activities, time, and relations, as they make the models portray dynamic behavior.
Figure 3: Conceptual view of the dynamic workflow models.
The Workflow is the first developed model. We achieved this by outlining the activities we believed were being performed, considering the previous modeling of storytelling and visual ConOps. We classify the activities into the four categories of the OODA process. Such as assigning area scanning with a thermal vision to the “Observe” category and prioritizing the safest route to the “Decide” category. Next, we identified the most critical activities and determined what tools and resources could be required to execute them, such as optics, radio, navigation system, and rangefinder. Additionally, we
Conceptual Modeling for Human Systems Integration in Manned-Unmanned Teaming 97
assume the duration of each task. Assumptions are used as a baseline, which later can be measured and adjusted with objective data.
We use a swim lanes diagram to visualize the allocation of tasks and infor- mation flow among the systems actors. Our system’s primary actors were the Team Leader, Rescue Team, HQ, small drone, and unmanned drone. We gain insight into how the operational events switch ownership, which can potentially impact the system’s functionality.
We employ a cartoon workflow approach to extend our previous work on visual ConOps and storytelling. The cartoon workflow involves placing mov- able figures representing actors in the operational environment, where each slide depicts one step in time, like creating a storyboard. The background canvas of the storyboard is the scenario map and the geography previously drafted in the visual ConOps. This low-level animation approach allows us to dynamically visualize the actors’ movements and activities. We observed that we could easily and quickly adjust using the movable figures based on stakeholder feedback. These dynamic modeling approaches allow us to com- prehensively understand the system’s functionality and explore the design space.
DESIGN AND DEVELOPMENT WITH CONCEPTUAL MODELING The knowledge from the conceptual modeling phase helped facilitate insight- phase workshops with end-users of the legacy system. The workshops pro- vided additional insight into the use and dynamic interactions for the system of interest and its environment. We ran two workshop sessions with end-users to better understand their decision-making processes and identify potential information overload during task execution. We did conceptual modeling before and after each session. The workshops gave insight into activities such as search and preparation planning at the base and how they evaluate the safety and triaging before entering an incident area. We observe the impor- tance of iteration with feedback, as we did not initially include these activities in the first conceptual modeling phase. Additionally, the workshops work as a verification of our conceptual assumptions. We incorporated the findings into existing and newly created conceptual models.
One example of discovery was that the end-user had to make cognitive adaptations when the system switched between different sensor views. There- fore, the developers must be aware of the mental transition involved in the process of changing camera views. The end-user frequently changes the sen- sors during “Sector Surveillance” and switches between sensors: a thermal imager, a high-resolution camera, and between the two drones. The cameras have different zoom levels and field-of-view on this system. This switching leads to changes in the image’s content and framing and creates challenges for the operators. The insight might suggest that when changing sensors, the operator needs to use cognitive abilities to adjust to the newly presented image. The camera operator cannot achieve near-instant adjustment, even though the technical change happens instantaneously. Future MUM-T sys- tems might amplify this challenge, as changing from one drone sensor to multiple autonomous drones involves an even more significant transition.
Developers must, therefore, consider this challenge when designing new MUM-T systems.
Conceptual modeling supports us in highlighting the known and unknown. Regarding the unknown areas, the conceptual models have helped us gain insight and inspiration for further data gathering and analyzing to turn the unknowns into knowns. Human factors testing goes beyond mere testing; it involves testing the appropriate factors. In our case study, we aimed to max- imize our time and resources. Therefore, conceptual modeling was a useful top-down approach for further testing and development.
EVALUATION OF CONCEPTUAL MODELING We evaluated the combined use of the three conceptual modeling approaches according to their desirability, feasibility, and viability (Brown, 2008) and their modeling impact factor values (Engebakken et al., 2010).
The desirability of the conceptual models has been observed by the researcher during the development team’s weekly integrated product team meetings. The meeting participants were software developers, systems engi- neers, test engineers, and interactive designers. When meeting participants started using conceptual models instead of the traditional textual bullet points, we observed an enhanced contextual understanding and discussion through illustrations of the system and its use. We observed that the developer established empathy for the end-users by verbally describing their interactions while navigating the visuals of the conceptual models. Their forward-leaning pointing and gesticulation suggest that the discussions became more engag- ing due to the visual support. We saw that engagement and participation had some effect through multi-view models, specifically through its quickness of changing the design on the fly. With easy-to-understand simplified drawings and highlighted system features, we saw that participants quickly picked up on the topic of discussion. Even though the models were simplified, they were close enough to understand the system context, while a close-to-reality simulation would not provide a cost-effective benefit for our participated meetings.
The feasibility of using a combination of storytelling, visual ConOps, and dynamic workflow models becomes evident when engaging with end-users and internal subject matter experts. Such models were helpful in the prepara- tion of the workshops in terms of creating our own understanding of the system and its use. The models were beneficial for highlighting the areas where we lacked data, information, and knowledge. Additionally, we gained some awareness of the kind of stakeholders to include in the system develop- ment, from end-user operators to in-house subject matter experts. The result was a focused meeting with end-users, which was valuable due to limited access to these stakeholders. During workshops, the models helped to create a shared understanding between developers and end-users. With pre-made multi-view models, we made system operational assumptions that resulted in engaged stakeholders in terms of system functionality usage and need. Additionally, the multiple views provoked feedback necessary for the devel- opment process, as some assumptions were incorrect. The fidelity level of
Conceptual Modeling for Human Systems Integration in Manned-Unmanned Teaming 99
the models was, on purpose, simple, and still, they were recognizable, under- standable, and acceptable for the end-users. By being close enough to reality, they showed the critical functions and information for the complex system.
The viability of the Company’s Human Systems Integration development process is supported by our findings. The Company desired a method for creating cognitive insight into end-user situational awareness with higher reliance on data-driven decisions. They will reach this desire using a usabil- ity laboratory during their human-in-the-loop simulations. The conceptual modeling creates synergy towards this goal in two matters. Firstly, concep- tual modeling aids developers in testing the right things in simulators due to better insight into the complex system. The format and effort increase the efficiency and effectiveness of simulators. Secondly, conceptual modeling pro- vides early and rapid learning. This agile low-tech approach is particularly beneficial in product and project execution when the prototype and simula- tor are expensive and resource-heavy to create and maintain. The Company is a Systems Engineering oriented organization and uses traditional Systems Engineering architecture and design methodologies. As described, concep- tual modeling aligns with such practices and is a stepping stone toward detailed architecture as they build on the same philosophy. The main differ- ence is that the low-fidelity conceptualized storytelling, visual ConOps, and dynamic workflow models are more iterative for evolving learning purposes and transdisciplinary for cross-language challenges with several stakeholders than traditional architecture and design.
THE USEFULNESS OF CONCEPTUAL MODELING DURING HUMAN SYSTEMS INTEGRATION We asked how Human Systems Integrators can use conceptual modeling to gain insight into the situational awareness of operators of MUM-T sys- tems. Engineers can use conceptual models as a planning and communication platform for internal development and external workshops during product development. Our observations suggest that models that established a sense of the situation were valuable. Diagrams and model types were useful: car- toon storyboard, user profiles, visualization of the tasks, communication, and their relations. We find that using storytelling, visual ConOps, and dynamic workflow models is appropriate for creating situational understand- ing, such as mapping dynamic behavior between a team of human operators and machines. Specifically, the swim lanes models were insightful for commu- nications and information handovers between the end-users and mechanical systems. Storytelling promotes encouragement and context to arguments and ideas. Visualization of the system and its operation provoked feedback and shared understanding.
We propose using conceptual modeling as an early-stage work during Human Systems Integration. The results of this study highlight the impor- tance of using conceptual modeling as an integrated part of Systems Engi- neering processes such as architecture, design, and testing. The studied approaches, storytelling, visual ConOps, and dynamic workflow models were practical during insight into human factors and manpower of complex
systems and the combination of these two HSI aspects. Human factors and manpower distribution are two central aspects when developing MUM-T sys- tems. Our work’s novelty lies in testing conceptual modeling in a company that develops complex systems for search and rescue, where the need was to establish an understanding of human factors and dynamic behavior in MUM-T systems. These findings provide insight into how engineers, inter- active designers, and end-users collaborate. Additionally, how to get useful data insight for early phase development of their Human-Machine Interface. Future research combines modeling with human factor data by following a conceptual framework for data sensemaking (Langen et al., 2023).
CONCLUSION
This paper demonstrates conceptual modeling as a suitable approach for Human Systems Integration in the early-phase of Manned-Unmanned Team- ing development. The paper is based on a case study of a company with limited access to end-users and high-fidelity human-in-the-loop simulators. Despite the limitations, the case company needs to map dynamic behav- ior between a team of human operators and machines. The paper reports on the need for an agile, accessible, and resource-friendly method that also supports the exploration and testing of various Human-Machine Interface solutions. The modeling approach focused on human factors and manpower distribution in the context of manned and unmanned systems in a search and rescue operation. This paper illustrates the use of three suitable modeling approaches.
Storytelling facilitates efficient communication and understanding among stakeholders. It is a low-fidelity insight model that is effortless to create and modify, making it a quick and tangible way to communicate complex information through a combination of text and visuals.
Visual ConOps uses an illustration of the system’s operation to communi- cate a visual narrative of its characteristics and interrelationships. It provides a common understanding that enhances further elaboration and modeling of various system viewpoints among various types of stakeholders.
Dynamic workflow models provide a comprehensive understanding of the system’s functionality and explore the design space. The models allow us to visualize the actors’ movements, activities, and relationships. We find the combination of task workflow, swim lane, and cartoon workflow along a common timeline framework to be useful.
Through visual support, these conceptual models facilitate engaging stake- holder discussions and enhanced contextual understanding. It is easy to change between such models during co-creation. The conceptual models are helpful in preparation for workshops and knowledge elicitation from end-user meetings.
ACKNOWLEDGMENT
The authors are grateful to the Company’s people who have participated in this research. The Norwegian Research Council grant number 317862 funded this research.
Conceptual Modeling for Human Systems Integration in Manned-Unmanned Teaming 101
REFERENCES
Ali, H. B., Langen, T., Falk, K., 2022. Research methodology for industry-academic collaboration – a case study. INCOSE International Symposium 32, 187–201. https://doi.org/10.1002/iis2.12908 Boy, G. A., Narkevicius, J. M., 2014. Unifying Human Centered Design and Sys-
tems Engineering for Human Systems Integration, in: Aiguier, M., Boulanger, F., Krob, D., Marchal, C. (Eds.), Complex Systems Design & Management. Springer International Publishing, Cham, pp. 151–162. https://doi.org/10.1007/ 978-3-319-02812-5_12 Brown, T., 2008. Design Thinking. Harvard Business Review 86, 84–92. Engebakken, E., Muller, G., Pennotti, M., 2010. Supporting the system architect: model-assisted communication. Syst. Res. Forum 04, 173–188. https://doi.org/ 10.1142/S1793966610000211 Engen, S., Falk, K., Muller, G., 2021. Conceptual Models to Support Reasoning in
Early Phase Concept Evaluation - a Subsea Case Study, in: 2021 16th International Conference of System of Systems Engineering (SoSE), pp. 95–101. https://doi.org/ 10.1109/SOSE52739.2021.9497467 Gaffney, M., 2008. Participatory action research. What makes it tick? Kairaranga 9, 9–15. https://doi.org/10.54322/kairaranga.v9i3.131 Kjørstad, M., Falk, K., Muller, G., 2020. Exploring a co-creative Problem Solving
Toolbox in the Context of Norwegian High-Tech Industry. IEEE Systems Journal 1–11. https://doi.org/10.1109/JSYST.2020.3020155 Langen, T., Ali, H. B., Falk, K., 2023. A Conceptual Framework for Data Sensemaking in Product Development—A Case Study. Technologies 11, 4. https://doi.org/ 10.3390/technologies11010004 Madni, A. M., 2015. Expanding Stakeholder Participation in Upfront System Engi-
neering through Storytelling in Virtual Worlds. Systems Engineering 18, 16–27. https://doi.org/10.1002/sys.21284 Muller, G., 2015. Tutorial Architectural Reasoning Using Conceptual Modeling, in: INCOSE International Symposium. Seattle, WA, USA. Muller, G., Falk, K., Syverud, E., 2019. Systems of Systems Architecting and Inte-
gration; Visualizing Dynamic Behavior and Qualities, in: 2019 14th Annual Conference System of Systems Engineering (SoSE), pp. 376–381. https://doi.org/ 10.1109/SYSOSE.2019.8753804
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-87-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional