Michael Preutenborbeck, Marcel Usai, Nicolas Herzberger, Frank Flemisch (2022). Multimodal Human Systems Exploration with Tangible XR for the Internet of Production: An expert survey. In: Production Management and Process Control. AHFE Open Access, vol. 36, pp. 62–69. https://doi.org/10.54941/ahfe1001620
Agile product development makes it possible to react flexibly to changing customer requirements. In this context, user and expert assessments, explorations and tests are very important, since customer requirements can often be easier identified by testing a first product or a prototype. Due to complex and long development processes, the development of prototypes is often time-consuming and costly. To improve this, we are developing a method in which the use of mixed reality (XR) allows certain product functionalities to be explored at a very early stage by using multimodal virtual prototypes. Traditional virtual reality (VR) addresses only the visual and acoustic modalities and is limited in its immersion, which is especially impeding for products where physical interaction and haptic experience is an important part of the product functionalities. Based on the findings of the multiple resources theory, Tangible XR allows part of this haptic experience to be experienced and integrated with the virtual experience, so that change requests to the product based on user or expert assessments can be identified and implemented at an early stage of the development process of socio-technical systems. In addition to this retrospective user assessment, Tangible XR can also be used to prospectively explore and develop product functionalities in an exploratory manner together with stakeholders. This exploratory paradigm can be an essential component of agile processes. The explorative approach developed is presented by using a Tangible XR car door opening mechanism as a demonstrator. Based on an expert study, a first proof of concept is presented.
The increasing market dynamics present companies with the challenge of how to quickly adapt their products to changing customer requirements in order to be successful in the long term. This new environment needs agility in deve- lopment. Supported by iterative development processes, it is possible to react more reliably and in a more targeted manner to changing requirements. In addition, the testing of prototypes often identifies requirements that are not known in advance.
Multimodal Human Systems Exploration with Tangible XR for the Internet of Production 63
In the German Research Foundation (DFG) funded Cluster of Excellence ‘Internet of Production’ (IoP), agile product development processes are sup- ported using and expanding of data from the digital shadow. In this context, virtual prototyping and testing methods are very important. Testing procedu- res represent a significant expense of the overall development effort. Because of downstream test procedures with physical prototypes, many problems are only identified late in the development process, resulting in a high level of rework. Through virtualization, changes to the prototype can be made much faster and more cost-efficiently than with a physically developed prototype. In addition, virtualization allows entirely new possibilities for human systems exploration, e.g., the participatory design and development of products with users and other stakeholders.
Technological advances such as higher computing power and more pow- erful Human-Machine Interfaces (HMIs) are increasingly enabling virtual prototyping which is achieved through virtual, augmented or mixed reality methods and technology. While visual and acoustic interaction is well sup- ported in this virtualization, e.g. by using VR, other sensory channels are often neglected. In particular, the haptic interaction with the prototype is very important for immersion. This gap is filled by Tangible XR: Visual and acoustic features are presented virtually, e.g. via Head-Mounted Displays (HMDs), while at the same time the relevant haptic components are emulated via physical mock-ups and/or haptic HMIs.
THEORETICAL FOUNDATION The theory of multiple resources provides a decisive indication of the cou- pling of tangible and non-tangible interaction resources. According to this theory, two tasks can only be performed simultaneously if they do not have to share the same interaction resource. The original theory by Wickens et al. (2008) focuses on visual and auditory resources in this context, but the authors suggest in early publications that haptics may also be an important interaction resource. Flemisch et al. (2012) therefore developed the ‘Exten- ded Wickens Cube’, which additionally considers haptics as an interaction resource (Figure 1).
In research on haptic-multimodal coupling, the advantage of processing visual, auditory and haptic resources in parallel has been shown, proving the extension of multiple resources theory to three or more channels (Altendorf et al., 2016). One of the findings from the multiple resources theory is that the perception, is more accurate and/or more robust when different intera- ction resources are addressed (Wickens et al., 2008). Related research shows that also in the field of VR additional interaction modalities such as haptics can increase the sense of presence and immersion (Ramsamy et al., 2006 & Flemisch et al., 2020a). To express what is meant by the term immersion, we orientate to the definition of immersion from Björk and Holopainen from game design theory. Björk and Holopainen differentiate between sensomo- toric, cognitive, emotional, and spatial immersion (2005). In this context, adding haptics has the greatest impact on sensomotoric immersion, but may also influence the other immersion types.
Figure 1: Extended Wickens cube (Flemisch et al., 2012).
The increase of immersion through haptic multimodal coupling can be used for virtual prototyping of human-machine systems. There is increasing research on multimodal prototypes that address haptics as an interaction resource in addition to the visual and acoustic resources (Bourguet, M. L., 2003; Flemisch et al., 2020b; Kelly et al., 2018). The use of multimodal and therefore more immersive prototypes allows to track the voice of customer more accurately and thus to obtain more detailed feedback from user and expert testing (Carulli et al., 2013; Stark et al. 2010). In this context, Meyer et al. derive the concept of the digital shadow of the customer. By using data from the digital shadow, developers can better explore and assess customer requirements in early phases of the development process (2021).
In addition to user and expert studies, multimodal prototypes also con- tribute to human systems explorations. Here, experts, users and other stakeholder jointly develop and evaluate new ideas and interaction meth- ods in a playful and creative way. By doing this, innovation impulses can be generated and integrated into the agile development, in addition to testing functions, that have already been implemented (Flemisch et al. 2022, Flemi- sch et al., 2013). In contrast to experiments, which prove hypothesises and minimize the error of false statements due to statistical variations, explora- tion optimize the finding of good hypothesises, e.g. of good design variants, which can then again be tested in experiments (Flemisch et al., 2019).
APPLICATION SCENARIO As part of the DFG-funded Cluster of Excellence, explorations on the haptic quality of a car door were conducted. In the context of agile product deve- lopment, the application scenario is to respond to rapid change requests in the design of a car door in such a way, that the haptic quality of the opening mechanism is addressed with the virtual prototype appropriately. Here, both the actuation haptics and the surface haptics will be investigated.
The haptic quality of the opening mechanism is expressed via objective measures such as torque and opening angle, as well as subjective measures such as usability or the haptic value of the car door. Part of the explorative
Multimodal Human Systems Exploration with Tangible XR for the Internet of Production 65
Figure 2: Tangible XR for the Internet of Production (IoP).
approach is the design and evaluation of the haptic impressions by different stakeholders, e.g. customers and experts. This evaluation is used to identify the optimal parameters, e.g. torque and resistances, for the car door opening mechanism.
As shown in Figure 2 the Tangible XR demonstrator contains both virtual and physical components. It consists of a frame made of aluminium profiles assembled to a car door mock-up. The mock-up is connected to an actuator that transmits torque to the door via an axis. The user perceives the forces of the actuator as passive resistances when opening the car door.
In addition to the haptic impression, the user visually perceives an augmen- ted version of the car door via a virtual environment. The transfer of the mock-up’s behaviour into virtual reality is done via a Linux-based system that provides torque and angle data of the actuator to a ROS (Robot Opera- ting System) interface. This interface sends data to a model implemented in Unity, which provides the virtual environment with the augmentation of the car door (Meyer et al., 2021).
EXPERT ASSESSMENT Study Sample
As a first step of evaluation, the Tangible XR demonstrator was tested and evaluated in an exploratory pilot study by a group of haptics and VR experts (N = 5) from the Institute of Industrial Engineering and Ergonomics (IAW) at the RWTH Aachen University. The average age of the participants was M = 34.8 years (SD = 9.7). The average experience with the evaluation of human machine systems was M = 12.4 years (SD = 11.88). No participant was incentivized for the 30 minutes of testing.
Study Design The exploratory pilot study took place at the Exploroscope of the IAW. After a short introduction with general information about the procedure and its
estimated duration a familiarization with the tangible XR demonstrator fol- lowed. The participants are given various tasks (T1-T4) in which they interact with the car door. In order to capture as many haptic impressions as possible, the tasks are based on Ledermann’s fundamental haptic exploration strate- gies (Ledermann, 1987). The participants perform the following tasks with the door handle:
T1: Lateral motion
T2: Enclosing
T3: Contour following
T4: Pressing Each task is followed by a short interview focused on the assessment of the perceived immersion. The expert assessment was based on a semi-structured interview guide. The division of the discussion with the interview guide was intentionally served as a rough guideline. Thus, the focus of the discussion could be set individually depending on the field of expertise of the partici- pants. The questions were structured into thematic blocks, which have been directed according to the assessment criteria described in the section below. The interviews were recorded, transcribed and analyzed afterwards.
Assessment Criteria The exploratory pilot study deals with the question to what extent the immer- sion can be increased by the use of Tangible XR. As illustrated above, the immersion of a virtual prototype is one of the key factors in capturing the voice of the customer or other stakeholders as reliable as possible. To inve- stigate the immersion of the Tangible XR demonstrator we are following the classification of Björk & Holopainen (2005):
C1: Sensomotoric Immersion Sensomotoric immersion is the result of feedback loops between human movements during task performance and the sensory output in the virtual environment (Björk & Holopainen, 2005). The study investigates to what extend Tangible XR can increase sensomotoric immersion by adding haptic interaction, as haptic interaction allows virtual objects to become tangible and to be perceived in a different way.
C2: Spatial Immersion Spatial immersion is achieved by the feeling of actually being in the virtual environment (Björk & Holopainen, 2005). In this context, the experts have to assess whether the virtual environment looks and/or feels “real”. When assessing the spatial immersion of Tangible XR a special focus is given to the haptics of surfaces and movements.
Cognitive and emotional immersion are not considered in the context of this study. In the given application scenario of opening a car door, cogni- tive and emotional immersion plays a subordinate role. Nevertheless, these immersion types can be highly relevant for other application scenarios, e.g. the operation and control of safety-critical systems.
Multimodal Human Systems Exploration with Tangible XR for the Internet of Production 67
Results
In the following, the results and key findings of the expert assessment of sen- somotoric and spatial immersion are summarized and analyzed with regard to the given task set (T1-T4). This provides the opportunity to explore the relationships and interactions of haptic impressions and their influence on the perceived immersion: C1: Sensomotoric Immersion 1 Overall, the sensomotoric immersion of the Tangible XR demonstrator is rated as good. The experts surveyed highlight the possibility of grabbing the door handle and interacting with it haptically as very positive.
However, there are differences in sensomotoric immersion between the tasks. For the static tasks (T1-T3), the VR lacks visual feedback about the user’s movements. This problem can be solved by modifying the demonstra- tor and integrating a hand tracking system. In the dynamic task, pressing the door handle (T4), the visual feedback of the hand is also missing, but most experts point out that the interaction with the door makes the movement more comprehensible even without hand tracking.
In addition to the comprehensibility of movements in the VR, visual-haptic feedback also plays an important role in sensomotoric immersion. In this context, the experts emphasize that the visual-haptic feedback for the door handle is adequate, due to its correct positioning in the room. Only the perceived proportions of the door handle do not precisely match the visual impression.
However, most experts point out in this context that for many other ele- ments of the cockpit (e.g. rearview mirror), haptic-visual feedback is not available at all or only very weak. This means that objects can be seen in VR but not touched, or conversely, that objects can be felt through the mock- up that are not visually represented in the VR. This is noted as a distraction factor in the user’s perception. In particular, the lack of a door panel, so that one could reach through it, is irritating when interacting with the door handle during task performance (T1-T4). However, by adding or modifying the phy- sical mock-up with key interaction elements, the haptic-visual feedback can be significantly improved.
C2: Spatial Immersion 1 Spatial immersion addresses the issue of how “real” the virtual environment looks and feels to the user. In this context, the experts assessed both the spatial impression itself and the feel of the surface and the movement of the door handle. Overall, the expert assessments confirm the potential for increasing spatial immersiveness by adding haptic elements to VR.
The spatial feeling in the VR is rated as coherent. The exact positioning of the door handle in the room as well as the possibility to interact with the door in real time contribute to this. Only the proportions are perceived as somew- hat inconsistent, especially when performing the static tasks (T1-T3). Some experts notice, when pressing the door handle (T4), the difference between the virtual and physical model of the door handle is less noticeable (n = 2).
In the static tasks (T1-T3), it is mainly the surface haptics that is perceived by the user. The experts rate the haptics of the surface pleasant, but criticize the texture and material of the 3D-printed door handle. In particular, the abraded side is perceived as rough and does not correspond to the classic feel of door handles. As a result, the perceived realism is also rated lower. To remedy this, the experts recommend reworking the door handle, e.g. by applying leather or another material.
When pressing the door handle (T4), the haptics of movement plays an important role. In this context, it is important for perceived realism that haptic feedback is consistent with the user’s mental models. The car door behavior implemented for the exploratory pilot study does not correspond to the usual behavior of car doors - since only passive resistance was tran- sferred to the door. The experts assess this as rather unrealistic. The classic latching points of car doors are also missed. However, since the actuator allows for an adjustment of the torque, the behavior of the door can be explo- red together with the users and thereby formulated consistently with mental models.
CONCLUSION AND OUTLOOK Extended reality in its forms VR, AR and especially Tangible XR have the potential to support agile development processes through the early parti- cipation of users and other stakeholders. Since haptics represent a central product functionality, early user and expert assessments of virtual and tangi- ble prototypes can be used to draw conclusions for engineering and other domains. This can avoid additional time-consuming and costly iterations in development processes and can improve the overall product quality. An application of mixed reality prototypes in related domains, such as agile pro- duction planning, is also conceivable and has even been implemented in some cases. Despite all the euphoria, there remain a number of research questions, which are also underlined by this explorative pilot study. Is there a measurea- ble advantage/benefit of tangible elements and how much haptics are enough to already take advantage of the technology? The results of the study suggest that a certain level of accuracy needs to be achieved in order to build a consi- stent mental model of haptics and visuality. Excessive discrepancies between the visual and touch environments can actually worsen the immersiveness of the prototype. At the same time - and this is also shown by the results of the study - if the design is appropriate, immersivity can be increased significan- tly and thus much more detailed and accurate feedback can be provided in contrast to common VR methods.
ACKNOWLEDGMENT
Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC-2023 Internet of Production – 390621612.
Multimodal Human Systems Exploration with Tangible XR for the Internet of Production 69
REFERENCES
Altendorf E, Baltzer M, Heesen M, Kienle M, Weißgerber T, Flemisch F (2016).
H-Mode, a Haptic-Multimodal Interaction Concept for Cooperative Guidance and Control of Partially and Highly Automated Vehicles. Springer International Publishing. Switzerland Bjork, S., & Holopainen, J. (2005). Patterns in game design. Hingham, Mass: Charles River Media. Bourguet, M. L. (2003). Designing and Prototyping Multimodal Commands. In Inte-
ract (Vol. 3, pp. 717–720). Carulli, M., Bordegoni, M., Cugini, U.: An approach for capturing the voice of the customer based on virtual prototyping. J. Intell. Manuf. 24(5), 887–903 (2013). Flemisch, F., Bielecki, K., Hernández, D. L., Meyer, R., Baier, R., Herzberger, N. D.,
& Wasser, J. (2020). Let’s Get in Touch Again: Tangible AI and Tangible XR for a More Tangible, Balanced Human Systems Integration. In International Con- ference on Intelligent Human Systems Integration (pp. 1007–1013). Springer, Cham. Flemisch, F., Diederichs, F., Meyer, R., Herzberger, N., Baier, R., Altendorf, E., ... & Kaiser, F. (2020b). Vorreiter: manoeuvre-based steering gestures for partially and highly automated driving. In Smart Automotive Mobility (pp. 231–304). Springer, Cham. Flemisch, F.; Baltzer, M.; Sadeghian, S.; Meyer, R.; Lopez Hernandez, D.; Baier,
R. (2019): Making HSI more intelligent: Human Systems Exploration versus Experiment for the Integration of Humans and Artificial Cognitive Systems; 2nd conference on Intelligent Human Systems Integration IHSI San Diego. Flemisch, F., Preutenborbeck, M., Kehl, C., Grünwald, C., Wasser, J., Baltzer, M., Dahlmann, A. (2022). Human Systems Exploration for Ideation and Innovation in potentially disruptive Defense and Security Systems: Disruption, Ideation and Innovation for Defence and Security for Springer Nature, Springer, in press. Flemisch F, Baltzer M, Altendorf E, Maier S (2012). Balanced Human Systems
Integration (lecture script).RWTH Aachen University, Aachen. Kelly, A., Shapiro, R. B., de Halleux, J., & Ball, T. (2018). ARcadia: A rapid prototyping platform for real-time tangible interfaces. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–8). Meyer R, Preutenborbeck M, Usai M, Briele K, Witzel A & Flemisch FO (2021).
Increasing Agility: A Tangible XR for Rapid Prototype Development, Design Exploration and Usability Assessment. In International Conference on Applied Human Factors and Ergonomics (pp. 85-91). Springer, Cham. Ramsamy P., Haffegee A., Jamieson R., Alexandrov V. (2006) Using Haptics to Improve Immersion in Virtual Environments. In: Alexandrov V.N., van Albada G.D., Sloot P.M.A., Dongarra J. (eds) Computational Science – ICCS 2006. ICCS 2006. Lecture Notes in Computer Science, vol 3992. Springer, Berlin, Heidelberg. Stark R, Krause FL, Kind C, Rothenburg U, Müller P, Hayka H, & Stöckert H (2010):
Competing in engineering design—The role of Virtual Product Creation. CIRP Journal of Manufacturing Science and Technology, 3(3), 175-184. Wickens CD (2008). Multiple resources and mental workload. Human factors, 50(3),
449–455.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Production Management and Process Control
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-12-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional