Michael Preutenborbeck, Frank Flemisch, Nicolas Daniel Herzberger (2024). Haptics in Virtual Assembly Systems: Insights into the Current State of Research. In: Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems. AHFE Open Access, vol. 119, pp. 609–618. https://doi.org/10.54941/ahfe1004526
In a rapidly evolving competitive landscape, manufacturing companies are under pressure to reduce the time-to-market for their products. Technological advances, particularly in immersive technologies such as virtual reality, augmented reality, or mixed reality, offer new tools to address this issue by accelerating production ramp-up. Applications for immersive technologies include product development, assembly planning, and assembly training. This paper addresses the integration of haptic feedback into virtual assembly systems. In order to provide a comprehensive overview of the current state of research, a systematic literature review is conducted focusing on applications, study designs, haptic devices used, and the effectiveness of haptic feedback. The review shows that haptic feedback can increase immersion in virtual environments and thus improve information quality. This paper highlights the potential of future virtual assembly systems and emphasizes the importance of ongoing research to fully explore and exploit the capabilities.
Technological progress in the field of immersive technologies, like virtual reality (VR), augmented reality (AR) or mixed reality (MR) opens up a wide range of applications in the product lifecycle. In this context, haptic feedback is emerging as a useful tool. It allows improving the immersion and promises to increase the quality of the information provided, which is already investi- gated in previous research under the term tangible XR (Preutenborbeck et al., 2022, p. 68). The importance of haptic interaction with virtual objects in the con- text of assembly systems has been investigated by a large number of authors as shown by Xia et al. (2013). Force-reflective haptics create a closed loop that links visual data with human motor control and cogni- tion. Haptics is therefore ideally suited to understand the interactions and processes involved in assembly (Gonzalez-Badillo et al., 2014; Liu et al., 2015, p. 207).
Virtual assembly refers to the assembly and disassembly of virtual objects in virtual environments (Xia et al., 2013) and has a wide range of potential applications as identified in the literature (Jayaram et al., 1997; Seth et al., 2011; Wu et al., 2019, p. 973).
Table 1. Virtual assembly applications.
Table available in the original PDF paper.
Ergonomics analysis Typical ergonomic analysis such as visibility and accessibility issues as well as the feasibility of assembly operations can be evaluated Process planning Physical realization of assemblies, can be planned and verified in terms of feasibility Tool design Tools and devices that are required in the assembly process can already be verified virtually Layout planning Assembly processes can be simulated and requirements for the workstation layout can be captured Design for assembly Product geometries can be examined for their suitability for assembly, early in the product development process Virtual training Production workers can be trained for new assembly processes even before the physical realization of new workstations
RELATED WORK
Previous literature reviews form the scientific foundation and starting point for our systematic research on virtual assembly systems in the manufacturing industry are: Azofeifa et al. (2022) present a systematic literature review on multimodal human-computer interaction in different application domains. They focus on how technologies have changed over time. They state that classical VR methods are now widely used in industry and science, the number of studies including haptic interaction is increasing and further research in this area is very promising for the future (Azofeifa et al., 2022).
Radhakrishnan et al. (2021) provide a comprehensive literature review of the use of immersive VR technologies for industrial skill training in vari- ous domains. They examine studies in terms of research design, technologies used and effect on training. Based on the review, the authors conclude that immersive VR can be used very effectively for industrial skill training and thus may be promising for future development in this field (Radhakrishnan et al., 2021).
These reviews show that research on the integration of haptic devices into virtual environments is increasing and what potential lies in immersive virtual environments for various applications. This review aims to investigate the current state of manufacturing specific research by answering the following research questions:
RQ1: What applications for haptic assembly systems exist?
RQ2: Which research designs and data collection methods are used?
RQ3: What haptic devices are used?
RQ4: How effective is the haptic feedback in these applications?
Haptics in Virtual Assembly Systems: Insights Into the Current State of Research 611
METHODOLOGY
This systematic literature aims to present a comprehensive overview of rele- vant papers, which are analysed. The filtering process was performed using the PRISMA-method according to Moher et al. (Moher et al., 2009). Filtering is based on defined inclusion and exclusion criteria according to Figure 1.
Figure 1: Applied PRISMA-method according to Moher et al. (2009).
The databases used were Scopus, Web of Science (WoS), and IEEE Xplore. Scopus lists a wide range of publications from different domains, both the technical field and the social sciences and medicine. WoS lists mainly pub- lications from the social sciences, arts and humanities. IEEE Xplore lists publications from the technical domain, mainly computer science, infor- mation technology and engineering. The selection of the three databases provides as complete an overview as possible of the relevant literature from the different domains.
The search query for title and abstracts was defined: (“virtual”OR “mixed-reality”OR “mixed reality”) AND (assembl* OR disassembl*) AND (haptic* OR multimodal* OR “force-feedback” OR “force feedback”) AND (manufacturing OR production)
The restriction to haptic systems was important in order to filter out those VR systems for virtual assembly that explicitly investigate the influence of
haptics. In addition to the term haptic, the terms multimodal and force feed- back are often mentioned in this context, which is why these two terms were also added as an extension of the search radius.
RESULTS
The included studies (n = 16) were analyzed with respect to the research ques- tions formulated above. An overview of all results of the systematic literature review is presented in Table 4.
RQ1: What Applications for Haptic Assembly Systems Exist?
Virtual assembly systems are mostly used to elicit early feedback from oper- ators, facilitating the integration of this feedback into the design of work tools (n = 2) or production processes (n = 7). As shown in Table 4, many studies include several applications. The primary goal of haptic enhanced process planning is to predict future processes regarding ergonomic feasibil- ity (Kind et al., 2020; Wolf et al., 2020) or process time (Al-Ahmari et al., 2016). Subsequent studies investigate the potential of virtual assembly sys- tems in automated generation of ergonomically optimized assembly plans (Gallegos-Nieto et al., 2020). Another large application field are virtual train- ing systems, as evidenced by a significant number of studies (n = 6). A survey of the existing literature affirms the efficacy of training systems, particularly in the context of procedural and sensorimotor industrial skills (Radhakrish- nan et al., 2021). The emphasis of virtual training within the manufacturing sector can be directed towards the introduction of new products or the train- ing of highly safety-sensitive scenarios in manufacturing processes (Neges et al., 2018). Virtual assembly training is particularly advantageous when handling complex products or in situations where damage to components or the entire product could result in substantial expenses (Abidi et al., 2018).
Moreover, virtual assembly systems find application in product develop- ment, fostering integrated approaches to product and production design. Design-for-assembly approaches can yield significant reductions in assembly times and production costs (Abidi et al., 2018; Read et al., 2017). In the area of virtual assembly systems, factory planning and remote expert applications currently still play a minor role (n = 2).
RQ2: Which Research Designs and Data Collection Methods Are Used?
The studies included in this review predominantly adopt a theoretical or con- ceptual approach, emphasizing the system development and the outcomes (n = 7). Beyond these conceptual papers, some studies conduct evaluations of developed system concepts, utilizing the objective and subjective measures outlined in Table 3.
These evaluations are either qualitative or quantitative. Qualitative evalua- tions (n = 4) involve expert interviews or case studies to assess various aspects of the system, such as usability or immersion within virtual assembly systems. This research approaches also provides insights that allow generalized con- clusions about the systems. Quantitative evaluations (n = 5) can consist of
Haptics in Virtual Assembly Systems: Insights Into the Current State of Research 613
both objective and subjective measurements. The primary objective measure employed is the assembly time (n = 4). Wang et al. (2022) also incorporate the number of errors as an additional objective measure in their evaluation. Subjective measures are employed using standardized questionnaires, such as the Simulator Sickness Questionnaire (SSQ), or custom-designed question- naires tailored to the specific study. A number of the included studies (n = 4) employed these individualized questionnaires to evaluate usability, workload, and immersion or sense of presence.
Figure 2: Applications divided by industry sectors.
RQ3: What Haptic Devices Are Used?
To increase the immersion during the interaction with virtual objects, a vari- ety of haptic devices has been developed. To classify the types of devices, we follow a taxonomy based on Seifi et al. (2019). The authors distinguish the following device classes and functionalities in the field of haptic devices: Hybrid haptic devices that combine virtual tools with physical components are predominantly used in various applications. Their utility is particularly evident in ergonomic analysis (n = 4) and process planning (n = 2), where the combination of accurate physical representation and virtual visualization is of great advantage. Physical interaction with the virtual environment is critical for identifying ergonomic issues. Grounded force feedback devices are preferred in product development (n = 3) due to their exceptional preci- sion in haptic feedback. In contrast, ungrounded force feedback devices are more likely to be used in assembly training, where mobility is an essential requirement (n = 2).
RQ4: How Effective is the Haptic Feedback in These Applications?
To answer this research question, only studies that also qualitatively or quan- titatively evaluated a haptic enhanced assembly system were considered. The haptic effectiveness can be assessed either directly by evaluating the haptic
Table 2. Haptic devices based on Seifiet al. (2019).
Table available in the original PDF paper.
Kinesthetic devices Grounded Ungrounded
Stimulate muscular activity Stationary force feedback devices with a relatively small workspace, e.g. Delta Robot, Haption Virtuose 6D, Phantom Premium 3.0 Mobile force feedback devices, e.g. data gloves
Tactile devices Vibration Fingertip
Stimulate the skin surface Haptic feedback via actuators vibrating on the body, e.g. smart bracelets Physical or ultrasound-based fingertip feedback Hybrid haptic devices Stimulate muscular activity and the skin surface Physical artefacts, e.g. 3D-printed, which are tracked and transferred to the virtual environment.
system or indirectly by enabling functionalities that would be unattainable in a virtual environment without haptic attributes.
Table 3. Influence of haptics on the effectiveness.
Table available in the original PDF paper.
Authors Influence of haptics on the effectiveness Wang et al. (2022) Haptic feedback improves collaborative work in remote expert applications Gallegos-Nieto et al. (2020) Haptic input devices enable intuitive user interaction with virtual assembly parts and automated generation and evaluation of assembly plans Grajewski et al. (2020) Increases the realism and immersion of the VR with significant impact on training success Wolf et al. (2020) Increases the immersion and reliability of various tasks in assembly planning Bu´n et al. (2019) Haptic feedback via forklift steering wheel can increase the effectiveness of training and improve participant performance Buchholz et al. (2017) Haptic interaction elements in virtual production systems for the evaluation of assembly and ergonomic issues Dombrowski et al. (2017) Use of haptic feedback in user interfaces to simulate assembly tasks to improve task performance Read et al. (2017) Haptic interaction with virtual prototypes support design for assembly in early stages of product development Grajewski et al. (2015) Haptic technologies can be used for effective training and for testing the workstation’s ergonomic quality Grajewski et al. (2013) Haptic devices and 3D printed artefacts increase realism of user feelings
Haptics in Virtual Assembly Systems: Insights Into the Current State of Research 615
Table 4. Systematic literature review overview.
DISCUSSION
Since haptics is a crucial part of human perception, the integration of haptic interaction in virtual environments is promising, especially for manual activi- ties. Currently, haptic interaction is mainly realized via hybrid haptic devices, i.e., 3D-printed physical elements, that are tracked and transferred to the vir- tual simulation environment. In this way, the human perception of haptics can be combined with the virtual representation of visual and acoustic inter- action modalities. The potentials in terms of immersion and applications in the system development process have been demonstrated by research under the keyword tangible XR (Flemisch et al., 2020; Preutenborbeck et al., 2022).
The literature review shows that different haptic devices are used depend- ing on the application. Grounded force-feedback devices provide the most precise haptic feedback. However, these devices limit mobility and workspace, which is why they are only suitable for special applications, e.g., the design for assembly in product development. Ungrounded force-feedback devices, such as haptic gloves, are more flexible, but they only provide kinaes- thetic feedback for fingers. This is the reason why they are mainly used where feedback is just required for fingers e.g., in virtual training. There are first approaches for a use-case specific and task-oriented identification of suitable haptic devices (Rückert et al., 2018).
The literature review shows that the integration of haptics can significantly increase the realism and immersion of a virtual assembly application. This results in a higher sense of presence and a higher reliability of subjective impressions when planning and evaluating new workstations or processes. In the case of virtual training, it has even been demonstrated on the basis of objective measures that fewer assembly errors can be made and the trainee’s learning time can be reduced through haptic enhanced training systems (Grajewski and Hamrol, 2020). In addition to the advantages regarding the sense of presence and the subjective reliability of user assessments, there are also additional assessment possibilities, especially with regard to the evalu- ation of ergonomic aspects in workstation planning. By integrating haptics into virtual prototypes, ergonomic issues can be identified and corrected in the early stages of assembly planning. Haptic enhanced virtual assembly sys- tems can also be applied to train operators for the launch of new products. First studies investigate the concept of integrating operator’s feedback to allow a continuous improvement of the workstation already in the training phase (Yildiz et al., 2019).
CONCLUSION AND OUTLOOK In this paper, a systematic literature review was conducted to investigate the current state and provide an overview of research in haptic enhanced virtual assembly systems. The investigation of virtual assembly systems encom- passed various aspects, including applications, industry sectors, research designs, data collection methods, haptic devices and the effectiveness of haptic feedback in virtual assembly systems.
The findings indicated that haptics facilitates user interaction in virtual assembly, particularly in enhancing immersion and the sense of presence.
Haptics in Virtual Assembly Systems: Insights Into the Current State of Research 617
These benefits are connected to a range of applications, e.g. assembly train- ing or assembly planning. However, the systematic literature review also revealed that many studies reported these effects in a qualitative manner, with a lack of research approaches that explore and quantify the effects. Quan- titative approaches that compare haptically supported and purely virtual systems are the exception or do not include all relevant assessment mea- sures for evaluating the human systems interaction. This review identified challenges in the design and development of haptic enhanced virtual assem- bly systems. Currently, no haptic device fully represents haptics in virtual environments since there are separated devices for kinaesthetic and tactile feedback. Consequently, when designing virtual assembly systems, it is impor- tant to specify the haptic requirements precisely and use-case specific to identify suitable haptic solutions. In conclusion, haptic enhanced assembly systems hold significant promise for the future. The integration of haptic elements can offer substantial benefits in various applications. Continued research has the potential to address current challenges and deliver an even more immersive and realistic experience for users.
REFERENCES
Abidi, M. H., Al-Ahmari, A. M., Ahmad, A., Darmoul, S. and Ameen, W. (2018) ‘Semi-Immersive Virtual Turbine Engine Simulation System’, International Journal of Turbo & Jet-Engines, vol. 35, no. 2, pp. 149–160. Al-Ahmari, A. M., Abidi, M. H., Ahmad, A. and Darmoul, S. (2016) ‘Development
of a virtual manufacturing assembly simulation system’, Advances in Mechanical Engineering, vol. 8, no. 3, 168781401663982. Azofeifa, J. D., Noguez, J., Ruiz, S., Molina-Espinosa, J. M., Magana, A. J. and Benes, B. (2022) ‘Systematic Review of Multimodal Human–Computer Interac- tion’, Informatics, vol. 9, no. 1, p. 13. Buchholz, C., Kind, S. and Stark, R. (2017) ‘Design of a Test Environment for Plan-
ning and Interaction with Virtual Production Processes’, Procedia CIRP, vol. 62, pp. 547–552. Flemisch, F., Bielecki, K., López Hernández, D., Meyer, R., Baier, R., Herzberger, N. D., & Wasser, J. (2020). Let’s get in touch again: Tangible AI and tangible XR for a more tangible, balanced human systems integration. International Conference on Intelligent Human Systems Integration (IHSI) 2020. https://doi.org/10.1007/978-3-030-39512-4_153 Gallegos-Nieto, E., Medellin-Castillo, H. I., Xiu-Tian, Y. and Corney, J. (2020) ‘Haptic-enabled virtual planning and assessment of product assembly’, Assembly Automation, vol. 40, no. 4, pp. 641–654. Gonzalez-Badillo, G., Medellin-Castillo, H., Lim, T., Ritchie, J. and Garbaya, S.
(2014) ‘The development of a physics and constraint-based haptic virtual assembly system’, Assembly Automation, vol. 34, no. 1, pp. 41–55. Grajewski, D., Górski, F., Hamrol, A. and Zawadzki, P. (2015) ‘Immersive and Haptic EducationalSimulations of Assembly Workplace Conditions’, Procedia Computer Science, vol. 75, no. 1, pp. 359–368. Grajewski, D., Górski, F., Zawadzki, P. and Hamrol, A. (2013) ‘Application of Virtual
Reality Techniques in Design of Ergonomic Manufacturing Workplaces’, Procedia Computer Science, vol. 25, pp. 289–301.
Grajewski, D. and Hamrol, A. (2020) ‘Low-cost VR system for interactive education of manual assembly procedure’, Interactive Learning Environments, vol. 104, no. 2, pp. 1–19. Jayaram, S., Connacher, H. I. and Lyons, K. W. (1997) ‘Virtual assembly using virtual
reality techniques’, Computer-Aided Design, vol. 29, no. 8, pp. 575–584. Kind, S., Geiger, A., Kießling, N., Schmitz, M. and Stark, R. (2020) ‘Haptic Interaction in Virtual Reality Environments for Manual Assembly Validation’, Procedia CIRP, vol. 91, no. 3, pp. 802–807. Liu, K., Yin, X., Fan, X. and He, Q. (2015) ‘Virtual assembly with physical
information: a review’, Assembly Automation, vol. 35, no. 3, pp. 206–220. Moher, D., Liberati, A., Tetzlaff, J. and Altman, D. G. (2009) ‘Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement’, PLoS medicine, vol. 6, no. 7, e1000097. Neges, M., Adwernat, S. and Abramovici, M. (2018) ‘Augmented Virtuality for
maintenance training simulation under various stress conditions’, Procedia Man- ufacturing, vol. 19, pp. 171–178. Preutenborbeck, M., Usai, M., Herzberger, N. and Flemisch, F. (2022) ‘Multimodal Human Systems Exploration with Tangible XR for the Internet of Production: An expert survey’, Production Management and Process Control, July 24-28, 2022, AHFE International. Radhakrishnan, U., Koumaditis, K. and Chinello, F. (2021) ‘A systematic review of
immersive virtual reality for industrial skills training’, Behaviour & Information Technology, vol. 40, no. 12, pp. 1310–1339. Read, A., Ritchie, J. and Lim, T. (2017) ‘Haptic Virtual Reality DFMA - A Case Study’, in Barbic, J., D’Cruz, M., Latoschik, M. E., Slater, M. and Bourdot, P. (eds) Virtual Reality and Augmented Reality, Cham, Springer International Publishing, pp. 24–38. Rückert, P., Wohlfromm, L. and Tracht, K. (2018) ‘Implementation of virtual reality
systems for simulation of human-robot collaboration’, Procedia Manufacturing, vol. 19, pp. 164–170. Seifi, H., Fazlollahi, F., Oppermann, M., Sastrillo, J. A., Ip, J., Agrawal, A., Park, G., Kuchenbecker, K. J. and MacLean, K. E. (2019) ‘Haptipedia’, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Glasgow Scotland Uk, 04 05 2019 09 05 2019. New York, NY, USA, ACM, pp. 1–12. Seth, A., Vance, J. M. and Oliver, J. H. (2011) ‘Virtual reality for assembly methods
prototyping: a review’, Virtual Reality, vol. 15, no. 1, pp. 5–20. Wolf, B., Kind, S. and Stark, R. (2020) ‘Smart Hybrid Prototyping in manual automotive assembly validation’, Procedia CIRP, vol. 88, pp. 82–87. Wu, H., Shi, X., Zhao, P. and Gui, L. (2019) ‘Research on Application of Vir-
tual Assembly Technology in Product Maintainability Design’, 2019 International Conference on Sensing, Diagnostics, Prognostics, and Control (SDPC). Beijing, China, 2019, IEEE, pp. 973–977. Xia, P., Lopes, A. M. and Restivo, M. T. (2013) ‘A review of virtual reality and haptics for product assembly (part 1): rigid parts’, Assembly Automation, vol. 33, no. 1, pp. 68–77. Yildiz, E., Melo, M., Moller, C. and Bessa, M. (2019) ‘Designing Collaborative and
Coordinated Virtual Reality Training Integrated with Virtual and Physical Fac- tories’, 2019 International Conference on Graphics and Interaction (ICGI). Faro, Portugal, 2019, IEEE, pp. 48–55.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-95-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional