Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Exploring Simultaneous Localization and Mapping(SLAM) Technology for Complex Equipment Maintenance with the Perspective of Human-Machine Collaboration

1 National Kaohsiung University of Science and Technology, Kaohsiung, KHH 82445, Taiwan ↗
2 Fu Jen Catholic University, New Taipei, NTPC 24205, Taiwan ↗
Publication year
Digital Object Identifier10.54941/ahfe1005060

Abstract

The rapid development of Augmented Reality (AR) and Mixed Reality (MR) technologies across various industries has intensified the need for advanced maintenance solutions in the context of the digital transformation of Industry 4.0. Taking the forming machine equipment in Taiwan's metal centre as an example. The declining birth rates and labor shortages makes it challenging for experts to be present on-site for timely repairs, leading to prolonged downtime and additional costs due to production line halts. This study aims to propose an MR-based maintenance guide, enabling on-site technicians to conduct repairs using digital technology even in the absence of equipment experts. By using head-mounted MR devices, technicians can instantly see system alerts and access 3D visualized step-by-step guidance, significantly improving repair speed and accuracy. This approach also reduces the need for expert travel, thus saving time and resources. More importantly, integrating Simultaneous Localization and Mapping (SLAM) technology ensures precise alignment between virtual and actual machine imagery, enhancing the accuracy of maintenance instructions and focusing on a human-centered experience. This technological innovation is not only applicable to forming machine equipment but can also be extended to other complex equipment maintenance fields requiring specialized knowledge. With the maturation of MR technology, this maintenance strategy is expected to become more prevalent in the future, promoting Industry 4.0's digital transformation in manufacturing.To achieve these goals, a systematic methodology was developed. Initially, the scope was defined, and the overall research process was planned during the preparation phase. The literature review phase involved an in-depth investigation into the development of head-mounted MR devices in various settings and the exploration of their application possibilities. Literature about remote collaboration was also reviewed to understand the technical requirements and limitations of this mode. Therefore, SLAM testing with Microsoft HoloLens2 in an indoor factory setting was conducted to evaluate its effectiveness in aligning virtual and physical machinery models. Finally, system development and testing were undertaken using Unity3D and Microsoft's Mixed Reality Toolkit (MRTK) to implement the MR device functionalities on-site. Overall, this methodology aims to comprehensively consider the application of digital technology in industrial maintenance to enhance efficiency and accuracy.This research has yielded several significant findings, including valuable perspectives on SLAM technology. It also sheds light on the performance capabilities of HoloLens 2 in industrial environments, particularly regarding system operation processes.

INTRODUCTION

Research Background In modern industrial environments, the correct and timely maintenance of forming machine equipment is key to ensuring smooth production processes.

© 2024. Published by AHFE Open Access. All rights reserved. 114

Exploring Simultaneous Localization and Mapping (SLAM) Technology 115

However, the context of declining birth rates and labor shortages often pre- vents experts from being on-site for timely repairs. This not only extends production line downtimes but also incurs additional cost expenditures. Therefore, there is an urgent need for a solution that can carry out repairs instantly and accurately, irrespective of the expert’s presence on-site.

With the advancement of Augmented Reality (AR) and Mixed Reality (MR) technologies, their potential applications in the manufacturing sec- tor have garnered significant interest among researchers. Max Juraschek, in 2018, implemented a transparent Mixed Reality Head-Mounted Display (HMD) in automotive manufacturing. This allowed designers to alter the appearance of vehicles through enhanced drawings, making the design pro- cess more collaborative and effective. In the aspect of machinery maintenance within the manufacturing process, the integration feature of Mixed Reality, which merges virtual and real-world elements, directly conveys the results of real-time data visualization. This aids less experienced engineers in quickly grasping on-site information and acting based on virtual image instructions. The direct and practical acquisition of 3D visualized maintenance guidance significantly enhances the efficiency and precision of maintenance work.

However, in practical applications, MR technology still faces a major chal- lenge: how to precisely align virtual machine components with the actual physical environment. This is crucial to ensuring that maintenance instruc- tions align perfectly with the real machines. Fortunately, through research, we found that Wang Shengkai (2020) in his study “Integration of Augmented Reality and Building Information Modeling (BIM) for Construction Stage Progress Control System,” successfully used Simultaneous Localization and Mapping (SLAM) technology to solve the alignment issues of AR devices in construction sites. This provides us with a research direction to explore how this technology can be applied in the field of forming machine equipment maintenance.

Based on the above research motivation, we aim to develop an MR-based maintenance guidance system that not only provides instant and accurate maintenance instructions but also ensures the precise positioning of virtual machines, thereby meeting the practical needs of industrial maintenance.

Research Motivation Traditional maintenance methods usually require experts to personally guide junior operators on-site. In the current context of human resource shortages, this approach leads to experts frequently traveling between different plants and possibly facing multiple operators’ guidance needs simultaneously. This not only significantly increases the consumption of time and costs but also brings considerable physical stress and burden to the experts themselves.

Considering these issues, we propose using MR technology to develop a virtual guidance expert system. This system can provide users with spe- cific operation and maintenance guidelines. By wearing the appropriate MR equipment, users can see virtual maintenance guidance in the real environ- ment, guiding them to complete various maintenance tasks. Thus, junior operators can receive clear operation guidelines even without the presence of

an expert, and experts are no longer required to frequently travel to various work sites.

However, the current MR technology still faces a significant challenge: how to ensure that virtual machine components are precisely positioned in the actual physical environment. Inaccurate alignment may lead users to follow incorrect maintenance instructions, thereby increasing repair time or even causing equipment damage. Therefore, we specifically consider this in our system, ensuring the accuracy of maintenance instructions, allowing virtual repair steps to align perfectly with the real machines.

Through this innovative approach, we can not only save a significant amount of time and costs but also ensure the quality and efficiency of main- tenance work, reduce the physical burden on experts, and improve overall production efficiency.

The main objectives of this study are as follows:

1. Integration and application of SLAM technology to ensure precise align- ment between virtual and physical machines: The aim is to address the main challenge of MR technology in the field of maintenance, i.e., how to accurately position virtual machine components in the actual physical environment, thereby ensuring the accuracy of maintenance instructions. 2. Evaluation of the impact of the MR-based maintenance guidance system on industrial maintenance efficiency and accuracy: Through experiments and actual case studies, evaluate the benefits of this system in real indus- trial environments, including savings in time and costs, improvements in the quality of maintenance work, and reduction of physical burden on experts, among other aspects.

Literature Review

This chapter introduces literature on machine maintenance guidance visu- alization and precision positioning technology, serving as a basis for under- standing the research development related to this study and discussing the direction of this research.

The core capability of Mixed Reality (MR) technology lies in its ability to integrate digital data with the physical world, enabling digital informa- tion to merge spatially with real machines, processes, production systems, and products. This technology not only enhances and visualizes informa- tion and states digitally but also provides actionable guidance. MR employs visual, auditory, and other natural user interface elements to offer learners novel instructional and learning pathways, thereby enriching the methods of imparting knowledge and skills.

Visualization of Machine Maintenance Guidance and Application of Mixed Reality In the early stages of machine maintenance processes, most guidance was provided to technicians in the form of 2D paper manuals or electronic doc- uments. While functional, this format often fails to provide sufficient visual information for complex repair tasks, making it difficult for technicians to understand and execute maintenance steps quickly and accurately. With the

Exploring Simultaneous Localization and Mapping (SLAM) Technology 117

development of digitalization and information technology, machine main- tenance guidance has gradually shifted towards the use of 3D visualization tools. However, these tools are mostly presented on 2D screens, displaying 3D models, which still limits user interaction and realism. The emergence of Mixed Reality (MR) technology has brought revolution- ary changes to machine maintenance guidance. Through MR, technicians can see virtual 3D maintenance instructions within the real machine envi- ronment, which not only enhances the intuitiveness of the instructions but also significantly improves the efficiency and accuracy of maintenance work. For example, in 2018, “Wangjie Intelligent Perception” and “Yili Repair Technology” collaborated to provide semiconductor equipment maintenance services. They transformed bulky maintenance manuals into 3D dynamic presentations directly on HoloLens, allowing technicians to view 3D visu- alized dynamic instructions through glasses and communicate in real-time with remote experts (TIIP - Application of Mixed Reality in Semiconductor Machine Maintenance Services, 2018). In recent years, with the development of MR technology, such applications have become increasingly common.

Simultaneous Localization and Mapping Technology The complexity of machine room environments often precludes the installa- tion of instruments, making SLAM technology suitable for indoor machine room environments. This technology estimates device positions by captur- ing images, identifying landmarks, and recording sensor (IMU) values. By continuously capturing feature points and recording IMU data, the cam- era’s position relative to these points is estimated, and a feature point map is created, achieving marker-less device positioning.

Microsoft’s HoloLens 2, featuring multiple feature cameras and IMUs, embodies a built-in multi-lens SLAM positioning device. Feigl et al. (AR Machine Operator Hololens, n.d.) tested a variety of devices, including HoloLens, that utilize SLAM in an Industry 4.0 environment to compare their positioning capabilities.

RESEARCH METHODOLOGY Preparation Phase

This phase involved exploring the motivation and objectives of the study and defining the scope of research based on these objectives. Subsequently, a research process framework was planned.

Literature Review Phase This stage comprised investigating the development of head-mounted MR devices in various environments to understand their potential applications and expected benefits. During this process, potential problems and solu- tions were identified, and attempts were made to apply these solutions to our system.

A collection of domestic and international applications of ‘remote col- laboration’ was studied to understand the benefits of ‘remote collaboration

models’. The technical requirements and limitations of various models were examined as a reference for the system architecture integrating onsite and remote systems in our study.

The literature on SLAM positioning in indoor factory environments was analyzed to explore the background of SLAM technology. Its techniques were understood and learned to be applied in aligning indoor factory machine models and planning tests of head-mounted MR devices for feature point scanning effectiveness.

SLAM Testing With HoloLens 2 in Indoor Factory The study will test indoor positioning methods based on SLAM technology in appropriate environments. The tests and results will explore the impact of SLAM technology on the alignment outcomes of virtual machine models. Attempts will be made to organize operational suggestions for HoloLens 2’s SLAM, including scanning methods and speed. User assistance recommen- dations will be based on the results of SLAM environmental feature point mapping.

System Development and Testing Phase The architecture of both onsite and remote systems was designed to explore how to combine these two systems to achieve a ‘remote collaboration’ operation model. The common progress management workflows in construc- tion practice were understood to design the system’s schedule planning and visualize progress data applications.

System development utilized Unity3D as the development engine for the onsite head-mounted MR device and remote system’s progress manage- ment interface. Microsoft’s Mixed Reality Tool Kit (MRTK) was used for onsite system development, implementing environmental feature point scan- ning and plane detection functions using SLAM technology. This achieved model alignment and device positioning effects. Virtual machine models were aligned with dynamic reference planes set up onsite based on the base planes placed in the preliminary work. Adjustments were made according to the measured error values to align the virtual machine models with the real environment’s structure. Virtual machine models and progress data were dynamically loaded into the universally designed onsite and remote systems through MQTT and Microsoft SQL Server relational database, achieving a lightweight operating system.

After system construction, each function was tested to ensure it operated as expected. Following the planning of cases close to real usage scenarios, the system’s operational workflow was reviewed to understand the capabilities of head-mounted MR devices in machine room settings and whether remote collaboration mode could achieve its purpose.

CONCLUSION AND RECOMMENDATIONS This study explores the outcomes of the research and proposes conclusions and future perspectives based on the findings. The conclusions serve as rec- ommendations and references for subsequent research. The research process and organization of chapters are illustrated in Figure 1.

Exploring Simultaneous Localization and Mapping (SLAM) Technology 119

Figure 1: The flowchart of this study.
Original PDF Figure 1:

Mixed Reality (MR) is identified as a powerful tool for teaching and learn- ing, capable of enhancing task efficiency by adapting to the user’s level of experience. In the context of equipment maintenance, an example of an aug- mented reality environment tailored to data availability and user skill levels can be found. The aim of this paper is to systematically analyze the potential applications of MR in factories by providing a hardware-agnostic overview, and to identify potential beneficial applications and use cases in the guidance of production engineering and learning environments.

REFERENCES

AR Machine Operator Hololens / AR instruction IoT, [Online], Available: https: //www.pdsol.com/product/vuforia-studio/. HoloLens 2 hardware | Microsoft Docs., [Online], Available: https://docs.microsoft

.com/en-us/hololens/hololens2-hardware. In Eight Months. Forbes.Retrieved from https://www.forbes.com/sites/paultassi/2022 /10/17/metas-horizon-worlds-has-somehow-lost-100000-players-in-eight-months. Jitesh Ubrani, Ramon Llamas, Michael Shirer. (2023). Global Ship- ments of AR/VR Headsets Decline Sharply in 2022 Following the Prior Year’s Strong Results, According to IDC. IDC. Retrieved from https://www.idc.com/getdoc.jsp?containerId=prUS50467723. Liu S., Xie J., Wang X., Meng H., (2023). Mixed Reality collaboration environment

improves the efficiency of human-centered industrial system: A case study in the mining industry. Autom. Constr, vol. 180, doi:10.1016/j.cie.2023. META Q3 2022 Earnings Call Transcript. (2022). Retrieved from https://s21.q4cdn.com/399680738/files/doc_financials/2022/q3/Meta- Q3-2022-Earnings-Call-Transcript.pdf.

Meta Reports Fourth Quarter and Full Year 2022 Results. (2023). Retrieved from https://www.sec.gov/Archives/edgar/data/1326801/000132680123000008/meta- 12312022xexhibit991.htm. Milgram P., Kishino F. (1994). A Taxonomy of Mixed Reality Visual Displays. IEICE

Transactions on Information and Systems, vol. E77–D. Mixed Reality Examples: 5 Uses in the Workplace, [Online], Available: https://virtua lspeech.com/blog/mixed-reality-change-future-workplaces. Patrick, Hübner., Kate, Clintworth., Qingyi, Liu., Martin, Weinmann., S., Wursthorn. (2020). Evaluation of HoloLens Tracking and Depth Sensing for Indoor Mapping Applications. Sensors, 20(4):1021. doi: 10.3390/S20041021. Paul Tassi (2022). Meta’s ‘Horizon Worlds’ Has Somehow Lost 100, 000 Players.

Cover of the proceedings volume Human Factors and Wearable Technologies

Explore the complete digital volume

Human Factors and Wearable Technologies

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-17-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →