Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Cognitive-oriented Process Information Presentation for Augmented Reality Assisted Product Assembly

1 School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, Hubei 430074, China ↗
2 School of Mechanical and Electrical Engineering, Hainan University, Haikou, Hainan 570228, China ↗
Publication year
Digital Object Identifier10.54941/ahfe1004862

Abstract

The information assistance during manual assembly process of mass customization production is indispensable in order to assure the assembly quality. The complexity of searching information and difficulty of understanding information brings about cognition burdens of workers under the varied and lengthy assembly process, where augmented reality (AR) can be an effective instruction tool. The cognitive issues of AR based information presentation for assembly guidance were discussed in this paper. The work about complexity of manual assembly process was analyzed both on error prediction and AR based cognitive perspective of visual assets. A four levels hierarchical task analysis was conducted for assembly process with three kinds of assembly stages, i.e. selection, installation and inspection. Cognitive reliability and error analysis method was taken to identify the possible human assembly errors during three stages considering the human cognitive activity and function. In order to provide right information at right place and right time for decreasing the assembly errors, the often used visual assets, i.e. 3D model, 2D annotation, text and video, were discussed for conveying assembly instruction intention in AR scenario. From the authoring complexity and potential cognitive efficiency aspects, different information presentation styles for three assembly stages were described.

INTRODUCTION

Assembly is an important stage of the entire product lifecycle, which is the process that two or more parts are joined together in order to realize the function. Assembly activities cover about 30–40% of product development time, up to 50% of manufacturing cost and average 40% of product cost (Krugh et al., 2016). It can be said that the assembly determines the product and quality of the product. Although automated assembly meets the require- ments of mass production, there are still a large number of assembly activities of complex variant products that cannot be automated and require manual operation. The increasing operation complexity and the lengthy assembly process can cause human errors and influence the overall performance of the production, lead to decreasing of the assembly quality. In order to reduce the

© 2024. Published by AHFE Open Access. All rights reserved. 41

human errors caused by cognitive load during the assembly process of com- plex products, assembly information assistance system was indispensable and helpful (Hinrichsen et al., 2019).

Generally, assembly information assistance systems had shown the abil- ity to reduce human errors and increase the productivity. Assembly manual was the most commonly used information assistant tool for delivering assem- bly instruction to operators (Kolbeinsson et al., 2023). The development of assembly manuals have gone through four stages and have four different types, i.e. paper manual, electrical file manual on computer screen, 3D PDF manual and AR manual (Wang et al., 2022). The complexity of searching information and difficulty of understanding information brought cognition burdens of workers, which decreased the work efficiency and result in the quality problem. In the traditional paper-based or electrical manual, text or drawings were usually shown for work instructions, where a high cognitive load was needed to understand the operation process (Werrlich et al., 2019). AR can superimposes the information in front of the users’ view to facilitate their works. AR-based assembly assistant systems provide step by step visual instructions, which focus on how to identify tools, parts and materials, where to find them, and how to use them for assembly operation. AR allows various types of visual assets shown in front of operators for more clearly conveying the assembly process. Many studies have shown that the AR assembly man- ual can reduce human error rates and assembly times in performing assembly tasks compared to other instructions types (Wang, et al., 2016; Giridhar et al., 2020; Lavric et al., 2021). Although there were various visual assets used for AR visualization (Li et al., 2019), there was limited research on the impact of different visual information on cognitive processes and error prevention. The cognitive issues of AR based information presentation were studied in this paper. By analyz- ing the assembly process complexity, the cognitive process was discussed for the assembly stages based on hierarchical task analysis. The Cognitive Reli- ability and Error Analysis Method (CREAM) was used to classify human assembly error and discuss who, when, where, what, how visual information was helpful to prevent these error in manual assembly.

PROCESS COMPLEXITY FACTORS AND ASSEMBLY TASK ANALYSIS Assembly is a process involving many factors, such as human, machine, mate- rial, rule and environment. Studies had shown that AR had a positive effect on complex assembly processes in terms of assembly time, error rate, and other aspects (Bernardo et al., 2022). However, there was still no complete answer on how to measure the complexity of assembly process under the AR environment. The impact factors of assembly complexity were numerous, and the evaluation objectives vary. The existing complexity research mainly focused on the convenience of operations. Kiyokawa et al. (2023) gave a comprehensive summary of assembly difficultly and complexity from three aspects, i.e. system, product/object, task. The related design, operation and sequence complexity was classified and many factors were physical related characteristics (Richardson et al., 2006; Samy et al., 2010), e.g. the number

Cognitive-Oriented Process Information Presentation 43

of parts, shape/symmetry, number and position of fasteners. The aim of these work on complexity was generally used to predict product assembly defect caused by human or human-robot task allocation.

Many research had focused on the human performance assisted by differ- ent kinds of visual assets (Michele et al., 2022). Few studies had analyzed complexity from the AR cognitive perspective of visual assets. Radkowski et al. (2015) presented that the complexity of visual features and assembly tasks might influence the effectiveness of AR assistance. They divided the degree of difficulty into low and high level and different visual assets, e.g. text on screen, static 3D models, one or multiple 3D arrows, animated 3D models, 2D sketch, were used to form a metric. But they did not explain what was the degree of task difficulty under AR scenario. In the experiments of Deshpande et al. (2018), average time to install, different arrangement of parts, multi transformation of parts and multiple patterns in the structure were more important to assess the factor of installation complexity when AR information was used. During part selection phase, the definition of low and high complexity task was that whether there was more interference parts with a similar shape (Yang et al., 2019). Petrone et al. (2019) used a 3D puzzle ball to classify the complexity by color-coding the pieces that were identical so that the users could easily distinguish the pieces. Three degrees of complexity, i.e. easy, medium and difficult level, were evaluated from entropy dimension, which was calculated based on the total number of parts and different parts quantity in product (Bendzioch et al., 2019).

Apparently, the complexity of assembly process was tightly related with the assembly task (Fast-Berglund et al., 2013; Mattsson et al., 2020). In order to determine the general tasks required in the assembly process, hierarchical task analysis (HTA) was used for task description. Assembly tasks can be described as discrete operations among parts and devices (physical elements such as tools). A task is typically represented as a series of consecutive steps. In general, each step has three iteration activities, i.e. selection, installation and inspection, with total seven main operations (see Figure 1). The identified operations in HTA will be mapped to the cognitive behavior for AR assistant assembly instruction. The three stages/activities were explained as follows.

  • Selection activity: The operator firstly identifies the part/tool (PT) to be assembled/used at material preparation area, and then he will move his hands and pick/grasp the PT in order to proceed the next installation activity.
  • Installation activity: The operator finds the location of PT in the assem- bly, moves part to its place. During this phase, the PT will be orientated, adjusted, aligned and inserted to its final location. After the PT is placed at it’s correct location, the connection operation will take place. The fastening, seal, gluing or rivet operation will be done to join the parts together.
  • Inspection activity: The state of a connection will be checked, such as miss- ing parts and incorrect position. Some measurements will also be applied for required assembly quality, such as the distance or depth of parallelism. At last, the assembly data will be recorded to information system.
Figure 1: Hierarchical task analysis of human assembly process.
Original PDF Figure 1:

COGNITIVE RELIABILITY AND ERROR ANALYSIS FOR MANUAL ASSEMBLY Many human assembly errors had been identified in literature (Yaniel et al., 2021). But deeply analysis was lacked from correlation among assembly task, assembly errors, cognitive activity, cognitive function and required infor- mation aspect. The Human Reliability Analysis (HRA) method had been developed to quantify the probability of human error when performing speci- fied tasks (Pasquale et al., 2018). Based on task analysis, the HRA techniques, e.g. CREAM, HEART (Human Error Assessment and Reduction Technique), SHERPA (Simulator for Human Error Probability Analysis), and THERP (Technique for Human Error Rate Prediction), provided support to assess the possibility of occurrence of human errors. HRA methods were initially proposed and mainly applied in safety critical industries. They were rarely used in industry assembly systems. Therefore a human cognitive behavior model should be formalized in order to describe human cognitive processes and errors during assembly process. In this paper, CREAM was used to ana- lyze the assembly cognition activities and the potential manual errors based on the HTA in Figure 1.

According to CREAM, cognitive process is the controlled use of exist- ing abilities (skills, procedures, and knowledge) and resources. Each typical cognitive activity can be described by a combination of the four cogni- tive functions it requires. The four cognitive function is Observation (O), Interpretation (I), Planning (P), Execution (E). Each function is associated with different cognitive activities. There are total 15 cognitive activities, i.e. Co-ordinate (CO), Communicate (CM), Compare (CP), Diagnose (DG), Evaluate (EV), Execute (EC), Identify (ID), Maintain (MT), Monitor (MN), Observe (OB), Plan (PL), Record (RD), Regulate (RG), Scan (SC), Verify (VF). The cognitive demands of human assembly activity and the potential errors is shown in Table 1.

For assembly selection activity, locating and then picking the current step part/tool is the main operation. The related cognitive activities are OB, SC, ID,

Cognitive-Oriented Process Information Presentation 45

PL and EC. The O, I and P, E are the cognitive function for locating and pick- ing respectively. The missed and incorrect selection are the potential human errors during this stage. For installation stage, placement (locating and plac- ing the part/tool on the right the installation position) and connection are executed, and different cognitive activities and functions are required, where the results errors are the incorrect assembly position/orientation/sequence/- force et al. At the last inspection stage, measurement and recording are carried out, where missed/incorrect insertions/reading always be the human assembly errors.

Table 1. Cognitive orientated assembly error analysis in different assembly acclivities.

Table available in the original PDF paper.

Process/result errors

Assembly activity

Assembly operation

Cognitive activity

Cognitive function OB, SC, ID O, I

  • Missed part/tool

Selection Locate part/tool

PL, EC P, E

  • Incorrect part/tool
  • Wrong locating Pick part/tool
  • Incorrect sequence

OB, SC, ID O, I

  • Incorrect quantity Installation Locate position O, P, E
  • Incorrect position
  • Wrong locating Place part/tool

CO, PL, EC, RG O, P, E

  • Missed connection
  • Incorrect orientation Connect part

CO, PL, EC, RG

  • Incorrect join sequence
  • Incorrect join force
  • Inadequate connection O, I, P, E
  • Missed inspection
  • Mismatched connection Inspection Measure state

EV, EC, VF

  • Non standard check EC, RE I, E
  • Missed recording
  • Incorrect reading Record data
  • Incorrect recording

COGNITIVE-ORIENTED INFORMATION PRESENTATION FOR ASSEMBLY INSTRUCTION UNDER AR ENVIRONMENT In order to instruct or reminder the human operators, different visual assets, e.g. text, video, animation, 3D model, picture and sign, can be superposed in the view. From the cognitive aspect, each visual asset has differentiated characteristics and need different level mental resource to learn/understand. How to present these visual assets for every assembly operation is related to assembly quality and efficiency. Although there were some design rules or guidelines in the literature (Chimienti et al., 2010; Rolim et al.,2015), no agreement was reached for the best way of assembly visual asset presentation based on AR (Agati et al., 2020).

The most important factor of information presentation in AR is the cogni- tive requirement for the assembly operation in HTA. In the review paper of Michele et al. (2022), the auxiliary model, product model and text was com- mon used for task locating, assembly operating and checking respectively. In order to describe the assembly operation, what-how-why classification was proposed according to what was displayed, how it conveyed information and why it was used. Lavric et al. (2022) used what-where-how to present AR assembly assets, and the assembly operation was described briefly in “what”, the operation physical location was indicated in “where” and the way to perform was described in “how”.

In this paper as shown in Figure 2, we proposed a 4W1H (who, when, what, where, how) method of AR information presentation for cognitive assisted assembly. We used who to indicate the operator of the current action, when for assembly activities. What meant the visual asset type, where implied the position of the visual asset in augmented view, and how was the state of the displayed visual asset. In-situ positioning mean the visual asset overlaid the exact location of the assembly part in 3D world coordinate system (I3DR) or 2D screen coordinate system (I2DR). A predefined position in 3D world coordinate system (P3DR) or 2D screen coordinate system (P2DR) were also used for placement of visual assets. If the content, appearance, color, or posi- tion of visual assets remained unchanged in the view, the state of assets was static. For the visual asset type, we only discussed the four styles as follows.

Figure 2: Who, When, What, Where and How for AR assembly information presentation.
Original PDF Figure 2:
  • 3D model (3DM): The models have two classes, product or auxiliary model. It can be 3D virtual models of product/parts/connectors/tools/ma- chines and obtained from CAD design stage. In AR scenario, they are the digital and visual representation of real objects, which can be visualized in solid or wireframe mode. The model can also be auxiliary models, which are used for abstract presentation to delivering hints or implicit instruction to the operator. The 3D arrows, circles and positioning boxes are common auxiliary models used in AR. The 3D model can be displayed statically or dynamically in AR.
  • 2D annotation (2DA): In order to mark the objects in the AR, annota- tions (e.g. 2D positioning box, text bounding box) are used for attention attraction. They are usually registered in 2D screen coordinate system.

Cognitive-Oriented Process Information Presentation 47

  • Text: Text is the common way to convey process information and is easy authored in AR. But the long text is hard to read in AR because of the limited view. Short text is the suitable choice for assembly instruction in AR.
  • Video: Some complex operations can not be clearly described using static or animated model. It is also hard to understand if text description is used. During AR assembly training, such complex operation can be demon- strated by videos, which are transcribed based on the real operation of experience workers.

In order to enhance cognition, improve efficiency, and avoid errors, we designed divers information presentation for AR assisted assembly consider- ing the characteristic of different assembly activities/operations (see Table 2). First, the sequence and quantity errors in Table 1 were desired to be elimi- nated based the step-by-step action decomposition for each parts/connectors. Second, the missed operation in selection, installation and inspection stage would be avoided attribute to the exactly positioning of the augmented visual assets for clearly understanding. Third, by integrating assembly state check- ing function with feedbacking function for AR visualization, the incorrect position/orientation and missed components would be displayed to warn the operators (Zhao et al., 2023). Lastly, more digital measurement tools were used in shop floor, the measurement reading was recorded automatically and human reading error would be eliminated. Of course, different visualiza- tion styles had various amount of virtual assets, which had varied authoring complexity. Under some complex situation, one style was not enough for clearly describing the assembly operation and two or three presentation styles (e.g. 3DM+text) were simultaneous used for showing the assembly intention. From the aspect of cognitive efficiency, 3DM and video style were more easy understood comparing with text and annotation. Figure 3 shows several AR assembly information presentation examples of our past works. The different visual assets (what), registration methods (where) and states (how) are shown for assembly activities (when). The placement and connection are installation activities in Table 2.

Figure 3: Who, When, What, Where and How for AR assembly information presentation.
Original PDF Figure 3:
Table 2. Category for AR assisted assembly information presentation.

Table available in the original PDF paper.

Tasks (when) Content style (what, where, how)

Authoring complexity

Cognitive efficiency

High

Selection 3DM of positioning box, I3DR, static or dynamic Medium High 3DM of objects, I3DR, static or dynamic Low High 3DM of arrows, I3DR, static or dynamic Medium Medium 2DA of bounding box, I2DR, static Low Medium Placement 3DM of objects, I3DR, static or dynamic Low (static) High (dynamic) High 3DM of arrows, I3DR, static or dynamic Medium Medium Connection 3DM of objects, I3DR/P3DR, dynamic High (I3DR) Medium(P3DR)

Low High

Numeric text, P2DR, static or dynamic Low Low Video, P2DR, dynamic Low High Inspection Video, P2DR, dynamic Low High 3DM of objects for assembly errors, I3DR, static or dynamic 2DA of box or text for assembly errors, I2DR, static Low Medium

CONCLUSION

Augmented reality based information presentation for assembly instruction demonstrated more and more benefits during manual operation. The relation among the assembly complexity, information presentation and cognition was still unclear. In this study, hierarchical task analysis was used to construct a four levels assembly process with three kinds of assembly activities. The human errors of these activities were analyzed based on CREAM. Several assembly information presentation styles in AR were described using 4W1H method. The assembly cognitive experiments will be conducted in the future to validate and improve the information design. The study can provide infor- mation support for personalized and differentiated requirement of operators in Industry 5.0.

ACKNOWLEDGMENT

The authors acknowledge the support from the National Natural Science Foundation of China (72271100).

REFERENCES

Agati, S., Bauer, R., Hounsell, M., Paterno, A. (2020). Augmented reality for manual assembly in industry 4.0: gathering guidelines. The 22nd Symposium on Virtual and Augmented Reality, 179–188. Bernardo, A. J., Bernardo, M., Carlos, F., Paulo, D., Sousa, S. B. (2022). Comparing

augmented reality visualization methods for assembly procedures. Virtual Reality, 26, 235–248. Bendzioch, S., Bläsing, D., Hinrichsen, S. (2019). Comparison of different assembly assistance systems under ergonomic and economic aspects. Proceedings of the 2nd International Conference on Human Systems Engineering and Design, 20–25. Chimienti, V., Iliano, S., Dassisti, M., Dini, G., Failli, F. (2010). Guidelines for

implementing augmented reality procedures in assisting assembly operations. IFIP Advances in Information and Communication Technology, 315, 174–179.

Cognitive-Oriented Process Information Presentation 49

Deshpande, A., Kim, I. (2018). The effects of augmented reality on improving spatial problem solving for object assembly. Advanced Engineering Informatics, 38, 760–775. Fast-Berglund, Å., Fässberg, T., Hellman, F., et al. (2013). Relations between com-

plexity, quality and cognitive automation in mixed-model assembly. Journal of manufacturing systems, 32(3), 449–455. Giridhar M., Lasin, Panicker, V. (2020). Experimental analysis of cognitive issues impacting manual assembly task. Proceedings of International Conference on System, Computation, Automation and Networking, 1–6. Hinrichsen, S., Bendzioch, S. (2019). How digital assistance systems improve work

productivity in assembly. Advances in Human Factors and Systems Interaction, Springer International Publishing, 332–342. Kiyokawa, T., Shirakura, N., Wang Z., et al. (2023). Difficulty and complexity definitions for assembly task allocation and assignment in human–robot collaborations: A review. Robotics and Computer-Integrated Manufacturing, 84, 102598. Kolbeinsson, A., Fogelberg, E., Thorvald, P. (2023). Information display preferences

for assembly instructions in 6 industrial settings. 14th International Conference on Applied Human Factors and Ergonomics. 73, 152–161. Krugh, M., Antani, K., Meras, L., Schulte, J. (2016). Prediction of defect propensity for the manual assembly of automotive electrical connectors. Procedia Manufac- turing, 5, 144–157. Lavric, T., Bricard, E., Preda, M., Zaharia, T. (2021). Exploring low-cost visual

assets for conveying assembly instructions in AR. International Conference on Innovations in Intelligent Systems and Applications, 1–6. Lavric, T., Bricard, E., Preda, M., Zaharia, T. (2022). A low-cost ar training system for manual assembly operations. Computer Science and Information Systems, 19(2), 1047–1073. Li, W., Wang J., Jiao, S., Wang, M., Li, S. (2019). Research on the visual elements

of augmented reality assembly processes. Virtual Reality & Intelligent Hardware, 1(6), 622–634. Michele, G., Evangelista, A., Uva, A. (2022). What, how, and why are visual assets used in industrial augmented reality? a systematic review and classification in maintenance, assembly, and training. IEEE Transactions on Visualization and Computer Graphics, 28: 2, 1443–1456. Mattsson, S., Fast-Berglund, Å., Li, D., Thorvald, P. (2020). Forming a cogni-

tive automation strategy for Operator 4.0 in complex assembly. Computers & Industrial Engineering, 139, 105360. Pasquale, Y., Miranda, S., Neumann, W., Setayesh, A. (2018). Human reliability in manual assembly systems: a systematic literature review. IFAC- Papersonline, 51(11), 675–680. Petrone, K., Hanna, R., Shankaran, G. (2019). A comparative examination of AR

and video in delivering assembly instructions. Proceedings of the 13th IMCL Conference, 445–456. Radkowski, R., Herrema, J., Oliver, J. (2015). Augmented reality based manual assembly support with visual features for different degree of difficulty. Interna- tional Journal of Human Computer Interaction, 31, 337–349. Richardson, M., Jones, G., Torrance, M., et al. (2006). Identifying the task variables

that predict object assembly difficulty. Human Factors, 48(3), 511–525. Rolim, C., Schmalstieg, D., Kalkofen, D., Teichrieb, V. (2015). Design guidelines for generating augmented reality instructions. IEEE International Symposium on Mixed and Augmented Reality, 120–123.

Samy, S., Elmaraghy, H. (2010). A model for measuring products assembly complexity. International Journal of Computer Integrated Manufacturing, 23(11), 1015–1027. Wang, J., Li, W., Fu, Y., Wu, L. (2022). Research progress on human factor adapt-

ability in augmented reality assisted assembly. Journal of Mechanical Engineering, 58(18), 16–30. Wang, X., Ong, S., Nee A. Y. C. (2016). A comprehensive survey of augmented reality assembly research. Advances in Manufacturing, 4, 1–22. Werrlich, S., Daniel, A., Ginger, A. et al. (2019). Comparing HMD based and paper-

based training. Proceedings of the 2018 IEEE international symposium on mixed and augmented reality, 134–142. Yang, Z., Shi, J., Jiang, W., et al. (2019). Influences of augmented reality assistance on performance and cognitive loads in different stages of assembly task. Frontiers in psychology, 10, 1703. Yaniel, T., Nadea, S., Landau, K. (2021). Classification and quantification of human

error in manufacturing: a case study in complex manual assembly. Applied Sciences, 11(2), 749. Zhao, S., Wang, J., Li, W., Lu, L. (2023). Online assembly inspection integrating lightweight hybrid neural network with positioning box matching, IEEE Access, 2023, 11, 139223–139235.

Cover of the proceedings volume Human Dynamics, Product Evaluation and Quality

Explore the complete digital volume

Human Dynamics, Product Evaluation and Quality

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-07-6

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →