Stefan Pfeffer, Marc Roessler, Simone Maag, Lisa Langwaldt, Lara Schunggart, Markus Strigel, Abigail Senger, Moritz Ochtrop (2024). Virtual Ergonomics - Ergotyping in virtual environments. In: Human Interaction and Emerging Technologies (IHIET 2024). AHFE Open Access, vol. 157, pp. 306–315. https://doi.org/10.54941/ahfe1005490
Ergonomic assessment of manual work processes is important to prevent workplace injuries. Virtual reality simulations can be used to carry out an evaluation of work equipment and workplaces very early on. In combination with motion tracking analyses, data on posture during task performance and product use can then be collected. However, not all work situations can be equally represented in a virtual simulation. In particular, the virtual analysis of load handling poses a challenge in simulation, as body posture changes under the influence of external load weights. The aim is to increase immersion to bring the body movements in the virtual simulation closer to those in the real simulation with weights.For building up VR simulations with different aspects of visual, auditory and haptic immersion a scheme called immersion cube is presented. In order to be able to simulate load handling in VR, the immersion cube is used to investigate how much haptic immersion is needed to obtain sufficiently good data for the body movements measured in a VR setting. The first study showed that the deviation between real and virtual executions depends heavily on the task (lifting from the ground, move while standing, lifting over the shoulder). In some tasks, virtual and real simulation are very close to one another for certain body movements and could therefore in principle be used for ergonomic assessment. On the other hand there are still movements that vary between these two forms of execution and therefore show a need for increasing the immersion.
Musculoskeletal disorders (MSDs) resulting from inappropriate ergonomic design of workplaces represent a main factor for days of incapacity for work and therefore loss of productivity of companies and economic burden of health systems all over the world (Bevan, 2015; Briggs et al., 2018). In particular, the ageing society and the need to continue working in old age make it necessary to pay even more attention to ergonomic workplace design in order to ensure the ability to work and to be able to employ older people and workers with reduced working capacity, taking into account their individual capabilities and limitations. This view goes hand in hand with the WHO’s guiding principles for the decade of healthy aging.
Virtual Ergonomics - Ergotyping in Virtual Environments 307
In Germany, most of the illnesses caused by mechanical impacts at work (reported suspected cases) can be attributed to “load handling” - with an increasing trend in numbers year by year (BAuA, 2023). For this reason, the German regulation on the implementation of EU directives in occupational health and safety, the so-called load handling regulation - also points out that physical suitability of the employees must be taken into account.
Load handling is defined as the handling of loads equal or greater than 3 kg (Serafin et al., 2020). Furthermore a distinction is made between the following forms of load handling:
Lifting, lowering or transferring is the movement of a load from one position to a lower, same or higher position by muscular force,
Holding is the fixing of a load in a certain position by muscular force
as a predominantly static process,
Carrying is the predominantly horizontal transportation of a load that does not reach the ground using human strength and by carrying it on the body.
In order to design a workplace ergonomically with regard to load handling, the nature of the load to be carried (weight, shape, size, etc.) and the working environment (space requirements, floor conditions, lighting, etc.) must be taken into account, as well as the employee’s work task. This includes, for example, body postures and movements and the relation of the load object to the person. Unfavorable stresses resulting from body postures and movements under load caused by the workplace design include, for example, all work requiring extreme trunk flexion, lifting with simultaneous trunk rotation and lateral flexion of the upper body or hollow back posture (trunk extension) when carrying loads.
VIRTUAL ERGONOMICS AND ERGOTYPING Virtual ergonomics refers to the use of computer-aided methods and tools for the ergonomic design of products, work systems and processes (Bullinger- Hoffmann & Mühlstedt, 2010). The term “ergotyping” can be traced back to Kamusella and Schmauder (2010) and, as an artificial word made up of the terms ergonomics and prototyping, describes a discipline in which ergonomic assessments can be implemented as early as possible in product or process development by using methods and digital tools to analyze, evaluate and design ergonomic aspects. The earlier an assessment of ergonomic conditions can take place, the more cost-effective and faster product and process development will be.
As a typical tool of virtual ergonomics, digital human models are used to create prospective simulations of a work system. For example, with the digital human model of the ema Work Designer (emaWD) software, scenes and behavior modeled in 3D can be assessed analytically with regard to ergonomic criteria (see Figure 1, right). In this context, analytical means that calculations, for example of posture, are not based on directly empirical collected data. In contrast, real simulations of work processes can be set up prototypically with simple materials (cardboard engineering). Here, for
example, body postures can be objectively determined using motion tracking systems. The use of motion tracking systems for the evaluation of workplaces has already been elaborated in several studies (e.g. Rybnikár et al., 2023; Caputo et al., 2018). The advantage of this method is that inter-individual differences in execution can also be taken into account. Without wanting to replace these tools, but rather as a supplement, virtual-empirical simulation is positioned here as a method between these two poles (see Fig. 1, middle). Here, workplaces are modeled virtually and empirical data is obtained through the use of virtual reality (VR) technologies in conjunction with motion tracking. The combination of VR and motion tracking has also been used in studies, although these were predominantly scenarios without load handling (e.g. Kaˇcerová et al., 2022; Simonetto et al., 2022). Virtual- empirical simulation offers the following advantages in addition to the other two methods:
individual movements of the workers can be taken into account, whereby the virtual workplace conditions can be adapted directly,
there is no risk from a real load, as the load objects are only handled virtually. This means that people with reduced working capacity and older people can carry out the simulation without injury,
the virtual environment can be scaled so that a person can experience the simulated workplace in different anthropometric body sizes - both as a very small person with challenges in accessibility and limited upward reach and as a very tall person with the challenges of space restrictions and limited downward reach.
Further advantages of the virtual-empirical approach are also mentioned by Kaˇcerová et al. (2022). Compared to real simulation, considerable amounts of material can be saved, simulations of workplaces in potentially hazardous environments are possible and many alternative solutions can be tested in a very short time.
Figure 1: Possibilities for early simulation and ergonomic testing of workstations.
For workplaces where movements without loads ≥3kg take place, body postures and movements, reachability, posture and dimensional analyses can already be assessed using the virtual-empirical method. However, as soon as movements with an external load ≥3kg take place, such as when
Virtual Ergonomics - Ergotyping in Virtual Environments 309
lifting, holding and carrying objects, the posture changes in comparison to load-free movements. Adaptive changes in posture take place in order to compensate for the additional load and ensure the stability and efficiency of the movement. The posture data collected in virtual-empirical studies then could lose validity and the assessment of ergonomic conditions could lead to misjudgments. Kaˇcerová et al. (2022) already found differences between real and virtual simulations with regard to the duration of postures taken.
In order to be able to use the virtual-empirical method for the early assessment of load handling, the aim must be to make the body movements in the virtual simulation as real as possible and to discuss a compromise between data quality, technical effort and benefits for early assessment. In order to investigate this compromise, a scheme called “immersion cube” was developed to systematically increase physical immersion in subsequent studies in the hope of increasing mental immersion and the experience of presence to such an extent that sufficiently good posture data can be generated. Weight perception is particularly important for the simulation of load handling in VR. There are already concepts for this from research. Lim et al. (2021) provide a comprehensive overview.
IMMERSION CUBE Since the appearance of the first standalone head-mounted displays, VR technology has become broadly attractive for use in industry and research. It is known from VR training that an increase in immersion and the associated more realistic simulation of a situation or process can lead to better learning outcomes (e.g. Månsson, 2018). The better the immersion and presence experience, the more valid the results that can be drawn from it in simulations. It is known from studies that immersion can be increased through a multimodal presentation (Dinh et al., 1999; Gallace et al., 2012).
Figure 2 presents a trimodal scheme for visual, auditory and haptic perception. As an extension of Milgram and Kishino’s diagram (Milgram & Kishino, 1994), the immersion cube lists auditory and haptic immersion in addition to visual immersion and represents these three modalities in the form of a cube. Each modality is divided into levels in terms of the expected degree of immersion, with the levels representing the degree of virtuality or reality of content from 0=purely virtual to 3=real.
In the visual dimension, a distinction is made between virtual objects in virtual environments (V1), virtual objects in real environments (V2) and real objects in real environments (V3). In V2, a further distinction can be made between virtual objects in indirectly (V2.1) or directly perceived real environments (V2.2). Whereas V1 is called “Virtual Reality”by Milgram and Kishino (1994) and V2.2 is called “Augmented Reality”, there’s a new form of visual immersion coming from the technological possibilities of passthrough modes, where the real world is only indirectly perceived through a real-time video (V2.2). Another sub-form of level V2.2 results from open-built headsets such as the Quest Pro, where the virtual objects are not superimposed on the real environment, but the virtual and real environments are experienced simultaneously adjacent to each other.
The auditory dimension would be possible without sound (A0), with virtual object sound and virtual surrounding sound (A1), real object sound and virtual surrounding sound (A2) as well as real object and surrounding sound (A3).
Both visual and auditory perception are distant senses. Haptics, on the other hand, is a near sense. In the real world, we perceive objects haptically in the area of exteroception through direct contact. These real haptics can also be made experienceable in virtual settings through the use and tracking of so-called haptic proxy elements. Following Lim et al. (2021), this form of haptic immersion is called “direct haptic” (H3). In addition, there are many studies and technical concepts for integrating haptic perception into VR via substitute cues. For the haptic submodality of force feedback, for example, this could be haptic gloves (Caeiro-Rodríguez et al., 2021). The force feedback of the gloves makes it possible for virtual objects that are gripped to be experienced indirectly haptically. Another form of indirect haptics would be the substitution of one specific submodality of haptic perception with another. An example of this is the replacement of proprioceptive perception (e.g. weight perception) with exteroceptive perception (e.g. vibrotactile presentation of information via a vest). Lim et al. (2021) call this “indirect haptic” (H2). Another way of transferring haptic information from the real world to the virtual world is to transmit the information via a perceptual channel other than the haptic one (H1). Substitution by the visual channel has been investigated many times. Here, for example, a pseudo-haptic effect is generated by changing the control-display ratio (Samad et al., 2019). Finally, level H0 describes a simulation in which no haptic information is transmitted. This is basically only possible using hand tracking, but the use of controllers (without haptic feedback) should also be included here.
In the combination of the three dimensions, simulations can now be described in terms of their immersion modalities. For example, a classic
Virtual Ergonomics - Ergotyping in Virtual Environments 311
VR simulation with hand tracking could be described as V1/H0/A1. With controllers that can reproduce vibrotactile haptic feedback, this would become a V1/H2/A1 simulation. The simulation described by Milgram and Kishino (1994) as “augmented virtuality” would be equivalent to e.g. a V1/H3/A1 and “augmented reality”, with headsets such as a Hololens 2 or Magic Leap 2, would be equivalent to a V2.2/H0/A1. The mixed reality made possible in many new headsets using passthrough mode and with hand tracking would be a V2.1/H0/A1.
EXPLORATORY STUDY A standard procedure was developed to investigate the different effects of haptic immersion options on body movement during load handling (see Figure 3). This includes the typical, problematic activities of lifting (1), in which the back extension is considered, setting down an object with flexed arms (2), transferring with back rotation (3), lifting above shoulder height with flexed arms (4) and setting down on the floor with back extension (5). In the first exploratory study, this procedure was carried out in the specified order 1–5 in four executions - in each case with three real weights (light, medium, heavy) - according to the Immersion Cube - in a V2.2/H0/A0 simulation. The three real executions were balanced for all test subjects in a test plan. The executions in VR always took place in the fourth position. The virtual scenario was set up in the ema Work Designer software (see Figure 3, above right) from imk Industrial Intelligence GmbH, Chemnitz, Germany. The VR headset used was the Oculus Quest Pro with controllers. This allowed the participants to see the virtual visual simulation in the central field of view and the real environment in the peripheral field of view. The controllers only served as a means of interaction for gripping the box and did not transmit any haptic feedback (H0). An acoustic simulation was not implemented (A0).
A differently filled bottle crate was selected as load object, which handles offers a typical gripping situation, in certain cases also a problematic gripping situation. An empty bottle crate with a weight of 1.7kg was selected as the light weight for both male and female participants. A distinction was made between men and women for the medium and heavy crates. The medium crate weighed 13.9 kg for men and 7.5 kg for women. The heavy crate weighed 19.1 kg for men and 10.4 kg for women. The reason for the differentiation is that both genders should be subjected to a similar load. The loads were estimated using the “Leitmerkmalmethode” (Serafin et al., 2020) and selected comparably for both genders. The heavy load condition was therefore selected so that it is in a rather low risk range with a moderately increased load. This ensured that the risk of injury to the participants is kept to a minimum, but that the heavy crate still requires effort to complete the task.
Body movements were measured using inertial sensors in all executions. The T-sens sensors and the CAPTIV software from TEA, France were used. In total 6 participants took part in the test. Of these, 3 participants were female and 3 participants were male. The average age of the participants was 23.8 years. The average arm length of the participants was 72.2 cm, with the
shortest arm length being 67.5 cm and the longest arm length being 79 cm. The average height of the participants was 176.5 cm. Both taller and shorter participants were represented, with the shortest participant being 168 cm tall and the tallest 186 cm. In order to limit individual forms of execution somewhat, the participants were instructed to observe the following: the box should be lifted from the floor with legs extended (step 1) and put down to the floor with legs extended (step 5). In step 3 the rotation should be made from the upper body and after each step the crate should be released and an upright posture assumed. Before the data collection took place in the virtual simulation, the participants had the opportunity to familiarize themselves with the VR situation and test the interaction with the controllers.
Figure 3: Standard procedure for the real (left) and virtual (right) simulation of load handling.
RESULTS
The results of the first study are presented below. The data was analyzed with regard to initial trends. Data analysis was ultimately possible for four of the six test participants, as the motion tracking data for two test participants could not be used for all executions due to test artefacts. Intra-individual comparisons were carried out for the remaining four participants. For this purpose, the measured movement data was first divided into categories commonly used for body posture analysis in ergonomics. As long as the body movement is in the “green”angle range, there is no risk potential. In the angle ranges “orange”, there is a possible risk and measures to eliminate the hazard are recommended. In the “red” range, there is a high risk and measures to eliminate the hazard are urgently required. Table 1 shows the angle ranges for the six movements evaluated in this study.
Virtual Ergonomics - Ergotyping in Virtual Environments 313
Table 1. Angle ranges for the six movements evaluated in this study.
Table available in the original PDF paper.
Movement Orange area Red area Lower back (rotation) −20
to −10
>20
10
to 20
<−20
Back (flexion/extension) 20
to 60
< 0
> 60
Shoulder right (flexion/extension) 20
to 60
< 0
> 60
Shoulder right (abduction/adduction) −20
to −60
< −60
> 0
Shoulder left (flexion/extension) 20
to 60
< 0
> 60
Shoulder left (abduction/adduction) −20
to −60
< −60
> 0
As an example, Table 2 shows the result of the comparison of the real executions for the three load cases with the virtual execution in step 3 “move box”. In this type of evaluation, it was only counted whether a test participant was in the orange and/or red area during the exercise. The numbers therefore represent the number of test participants who experienced a medium or higher level of strain during this activity. In terms of ergotyping, it would be these activities that could be considered for optimization.
Table 2. Number of test participants for the two areas orange and red in the four executions.
Table available in the original PDF paper.
Movement Light Medium Heavy VR Orange Red Orange Red Orange Red Orange Red
If Table 2 is viewed in terms of signal detection theory, the simulation in VR would produce the same result for the “move” (step 3) activity as the real execution for certain movements. Even for different load weights. The geometric conditions of the standardized workstation would therefore have the same effect on the considered body angles of e.g. the shoulder in the real and virtual situation. The rotation of the lower back on the other hand would be overestimated. The VR simulation would produce more false alarms regarding this movement. In our small study, it was even the case that the heavier the weight, the smaller the range of lower back rotation was. Lower back rotation in VR and with light weight in the real execution were wider and therefore more often found in the risky areas. The results for
flexion and extension of the back were again comparable, although one test subject only had movements in the red area and there was no orange area.
CONCLUSION
The results of this first small study give us hope that it is possible to assess load handling using motion tracking in a virtual simulation. In our case of the defined standard procedure, risky body movements occurred more frequently in the virtual simulation than in the real situations. The aim is not to measure body movements exactly. The method would be successful for early assessment in terms of ergotyping if it can be used to identify neuralgic points of a work system. Nevertheless, further studies are to follow in which the haptic immersion in the sense of the Immersion Cube is to be increased in order to bring the movements in the virtual simulation even closer to those in reality. The interesting trade-off here will be to be able to create a sufficiently accurate simulation with as little effort as possible. In terms of quality, when comparing the movements in reality with the movements in the virtual simulation, we were able to observe that a simulation like the one we carried out without haptic immersion would not have been sufficiently accurate for some activities. For example, the virtual crates were held overhead for longer in the transition from step 4 to 5 and even made a body turn in this position, whereas the real weights were brought directly close to the body at hip height when they left the table. This is also partly consistent with the findings of Kaˇcerová et al. (2022) who observed that subjects spent longer in shoulder flexions >60
in VR execution than in real execution.
For this reason, haptic immersion is to be successively increased in further studies. Initially in the sense of H1 by means of pseudo-haptic feedback as control display ratio. Subsequently, indirect haptic feedback will be used to investigate whether this can increase the awareness of a load and bring the movements in VR closer to reality. The plan is to use vibrotactile wristbands (H2) here. In a further step, a model for direct haptics will be developed (H3), which will act as a load proxy and enable adaptive weight adjustment.
In addition, deficiencies from the first study are to be rectified in the follow- up studies. Carrying loads over 5m will also be included as a further activity. Since the controllers caused artifacts in the motion tracking due to their condition, the following studies will be set up using hand tracking, which will also ensure a more realistic gripping situation. Initial attempts to work with haptic gloves were not promising at this point. A further test for the method will be to gradually dissolve the standard procedure and move on to free forms of execution in order not to exclude individual differences in execution.
With regard to the equipment used so far, we can say that we have had good experiences with the Quest Pro stand-alone headset for this type of study. Due to the design of the headset, the test participants were able to see the real environment and their bodies in the peripheral field of view, which meant that the movements were more realistic than in the first test runs of our study with a completely closed headset (Quest 3).
REFERENCES
Bevan, S., (2015). Economic impact of musculoskeletal disorders (MSDs) on work in Europe. Best Practice & Research Clinical Rheumatology. 29.
Virtual Ergonomics - Ergotyping in Virtual Environments 315
Briggs, A. M., Woolf, A. D., Dreinhöfer, K., Homb, N., Hoy, D. G., Kopansky- Giles, D., Åkesson, K. and March, L., 2018. Reducing the global burden of musculoskeletal conditions. Bulletin of the World Health Organization, 96(5), pp. 366–368. Bullinger-Hoffmann, A. C. and Mühlstedt, J., 2017. Homo Sapiens Digitalis -
Virtuelle Ergonomie und digitale Menschmodelle. Berlin Heidelberg New York: Springer-Verlag. Bundesanstalt für Arbeitsschutz und Arbeitsmedizin, 2023. Sicherheit und Gesundheit bei der Arbeit - Berichtsjahr 2022. Unfallverhütungsbericht Arbeit. 1. Auflage. Dortmund: Bundesanstalt für Arbeitsschutz und Arbeitsmedizin. ISBN: 978-3-88261-761-0 Caeiro-Rodríguez, M., Otero-González, I., Mikic-Fonte, F. A. and Llamas-Nistal, M., 2021. A Systematic Review of Commercial Smart Gloves: Current Status and Applications. Sensors, 21(8), p. 2667. Caputo, F., Greco, A., D’Amato, E., Notaro, I. and Spada, S., 2018. On the use
of Virtual Reality for a human-centered workplace design. Procedia Structural Integrity, 8, pp. 297–308. Dinh, H. Q., Walker, N., Hodges, L. F., Song, C. and Kobayashi, A., 1999. Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments. In: Proceedings of IEEE Virtual Reality. pp. 222–228. Gallace, A., Ngo, M. K., Sulaitis, J. and Spence, C., 2012. Multisensory presence in
virtual reality: Possibilities & limitations. In: Multiple sensorial media advances and applications: New developments in MulSeMedia. IGI Global, pp. 1–38. Kaˇcerová, I., Kubr, J., Hoˇrejší, P. and Kleinová, J., 2022. Ergonomic Design of a Workplace Using Virtual Reality and a Motion Capture Suit. Applied Sciences, 12(4), p. 2150. Kamusella, C. und Schmauder, M., 2010. Ergotyping im rechnerunterstützten
Entwicklungs- und Gestaltungsprozess. München: GRIN Verlag. Lim, W. N., Yap, K. M., Lee, Y., Wee, C. and Yen, C., 2021. A Systematic Review of Weight Perception in Virtual Reality: Challenges and Road Ahead. IEEE Access Månsson J., 2018. Using a Virtual Fire Extinguisher as a Tool for Safety Training.
Ph. D. Master Thesis. Lund: Faculty of Engineering, Lund University. Milgram, P. and Kishino, F., 1994. A taxonomy of mixed reality visual displays.
IEICE Transactions on Information and Systems, 77(12), pp. 1321–1329. Rybnikár, F., Kaˇcerová, I., Hoˇrejší, P. and Šimon, M., 2023. Ergonomics Evaluation Using Motion Capture Technology—Literature Review. Applied Sciences, 13(1), p. 162. Samad, M., Gatti, E., Hermes, A., Benko, H. and Parise, C., 2019. Pseudo-Haptic
Weight: Changing the Perceived Weight of Virtual Objects By Manipulating Control-Display Ratio. In: Proceedings of the CHI Conference on Human Factors in Computing Systems. Glasgow, Scotland, 4–9 May 2019. New York: ACM, p. 13. Serafin, P., Klußmann, A., Liebers, F., Schust, M., Brandstädt, F., Hartmann, B., and Gebhardt, H, 2020. Die neue Leitmerkmalmethode Manuelles Heben, Halten und Tragen von Lasten (LMM-HHT): Darstellung der Methode und Ergebnisse der Methodentestung, GfA Frühjahrskongress, Berlin. Simonetto, M., Arena, S. and Peron, M., 2022. A methodological framework
to integrate motion capture system and virtual reality for assembly system 4.0 workplace design. Safety Science, 146.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Interaction and Emerging Technologies (IHIET 2024)
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-33-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional