Mohammad-ehsan Matour, Christian Thormann, Alexander Winkler (2024). Innovative Methods for Robot Programming: Development and Comparison. In: Human Factors and Systems Interaction. AHFE Open Access, vol. 154, pp. 208–217. https://doi.org/10.54941/ahfe1005368
In considerations of Human-Machine Interaction or Human-Robot Interaction, humans are often regarded as users or employees. However, the programming of the robot also plays a crucial role, as it is a form of Human-Machine Interaction. According to the International Federation of Robotics, over 500,000 industrial robots were newly installed worldwide in 2022. It is predicted that annual installations will continue to rise in the coming years. This necessitates providing access to robot programming for less specialized personnel. In addition to traditional programming methods, there are already some more intuitive concepts in this field. This includes, for example, guiding the robot arm by hand as an alternative to control through buttons or a joystick. Furthermore, classical textual programming is often replaced by a form of Drag & Drop programming.In addition to these commercially available programming variants, novel programming concepts are conceivable, such as the integration of voice control, the use of mobile devices such as phones or tablets, robot control via gestures, or the utilization of products from the field of video games. Moreover, Augmented Reality technology exists, which can support intuitive robot programming.In this article, we present our own developments on the topic of innovative methods of robot programming. Firstly, an innovative approach to robot programming using gesture control via the Kinect camera is introduced. This method allows the operator to precisely control the robot through multiple waypoints and save them using virtual elements projected by a video-projector. Subsequently, the robot can efficiently traverse the saved points, demonstrating significantly faster programming compared to conventional methods in experiments.Furthermore, we present an innovative Mixed Reality approach. By using a Head-Mounted Display, the operator can control the robot in a particularly advanced manner. This includes not only intuitive path planning in both joint and Cartesian space but also the realization of force-controlled movements and the provision of crucial status information from the robot. This holistic approach creates a novel interface between humans and robots, significantly improving efficiency and control in robotics.Next, a joystick-based robot movement with haptic feedback is introduced. This control allows the operator to precisely and intuitively control the robot using a joystick. The haptic feedback enhances the user experience by providing tactile feedback to better understand the robot's movements.The goal of all these methods is to simplify programming, even without in-depth expertise. We compare these methods with each other and with established programming procedures in terms of time efficiency, ergonomics, programming comfort, etc. Additionally, we analyze whether the considered procedures are suitable for individuals without specific expertise.
In the contemporary landscape of manufacturing technologies, the role of humans has become increasingly central, prompting a profound impact on the exploration of innovative manufacturing methodologies. It becomes imperative for human factors, including psychological, cognitive, and social influences, to be accounted for within the realm of human-machine systems. The overarching goal is to empower operators to execute tasks efficiently within a conducive environment, thereby contributing positively to value cre- ation. Central to this endeavor is the strategic minimization of operator errors achieved through a thoughtfully designed interface.
Conventional programming methods, typified by textual programming, teach-in approaches or programming by demonstration, pose challenges in terms of time investment, especially for intricate tasks. This underscores the critical necessity for ergonomic, efficient, and precise workflows that allow users to focus on tasks rather than grappling with system intricacies.
Innovative Methods for Robot Programming: Development and Comparison 209
From the vantage point of Human Factors and Ergonomics, the quest for optimal robot programming unfolds through innovative paradigms that transcend traditional approaches. This paper explores methodologies, incor- porating gesture control, joystick-based control with haptic feedback, and mixed reality (MR) through Head-Mounted Displays (HMDs). Gesture control introduces a fresh programming approach emphasizing ergonomic interaction between the operator and the robot, reducing cognitive load and enhancing intuitive programming experiences.
Controllers featuring haptic feedback emerge as instrumental components in the realm of telerobotic – the remote control of robots from a distance. The inherent capabilities of these controllers, position them ideally for such appli- cations. Notably, the incorporation of haptic feedback sets these controllers apart from conventional systems, adding substantial value where typical industrial control interfaces fall short. Applications of teleoperators span sce- narios involving potential risks to human safety, such as handling hazardous materials, working in contaminated environments, or operating in disaster- stricken areas. Controllers with haptic feedback aligns with human-centered principles, providing tactile cues that deepen the operator’s understanding of the robot’s movements.
The integration of MR technology further amplifies the human-centric approach to programming, offering real-time interaction through HMDs. This fusion seamlessly blends virtual elements with the physical environ- ment, enhancing the operator’s control over the robot and underscoring the importance of digital simulation in a user-friendly manner. The collaborative nature of cobots, particularly lightweight robots with safety-specific func- tions, aligns with the principles of Human Factors and Ergonomics, enabling hand-guided and drag-and-drop programming styles.
This paper introduces not only a comprehensive toolkit for controlling and path planning of cobots but also underscores the pivotal role of ergonomics in the programming interface. Through a comparative analysis of diverse programming approaches, the research aims to elucidate their contributions to time efficiency, ergonomic considerations, and overall user experience, advancing the discourse on Human-Machine Interaction in the dynamic land- scape of industrial robotics. In doing so, it aligns with the principles of accessibility, efficiency, and user-centric design, marking a significant stride towards the future of manufacturing technologies.
RELATED WORKS In the field of teleoperation and haptic feedback, numerous works have explored different control methods. One category utilizes small handheld controllers, including joysticks, keyboards, computer mice, and touch screens (Boboc et al., 2013). Another category involves controllers that capture nat- ural body movements to control a robot, resulting in a more intuitive system. Motion capture systems, similar to the gesture control approach presented in this contribution, use cameras, body markers, and computer vision to detect user positions.
In (Herlant et al., 2016), an assistive robotic arm is introduced, controlled through a low-dimensional joystick. Recognizing the difficulty of control- ling a high-dimensional system through simple interfaces, the author has developed multiple control modes to achieve a fully interactive experience. In contrast, our approach employs a Sony DualSense game console for robot motion control, offering increased degrees of freedom.
Additionally, interactive gloves have been explored in robot teleopera- tion. Li et al. (2019) proposed a novel glove with force and haptic feed- back, enabling users to accurately differentiate between secure and insecure grasps. As the field of robotics continues to advance, novel interfaces and technologies are being integrated to enhance the overall user experience.
In recent years, the use of Augmented Reality (AR) in robotics has seen a significant surge, driven by its immense potential. Researchers and scientists have leveraged AR to develop innovative applications that not only stream- line the operation of robots but also introduce new levels of convenience and interaction.
Liu et al. (2018) proposes an AR-based approach to abstract the func- tionality, decision making process, and diagnosis of a robotic system while carrying out a task. By means of the temporal graph network, the user can intuitively comprehend and interpret the inner workings of the robot and easily diagnose any errors in performance. Moreover, by parsing the graph, the user is able to oversee the robot’s action planner and visually track the robot’s latent states. While this method provides usable information about the robot’s state and intentions, it does not go beyond a visualization framework and does not program the robot directly through the head-mounted-display (HMD), rather through human demonstration. Similar to this work, Avalle et al. (2019) developed an adaptive AR system to display faults in industrial robots using an HMD. Blankemeyer et al. (2018) introduces an intuitive pro- gramming method for speeding up the execution of assembly tasks through marker-based tracking AR. The operator first assembles components virtu- ally using CAD models via an HMD. Once the process is acquired, the robot can reproduce the task in reality. This method has been successfully validated and tested, with an accuracy of about 1–2 mm. A similar approach is pro- posed in (Fang et al., 2014), where robot path planning and end effector orientation planning can be achieved using a hand-held interaction device and the tracking process is carried out by a marker cube. However, the pro- posed system is only tested through a virtual model and does not consist of a real robot. In (Quintero et al., 2018), a multi-feature system is presented that enables the user to program a 7-DoF robot through an augmented trajectory. In addition to visualizing robot parameters, the robot can follow a user prede- fined path both in free space and in contact with the surface. The first mode can be applied to a pick-and-place task, while the latter involves planning the trajectory using a MYO armband. Manring et al. (2020) features two control modes for effective and intuitive control of a robotic manipulator via Microsoft HoloLens. The interesting feature of this work is the opera- tor’s ability to drag and move the virtual end effector. After successful path planning, the robot moves to the desired position. In addition, the torques
Innovative Methods for Robot Programming: Development and Comparison 211
experienced by the manipulator’s joints are visualized via color media. A sim- ilar work to ours is (Ong et al., 2020), however, the authors have developed a handheld device with a wireless mouse and infrared markers that allows the operator to move around the work cell to define waypoints for the real robot to follow. The path and waypoints drawn by the user are displayed via the HMD. However, in our work, we use a virtual pointer integrated with the virtual robot model.
Robot control may be achieved also by using RGB+D cameras such as Microsoft Kinect which is an often-used sensor for such tasks in the robot community. For example, in (Weiming et al., 2020), a virtual UR5 robot is moved according to the skeleton of the human body, which is recognized by Kinect camera. Cueva et al. (2017) proposes the control of a 7-DoF robotic arm by gestures and this approach is enhanced to a teleoperation scenario in (Salamea et al., 2020). In (Maraj et al., 2016), mobile robots are controlled via Kinect camera by a set of gestures. So, gesture control may be seen as a universal method of robot operating and programming.
INTUITIVE HUMAN-ROBOT INTERFACES In this section the three proposed approaches are described.
Robot Programming Through Gesture Control and Augmented Reality This approach emphasizes human factors and ergonomics in robot program- ming through the utilization of Microsoft Kinect and gesture-based methods. Unlike conventional manual control pendants, this method prioritizes intu- itive and universal gesture-based interactions supported by AR. This not only simplifies the manipulation of the robot’s arm but also ensures that operators can control it without specialized knowledge.
This study meticulously designs a set of ergonomic gestures for control- ling the robot’s movement in Cartesian space and operating the gripper. This approach utilizes AR for program verification and modification, employing accessible tools such as monitors or video projectors rather than expensive hardware like HMDs.
Human operators’ gestures are tracked using a Microsoft Kinect RGB+D camera, ensuring a natural and ergonomic interaction. The AR implementa- tion, facilitated by a video projector, enhances the visual interface, providing an immersive and ergonomic experience for the operator.
The gesture recognition considers three ergonomic situations for each hand: thumb, fist, and palm. The left hand selects the robot’s mode, using gestures like an open palm for motion and a fist for gripper closure. This aligns with human factors principles, ensuring understanding and reducing cognitive load.
To start the AR experience, both thumbs point, keeping the live image in the natural field of view. The menu bar, projected for interaction, has clear symbols adhering to ergonomic design, with active buttons in green. Left- hand controls basic functions, right-hand controls supplementary actions, promoting an intuitive workflow.
Teaching the robot involves two modes: balancing speed and precision. The user interface easily switches between them for task adaptability. During motion, the gripper is controlled with natural gestures, reducing cognitive strain.
In programming mode, activated by the left thumb, tool positions are saved using gestures with the right hand. Confirmation is a white LED stripe flash- ing once. Figure 1 illustrates the robot work cell while the operator guides the robot using gestures.
Figure 1: Human operator during gesture control.
Robot Programming Through Mixed Reality and HMD This approach integrates MR technology, specifically Microsoft HoloLens 2, to create a user-friendly and ergonomic interface for cobots. The primary focus of this design is to ensure an intuitive and efficient experience for operators.
Figure 2: Mixed reality human-robot interface with superimposed virtual elements.
By wearing the HMD, operators gain direct control over the robot’s motion, receiving real-time feedback on crucial parameters such as joint
Innovative Methods for Robot Programming: Development and Comparison 213
position, velocity, and applied force. This immersive interface promotes an ergonomic interaction, allowing users to define waypoints for streamlined path planning and execute force-controlled motions around objects of inter- est. Visual feedback is provided to enhance the operator’s understanding of the force exerted on objects. User-friendly sliders are incorporated to allow convenient and dynamic adjustment of applied force, minimizing complexity and effort in programming tasks.
Figure 3: Intuitive path planning through HMD by interacting with virtual cursor.
The user interface (see Figure 2) is designed with a focus on clarity and precision, incorporating text, buttons, and sliders for manipulator control. A virtual cursor aligned with the tool center point (TCP) enhances motion plan- ning in Cartesian space, and users can preview and validate motions using the robot model in the scene. This user-centric design adheres to ergonomic principles to ensure ease of use and understanding and safety.
In this integrated approach, the arrow-like pointer attached to the simu- lated robot flange serves as an input device for defining precise paths. This user-friendly approach simplifies robot programming by allowing the defini- tion of waypoints through the HMD. The motion control initiation involves repositioning the arrow through the HMD and pressing the Teach Way- point button. The emphasis on ergonomic design principles throughout the interface ensures an enhanced robot control experience for operators.
Robot Control Through Joystick and Haptic Feedback Traditional industrial robots lack feelings and are often tasked with delicate operations. How can we enable operators to program robots with precision and sensitivity? The solution lies in letting operators feel what the robot feels through its sensors. Using haptic feedback via a joystick or input device achieves this, allowing operators to sense the forces in real-time. This con- trol system measures force and direction through sensors, transmitting this information as a haptic signal.
The primary goal is to let the operator feel the tactile stimulus, empow- ering them to perform delicate tasks involving fragile or sensitive objects. Applications range from medical procedures to artistic fields and tasks like grinding or delicate handling. This approach could enhance human-robot collaboration by transferring sensitivity to the robot.
In human factors and ergonomics, integrating haptic feedback enhances the operator’s understanding, leading to more precise manipulation. Tactile
feedback fosters a more intuitive connection between the operator and the robot.
Applying gamification principles in robot control tasks maintains operator motivation. This project uses a Sony DualSense Controller connected to the PC and the robot. A GUI explains controller functions and visualizes system values.
The software processes information from the robot and controller for effi- cient interaction. The project aims to control a collaborative robot using a controller transmitting haptic feedback (see Figure 4). The Sony DualSense conveys forces recorded by the robot to the operator, enhancing situational awareness.
Vibration serves as a haptic signal for torques and forces. The right-hand vibrator reacts to torques on the Tool Center Point, while the left-hand vibra- tor intensifies with increasing forces. This integration of haptic feedback and gamification ensures a more intuitive and user-friendly human-robot interaction, aligning with ergonomic considerations.
Figure 4: System structure for robot control via handheld controller.
COMPARISON OF HUMAN-MACHINE INTERACTION METHODS Gesture Recognition Through Kinect
Gesture recognition, facilitated by devices such as Kinect, introduces a hands-free approach to human-machine interaction. One of its significant advantages lies in the intuitive and natural control it offers. Users can com- municate with robots through simple gestures, reducing the cognitive load and enhancing the overall programming experience. This method promotes a user-friendly environment, particularly suitable for individuals without spe- cialized expertise in robotics programming. However, challenges may arise in accurately interpreting complex gestures, potentially leading to errors or delays in command execution. Moreover, prolonged usage might lead to user fatigue, impacting long-term programming comfort. It also restricts human operators’ flexibility as these cameras are typically stationary devices that need to be mounted or placed in fixed locations. This method also requires an external display either a monitor or a projector in our case to show virtual contents which puts more restriction onto the system.
Innovative Methods for Robot Programming: Development and Comparison 215
Mixed Reality HoloLens, leveraging MR, transforms the programming interface by seam- lessly blending virtual elements with the physical environment. This method enhances the operator’s spatial awareness and control over the robot, con- tributing to an ergonomic and immersive programming experience. This allows for interactive and dynamic integration of virtual and real objects, enabling intuitive path planning and cobot control. The HMD also offers greater mobility and flexibility, allowing users to move around freely in their environment while experiencing MR content. The visual overlay of holo- graphic information offers a real-time and intuitive interaction, potentially reducing the learning curve for non-specialized personnel. On the down- side, the need for an HMD may pose discomfort during extended usage, and visual distractions could impact programming focus. Additionally, the technology’s reliance on precise spatial mapping may introduce challenges in certain environments.
Joystick With Haptic Feedback Joystick-based control with haptic feedback provides tactile cues, aligning with human-centered principles. The haptic feedback enhances the operator’s understanding of the robot’s movements, contributing to both time efficiency and programming comfort. This method is advantageous in scenarios where precise control is paramount, offering a familiar interface for users with gam- ing or joystick experience. However, it may require a learning curve for individuals unfamiliar with joystick-based controls. The physical manipu- lation involved can be tiring during prolonged use, and the effectiveness of haptic feedback may vary based on the complexity of the programming task.
DISCUSSION
Traditional programming methods, such as teach-in approaches or program- ming by demonstration, have been foundational in industrial settings. How- ever, these methods often demand significant time investments, particularly when dealing with intricate tasks. In comparison, the three discussed methods offer potential advantages in terms of time efficiency and user-centric design. The intuitive nature of gesture recognition, MR, and joystick-based controls could streamline programming workflows, allowing users to concentrate on tasks rather than navigating complex programming procedures.
All three methods hold potential for making robot programming accessible to individuals without specialized expertise. Gesture recognition and joystick- based controls, in particular, offer a more approachable interface, potentially reducing the barriers to entry for non-specialized personnel. Mixed reality, while immersive, may require a brief learning curve for individuals unfamiliar with HMDs. The effectiveness of these methods in empowering users without specific expertise is contingent on their ease of use, learning curve, and the provision of adequate training resources.
This comparative analysis aims to shed light on the nuanced aspects of each method, providing insights into their applicability from a human factors and ergonomics perspective.
CONCLUSION
In the world of industrial robotics, making robots easier for people to control is crucial. This paper explores three new ways to program robots, mak- ing it more user-friendly. These methods are gesture recognition, HoloLens mixed reality, and joystick with haptic feedback. The goal is to make robot programming more efficient, comfortable, and accessible.
Comparing these methods, Gesture Recognition allows hands-free pro- gramming, but it can be challenging to interpret complex gestures. HoloLens provides an immersive experience but may be uncomfortable for prolonged use. Joystick with Haptic Feedback is great for precise control but may require a learning curve and cause fatigue during extended use.
Compared to traditional programming methods, these new approaches show promise in simplifying workflows and making robot programming accessible to everyone. The study emphasizes the importance of ergonomics in designing the programming interface for robots. As the demand for user-friendly robot programming solutions grows, this research contributes valuable insights for more intuitive and accessible programming interfaces in the future.
ACKNOWLEDGMENT
The authors would like to thank the Federal Ministry of Education and Research (BMBF) for the financial support, as well as Project Management Agency Karlsruhe (PTKA) for the administrative support of the collaborative project “PerspektiveArbeit Lausitz (PAL)”.
REFERENCES
Avalle, G. et al., 2019. An Augmented Reality System to Support Fault Visualization in Industrial Robotic Tasks. IEEE International Conference on Robotics and Automation (ICRA), pp. 132343–132359. Blankemeyer, S. et al., 2018. Intuitive Robot Programming Using Augmented Reality.
Procedia CIRP, pp. 155–160. Boboc, R. G., Moga, H. & Talaba, D., 2013. A Review of Current Applications in Teleoperation of Mobile Robots. pp. 9–16. Cueva, C. et al., 2017. 7 DOF industrial robot controlled by hand gestures using
microsoft kinect v2. IEEE 3rd Colombian Conference on Automatic Control (CCAC), pp. 1–6. Fang, H. C., Ong, S. K. & Nee, A. Y. C., 2014. A novel augmented reality-based interface for robot path planning. International Journal on Interactive Design and Manufacturing (IJIDeM), pp. 33–42. Herlant, L. V., Holladay, R. M. & Srinivasa, S. S., 2016. Assistive teleoperation
of robot arms via automatic time-optimal mode switching. 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 35–42. Li, S., Rameshwar, R., Votta, A. M. & Onal, C. D., 2019. Intuitive Control of a Robotic Arm and Hand System With Pneumatic Haptic Feedback. IEEE Robotics and Automation Letters, pp. 4424–4430. Liu, H. et al., 2018. Interactive Robot Knowledge Patching using Augmented Reality.
IEEE International Conference on Robotics and Automation (ICRA).
Innovative Methods for Robot Programming: Development and Comparison 217
Manring, L. et al., 2020. Augmented Reality for Interactive Robot Control. Special Topics in Structural Dynamics & Experimental Techniques, pp. 11–18. Maraj, D., Maraj, A. & Hajzeraj, A., 2016. Application interface for gesture
recognition with Kinect sensor. IEEE International Conference on Knowledge Engineering and Applications (ICKEA), pp. 89–102. Ong, S., Yew, A., Thanigaivel, N. & Nee, A., 2020. Augmented reality-assisted robot programming system for industrial applications. Robotics and Computer Integrated Manufacturing. Quintero, C. P. et al., 2018. Robot Programming Through Augmented Trajectories
in Augmented Reality. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Salamea, H. T., Auquilla, A. R., Alvarado-Cando, O. & Oñate, C. U., 2020. Control of a Telerobotic System Using Wi-Fi and Kinect Sensor for Disabled people with an industrial Process. IEEE ANDESCON, pp. 1–6. Weiming, Q., Xiaomei, Z., Jiwei, H. & Ping, L., 2020. Real-time virtual UR5
robot imitation of human motion based on 3D camera. 2020 5th International Conference on Mechanical, Control and Computer Engineering (ICMCCE), pp. 387–391.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors and Systems Interaction
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-30-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional