Fady Youssef, Thorsten A Kern (2023). Remote Haptics: Introduction to a touchless remote inspection system for industrial experts. In: Human-Centered Design and User Experience. AHFE Open Access, vol. 114, pp. 676–682. https://doi.org/10.54941/ahfe1004288
Physical inspection poses significant risks, such as exposure to hazardous or even impossible working situations like the presence of rotating parts or a high-temperature environment. Remote inspection with robotic platforms is therefore an established method. This paper introduces a prototype of a touchless remote inspection system for industrial experts, combining both visual and haptic feedback. The system is split into two sub-systems, user- and environment-side.The user side consists of a wearable hand glove, arm exoskeleton, and a VR headset. The environment side consists of a robotic arm, stereo camera, and sensor system mounted on the TCP of the robotic arm that acquires the object properties touchless. The glove and the arm exoskeleton are used to teleoperate the robotic arm and the data gathered by the sensor system are transcoded into haptic signals and realized using actuators mounted on both the glove and the arm exoskeleton. A Robot Operating system (ROS) environment acts as a signal interface between the two sub-systems and gives the opportunity to create a digital twin for testing and applying new control algorithms.To allow a natural feeling in inspection, the sensor system with µm accuracy is used to acquire objects' properties, such as shape, texture information, and vibration. This system consists of a Structured Light (SL) camera with an accuracy of 30 µm at a range of 0.3 m, along with a Continuous Wave (CW) radar sensor.The glove and the arm exoskeleton are designed to allow the natural movement of the user's hand and arm. Twelve IMU sensors measure the fingers’ joints to calculate the fingertips' positions and the orientation of the palm, while the exoskeleton has five Degrees of Freedom (DOF), four for the shoulder, and one for the elbow. On the other hand, they are equipped with different types of actuators to allow the proper haptic feedback. The glove is equipped with Linear Resonance Actuators (LRA) to allow tactile feedback and a braking mechanism for kinesthetic feedback. Another braking mechanism is used in the arm exoskeleton in order to give the proper feedback in the case of collision, for example.With the accuracy of the SL camera, several algorithms are implemented, with the fingertips' positions as input, to extract the texture information and detect the object's shape from point cloud data created by the SL camera. Another algorithm is implemented to make use of the high accuracy of the radar sensor, especially in the case of a vibrating object, to allow the inspection of such dangerous and rather impossible objects physically. The output of these algorithms is then mapped into haptic signals and sent to the user side.The live visual feedback from the stereo camera is fed to a VR/AR environment. Also, the haptic glove is augmented in the environment. The stereo camera is mounted on a mechanism independent of the robotic arm. This mechanism allows mimicking of the user’s head movements while wearing the VR headset with a current latency of 400 ms.
Remote Haptics: Introduction to a Touchless Remote Inspection System 677
INTRODUCTION
Industrial inspection is a critical process for ensuring the quality and reli- ability of products, equipment, and machinery (Harris, 1969). Despite its effectiveness in identifying defects and improving overall quality, physical inspection poses significant risks to inspectors’ health and safety. These risks include exposure to hazardous materials, high-temperature environments, falls from heights, and other accidents related to working with heavy machin- ery or equipment failure (Yu et al., 2019). Remote inspection with robotic platforms is therefore an established method. In such hazardous cases, con- tacting the surfaces may not be possible even remotely, although tactile information may provide relevant details on the situation at hand. Thus a touchless sensor system is required.
The Human Machine Interface (HMI) devices available frequently uti- lize vision sensation displaying information using a graphical user interface (Kristoffersson et al., 2013). However, for the inspection applications, it’s required to include haptic sensations to the inspector, and therefore, the syncronization between the visual- and haptic feedback is important.
Virtual Reality (VR) and Augmented Reality (AR) offer the potential to develop interfaces that are more intuitive and natural by placing users within a 3D environment where they can perceive and engage with robots in shared or remote environments. This immersive experience can enhance situational awareness and facilitate smoother interaction between the user and the robot (Wonsick and Padir, 2020).
In this paper, a touchless remote inspection system is introduced. The system combines both the visual- and haptic feedback.
Figure 1: Sketch of the proposed system.
PROPOSED SYSTEM The proposed system (see Figure 1) is designed to combine both visual and haptic feedback. It is split into two subsystems, user- and environment-side. The user side consists of a wearable glove, an arm exoskeleton, and VR
678 Youssef and Kern
headset, while the environemnt side consists of a 6 DoF robotic arm, a camera, and a sensor system.
The glove and the exoskeleton are used to teleoperate the robotic arm where the sensor system is mounted on its TCP. In order to allow a natural inspection scenarios and mimicking the same analogy as the human body, the camera is mounted on a 3 DoF mechanism that is independent from the robotic arm, this allows controlling the visual feedback without interfering in the teleopration of the robotic arm. At the current stage of the system development, a 6 DoF UR10e robotic arm is used.
The sensor system mounted on the TCP of the robotic arm extracts the information of the inspected object touchless. Several algorithms are designed and implemented to map the information such as shape, vibration frequecy, and surface roughness, into haptic signals.
The live visual feedback from the camera is transmitted to the VR headset, where an AR environment is introduced to the user, with the glove along with the real-time positions of the fingers are augmented in the environment.
A signal interface and control layer connects both sides of the system. It passes the input positions signals from the head, glove, and the exoskeleton to the camera mechanism and the robotic arm. Also it passes to the user side, the haptic signals generated from the sensor system.
Requirements The design of the glove, the exoskeleton, the camera mechanism and as well the choice of the components of the sensor system are done based on differ- ent requirements. The requirements are mainly human-centered. They vary, from the anatomy of the human body, passing through the limits of human’s tactile and force feedback, ending at the frequency of head movements. Other requirements are introduced as a result of the application itself, such the accu- racy of the sensors and the ability of gathering information without touching the object.
The prototypes of the glove and the exosekelton are based on the anatomy of the human body, in the form of, for example, the range of motion of each joint, the dimensions of the hand and the length of the arm (Tilley, 2002). Both prototypes are designed to allow the free movement of the user’s arm and hand, and to stop the movement whenever needed. Similarily, the camera mechanism is designed to follow the same range of the user’s head movement, taking into consideration the frequency of the head movements and the time delay limit for motion sickness, as these affect the control of the actuators used in the mechanism.
On the other hand, the sensors composing the sensor system are cho- sen based on the haptic requirements of the human body, for example, the human body is able to detect tactile signals up to 1 kHz (Kern et al., 2023). This affects the choice of the sensor to detect the vibration of the objects. Additionaly, the natural inspection scenario requires the extraction of texture information, which is translated mechanically to surfaces that has a relative roughness in the µm range. This means that the sensor used should have a spatial resolution in the µm range as well.
Remote Haptics: Introduction to a Touchless Remote Inspection System 679
Transcoding Sensors’ Data Into Haptic Signals To fulfill the requirements, both Radar and Structured Light (SL) technolo- gies are chosen to build the sensor system. Radar technology is chosen for its biggest advantage of detection of objcts’ speeds. On the other hand, with Structured Light technology, µm spatial resolution could be reached. At the current stage of the system development, the sensor system cinsists of a Con- tinous Wave (CW) Radar sensor BGT60AiP from infineon technologies, and Mech-Eye UHP 140 structured light camera (SL).
As mentioned earlier, the natural feeling of inspection requires the extrac- tion of the object’s information. A method was designed (see Figure 2) to extract and map texture information, shape, and vibration of the object.
Figure 2: Method to transcode the acquired object information touchless into haptic signals.
Several algorithms are implemented, with the signals of the sensors and the real-time fingertips positions from the glove. The SL camera produces a point cloud (around 3 million points) with a spatial resolution of 30 µm at a range of 0.3 m. The sampling frequency of the CW radar sensor is 2 kHz with an average frequency error of 0.2% at a distance of 0.3 m from the object. The output of the algorithms are then mapped into haptic signals, in the form of vibrations and forces on both the glove and the exoskeleton.
Glove Design The prototype of the glove is designed to allow natural movement of the user’s fingers (see Figure 3). Twelve Inertial Measurement Unit (IMU) were used to measure the joints’ angles, one is used for the orientation of the palm. The other 11 IMUs are used for the fingers’ joints, three for the thumb, and two for the rest of the fingers. The decision of using only two sensors to measure the angles of the MCP, DIP, and PIP angles of the other four fingers was based on the correlation between PIP and DIP angles of each finger (Hrabia et al., 2013).
680 Youssef and Kern
The glove is equipped with five Linear Resonance Actuators (LRA), at the fingertips, for realization of the tactile feedback signals generated from the different algorithms (see Figure 2). Additionally, a braking mechanism is designed to allow the force feedback to the fingers. The mechanism is com- posed of a servo motor driving a rack and pinion to set the stopping position of each finger (see Figure 3).
Figure 3: Prototypes of haptic glove (left) and arm exoskeleton (right).
Arm Exoskeleton Design Similar to the glove, the arm exoskeleton’s prototype is designed to allow the natural movement of the arm. It consists of five DoF, one for the elbow, and the other four are for the shoulder (see Figure 3). Five rotary encoders are used to measure the joints’ angles. The kinematic model of the exoskeleton helps in ensuring the stopping of the joints as a result of the collision detection algorithm (see Figure 2).
The exoskeleton is equipped with a braking mechanism, that is com- posed of five identical compact units, one for each DoF. Each unit consists of a servo motor connected to a gear system. The average time needed to actuate the servo motor and thus block the movement of one joint is 100 ms.
Signal Interface and Control Signal interface environment is designed on Robot Operating System (ROS). The environment (see Figure 4) is used to pass the signals from the user side to the environment side and viceversa. The main advantage of the ROS envi- ronment is the ability of integrating several sub-systems. With this ability, the components of the system, such as, the glove, exoskeleton, sensor system, etc. could be considered as sub-systems.
The other advantage of using ROS enivornment is the creation of a Digital Twin for the system. The digital twin offers the ability to simulate the system offline, such that, control algorithms could be tested in the digital twin before implementation on the hardware. The importance of the digital twin is clear in testing the teleoperation of the robotic arm using the position signal of the fingertips of the user, for example.
Remote Haptics: Introduction to a Touchless Remote Inspection System 681
Figure 4: Architecture of the signal interface environment.
Visual Feedback The live visual feedback from the ZED Mini stereo camera is fed to the HTC VIVE Pro VR headset with a refresh rate of 100 FPS.The VR/AR environment is created on Unity.
To allow a better user experience, a 3 DoF mechanism, holding the cam- era is designed and implemented to mimic the user’s head movements. The angles of the head are measured using the internal headset accelerometer and gyroscope, the signals are then transmitted to the mechanism via the ROS envrionment. The current average latency of the mechanism is 400 ms.
The glove is augmented in the AR environment (see Figure 5). The real- time angles of the palm and fingers are measured and sent to Unity via the ROS environment.
Figure 5: Augmentation of the haptic glove in the live visual feedback from the camera.
682 Youssef and Kern
NEXT STEPS IN DEVELOPMENT The current sensr system could be extended with including temperature mesaurement in the inputs of the method (see Figure 2). This will enrich the user exeperience by the temperature information of the object to the user along with the texture and the vibration information.
Additionally, the research is still running on the effectivness of using LRA actuators. Although LRA offers a relatively high frequency bandwidth, it will also be a challenge provide a transperant transmission of vibrations of the objects measured with the radar sensor. The research could be done on the user experience using Piezoelectric actuators.
CONCLUSION
In this paper, a touchless remote inspection system for industrial experts was introduced. The system is split into two sub-systems, user- and environment- side. The user-side consists of a haptic feedback glove, an arm exoskeleton, a VR headset. A 6 DoF robotic arm, a sensor system, and a camera comprise the environment-side. The glove and the exoskeleton are used to teleoperate the robotic arm. The sensor system acquire the inspected object’s texture- and vibration information touchless. The camera is mounted on a 3 DoF mechanism to mimic the head movements of the user. The sensors used and the design of the glove, exoskeleton, and the camera mechanism prototypes are based on the different requirements, varying from human body anatomy, ergonomy, and the ability to acquire inforamtion touchless with µm accuracy. Several algorithms are implemented to transcode the texture- and vibration information into haptic signals (tactile and force), and realized using haptic actuators and stopping mechanism equipped in the glove and the exoskeleton, respectively.
REFERENCES
Harris, Douglas H. (1969). The Nature of Industrial Inspection. In: Human Factors 11.2 pp. 139–148. doi: 10.1177/001872086901100207. Hrabia, Christopher-Eyk, Wolf, Katrin, and Wilhelm, Mathias. (2013). Whole
hand modeling using 8 wearable sensors. In: Proceedings of the 4th Augmented Human International Conference on - AH ‘13. Ed. by Albrecht Schmidt, Andreas Bulling, and Christian Holz. New York, New York, USA: ACM Press, pp. 21–28. ISBN: 9781450319041. DOI:10.1145/2459236.2459241. Kern, Thorsten A., Hatzfeld, Christian and Abbasimoshaei, Alireza, eds. (2023).
Engineering haptic devices. Springer. Kristoffersson, Annica, Coradeschi Silvia and Loutfi, Amy. (2013). A Review of Mobile Robotic Telepresence. Advances in Human-Computer Interaction, vol. 2013, Article ID 902316, 17 pages, https://doi.org/10.1155/2013/902316 Tilley, Alvin R. (2002). The measure of man and woman: Human factors in design.
Revised edition. New York: John Wiley & Sons. ISBN: 9780471099550. Wonsick, Murphy and Padir, Taskin. (2020). A systematic review of virtual reality interfaces for controlling and interacting with robots. In: Applied Sciences 10.24, p. 9051. Yu, Leijian, Yang, Erfu, Ren, Peng, Luo, Cai, Dobie, Gordon, Gu, Dongbing, and
Yan, Xiutian. (2019). Inspection Robots in Oil and Gas Industry: a Review of Cur- rent Solutions and Future Trends. 1–6. DOI:10.23919/IConAC.2019.8895089.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human-Centered Design and User Experience
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-90-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional