Angelo Compierchio, Phillip Tretten, Solomon Oyelere, Chris Karagiannis (2023). Assistive VR platform design for Telemanipulation at the Super Fragment Separator Facility. In: Human Interaction and Emerging Technologies (IHIET-AI 2023): Artificial Intelligence and Future Applications. AHFE Open Access, vol. 70, pp. 186–197. https://doi.org/10.54941/ahfe1002943
An assisted remote manipulation (ArM) platform has been defined for the Super FragmentSeparator (Super-FRS) main tunnel and hot cell at the High Energy Physics (HEP) Facilityof Anti-proton and Ion Research (FAIR). The designed platform positioned within a VirtualReality (VR) based framework ensures dynamic collaboration and effective humaninteraction to assist with Remote Handling (RH) operations. To visually stimulate operatorassisted intervention in harsh environments, enhanced interaction based on syntheticvision has been adapted with simultaneous localization and mapping (SLAM) techniquesinterlinked with virtual layers representing a three dimensional manipulation of RHmaintenance tasks. The proposed platform also included a sequence mapping toolevaluated with RH task variables specific to the sequence space analyses of pathplanning, motion check, and collision detection performed in both real and virtual RH taskenvironments. Further assistance was envisaged from multimodal feedback categoriesthrough force feedback, in this case, a backpropagation algorithm was tailored to define aforce limit and to send feedback signals to the operator every time the actual patternexceeded the desired output pattern. Overall, the ArM platform ensures the application ofbest engineering practices to RH needs as a basis to maximize information gathering andsharing driven by continuous improvement initiatives.
The extension of the Facility for Anti-Proton and Ion Research (FAIR) has resulted in an operational complexity increase from presently O(n2) (GSI) to O(n2 (FAIR)) due to the longer accelerator chains (Steinhagen, 2018). The Super-FRS main tunnel and hot cell at the FAIR facility contain enclosed environments far too hazardous for any manned maintenance. Therefore, the operator needs to perform both transportation and manipulation of
Assistive VR Platform Design for Telemanipulation 187
hazardous items at a distance, amongst predefined maintenance paths. This function introduces the field of teleoperation in the context of Remote Hand- ling (RH). Similarly, telemanipulation extends the RH concept further from the most basic to high-level manipulation tasks.
The duty of performing maintenance tasks remotely requires human intel- ligence attesting the operator rather than a computer in the control loop, as quoted: “a machine enabling a human operator to move about, sense and mech- anically manipulate objects at a distance... generally, any tool which extends a person’s mechanical action beyond his reach is a teleoperator” (Sheridan et al. 1995).
At the forefront of every maintenance activity the human presence or fee- ling of “being there” is highly dependent on the capability of the system employed (Almeida, Menezes and Dias, 2020). This approach has been consi- stent at the GSI facility through a “look and feel” approach provided for the operator while interfacing with control programs (Krause, Schaa and Stei- ner et al. 1992). Since the introduction of the control architecture in the late seventies, control tools have also provided higher degree of abstraction and modelling to facilitate operators’ handling tasks in real-time. Nowadays, this approach continues offering to operators the opportunity of performing RH repeatedly in a safe and reliable manner.
The Super-FRS handling scenario encompasses a target area, a hot cell and a main separator open tunnel. The target area contains a heavily shielded closed tunnel for positioned components that must be brought to the hot cell by the hall crane and handling shielding flask system (Fs1, Fs2) illustrated in Fig. 1. The hot cell layout combines two master slave manipulators and a power manipulator with various general-purpose tools, whereas a mobile robot platform enables RH in the main separator open tunnel.
Figure 1: Super-FRS Target Area overview (Public Domain).
Furthermore, the careful intervention planning in radiation areas presents enhancement opportunities from combined simulation measurements with robotic tools engaged in transportation and dexterity manipulation.
Enhancement herein is not considered in the condition of “perfecting” or “optimizing” an entity end of state, rather a measurable enhancement direction consisting of a measurable qualitative or quantitative parameter. When a maintenance event is initiated, timing becomes the main issue for both multisensory information integration and attentional processes (Martin, Malpica, Gutierrez, Masia, and Serrano, 2022). This concern can sometimes lead to multisensory overload that might hinder the manipulation experience.
Therefore, sensory and proprioception information must be synchronized by balancing the creation of virtual simulated environments with the operator inherent manipulation capacity. Advanced modelling with virtual simula- tion allows to validate RH maintenance tasks, while minimizing task design planning through equipment and process optimization. Furthermore, VR technology induces in an organism targeted behaviour stimulated by artificial sensors, (La Valle, 2019). The computational model alone does not represent a complete VR system without the organic interaction experienced with arti- ficial stimuli transmitted from the hardware. As a result, the stimulation of human’s senses extends the VR engineering capability to the human physiolo- gical and perceptual activity enabling many powerful interaction mechanisms under a “Symbiotic” (S) condition interpreted as a Brain-Machine Interface as shown in Table 1.
Table 1. Intelligent Levels (Adapted with permission from [Bezdek], Bezdek et al. 1994).
Table available in the original PDF paper.
BI Your software: the mind Cognition, memory and action: you have them! AI Mid-level models: CI (+) Knowledge Tidbits
Mid-level cognition in the style of the mind CI Low-Level algorithms that reason computationally
Low-level cognition in the style of the mind SI High-Level algorithms VR (+) Near-Real Time Intuition
High-level cognition in the style of the mind
This concept extends the levels of Intelligence (I), Artificial (A), Biological (B) and Computational (C) to four levels A, B, C and S. This terminology has been portrayed in the pyramid scheme within the assistive platform interconnected data structures developed in Fig. 2.
Figure 2: The Intelligent Pyramid of Knowledge (a), applied to the ArM Structure (b).
Assistive VR Platform Design for Telemanipulation 189
The ability of VR has extensively been adopted in manufacturing robotic applications (Eswaran and Bahubalendruni, 2022), medical (Zhou, Zhang, Feng, Zhao and Fu, 2018) and space applications (Hulin, 2021) and deep imitation learning capabilities (Zhang, 2017). With a proven record in HEP facilities, simulation in VR nowadays form the backbone of RH training, veri- fication and validation of maintenance tasks (Domínguez, 2017). The Super- FRS ArM platform allows a full integration of interactive multimodal stimuli and combinatory sensory feedback. The operator’s interaction responsiveness was also investigated by mapping individual RH tasks through direct-based visualization within a spatially aligned model. (Szalavári, Schmalstieg, Fuhr- mann and Gervautz, 1997) The benefit of this approach enabled to explore decision uncertainty in terms of degraded accuracy, latency and low feedback update rate anchored to the operator inherent near-real-time cognitive state (Bryson et al. 2003).
ASSISTIVE DESIGN ELEMENTS The design of the assistive platform begins with the analysis of the RH Spe- ctrum (RHS) exhibiting the “unknown vs hazardous” process, the spectrum provides a good basis to develop a process to optimize operation and main- tenance activities (Amjad, 2018). The process model depicted in Figure 3 emphasizes an RHS extension diving into the environment where a high level of radiation presents a real challenge when maintenance needs to be addressed.
Figure 3: The RHS of Unknown vs Hazardous (a), developed ArM assistive elements (b).
In the maintenance domain, decision support relates to the situational awareness concept and underlying cognitive processes. Therefore, emphasis will be placed on a human-centered simulation approach and the associa- ted psychological processes of the target tasks (Kozlowski and DeShon et al. 2004). In practice, the RHS is extended into a world including two outer edge working zones. Since, this definition is a way to mimic decision-making, to capture the operator’s knowledge level, the definition of reasoning zones is
considered most relevant, for recognizing a situation or coping with the unfo- reseen. The shaped sides of the outside world are seen as distributed across the operator, artifacts, and time setting (Hutchins et al. 1995). From this per- spective, a distributed viewpoint of cognition regarding the operator actions and information processing of the RHS has been categorized as a processor with stimuli/response elements as shown in Table 2.
Table 2. ArM assistive elements and modules.
Table available in the original PDF paper.
Elements Primary Level Secondary Level ArM Module Virtual Reality Sensing Interacting with the environment Seeing, hearing, touching, measuring distance
Synthetic Vision
Effectors Interfacing with the environment
Producing signal / information or moving about
Sequence Mapping
Information acquired through perception
Interpreting Interpreting information by means of intelligence Generating Influencing environment by means of intelligence
Corrective Feedback (Multimodal e.g. Visual. Haptic, Tactile)
Direct actions (as manipulating, assembly) and Indirect actions (as language / picture generation)
The coordination of senses both visual and motorial is organized according to the operator’s activity task at hand. At this point, the operator’s handling task and VR task-relevant information must be designed when perception and action coincide at the same time for manoeuvrability and dexterity acti- vities. The operator’s proprioception is engaged by directly mapping physical movements to a noticeable output in space, causing them to feel present in a virtual environment.
HUMAN FACTORS The effects of sensory response on VR task were taken into consideration with the aim of offering a larger and richer spatial arrangement to the opera- tor. The opportunity of bringing spatial cognition into VR was reinforced by bringing in direct manipulation training. This capability enabled the opera- tor to adopt external spaces to extend spatial accuracy, spatial response, and spatial manipulation (Andrews, Endert, and North, 2010).
Improved task completion performance was shown through the combi- nation of visual and haptic information rather than haptic feedback alone without proprioceptive neural coordination. Therefore, to improve intera- ction efficacy the operator-VR interface transfer points had to be shifted
Assistive VR Platform Design for Telemanipulation 191
transforming the classical representation from a human-machine system to a human-virtual machine system.
In this context, the level of knowledge gained by the operator might not be sufficient to conduct time stamp RH tasks. For this reason, assistive vir- tual layers were created upon the task response time divided according to time-based event information using perceptual stimuli and tasks as shown in Figure 4.
Figure 4: Classification and Validation Loop of Maintenance Learning Tasks.
Each broadcasted event was classed according to task-related stimuli, spe- cifically to stresses influencing memory retrieval (Shields et al., 2017), spatial orientation (Keller et al., 2020) and decision-making tasks.
Recognition Memory Tasks where the operator learns to recognize locali- zed sequential targets from a virtual environment map.
Spatial Orientation Tasks enabling the operator to orient specific compo- nents in predefined locations.
Decision Making Tasks allowing to prioritise maintenance classification schemes.
The inter-correlation of each task indicated in the loop is intended to high- light cognition distributed in both direct and indirect manipulation. While direct manipulation allows to create rapid responses with greater flexibility, it proves beneficial also to common indirect manipulation practices. These pra- ctices are commonly adopted in graphical environments where the change of state of an object respond to control exerted by manipulating another object.
This degree of association is paramount to anchored RH task results repor- ted on subjective authority assessment protocols and metrics such as fidelity, confidence, usability, and reliability. The loop is intended to supply RHS and sequence-based metrics for individual RH tasks between the operator’s taskwork and appraised observer teamwork. This form of mediated cognition and collaboration is considered in terms of collaborative mitigation/adapta- tion of actions/responses. In this innovative context, the ArM is positioned to enhance interaction, decision-making and control potential actions trigge- red by uncertainty, as guided by the distinguished “simulative model-based reasoning” approach (Nersessian et al., 2002).
PLATFORM The design of the ArM platform follows a State-of-The-Art study tailored to fulfill the Super-FRS RH requirements whilst ensuring the integration of RH tasks into the facility’s phase of operation and maintenance. The functionality of the RH system is integrated in a closed-loop structure where the operator is able to perform maintenance with assistive modules as indicated in Figure 5.
Figure 5: The ArM intelligent closed loop model.
Assistive modules allow the operator to fulfil maintenance actions synch- ronized with virtual layers created from the facility White Rabbit event gene- rator through direct manipulation visualization techniques. Since RH tasks are subjected to time constraint, the conflict between accuracy and respon- siveness is mitigated through a time variance model. In addition, unexpected behavior of manipulation contact points has been explored with extracted manipulation primitives to ensure verification, validation and integration of each task.
The advantage to impart training via direct manipulation allowed the operator to develop interaction responsiveness abilities and time-critical maintenance skills indispensable in radiation areas.
VISION SYSTEM The increasing pace of using VR to train operators for industrial maintena- nce and assembly elicited the bringing of virtual information directly to the
Assistive VR Platform Design for Telemanipulation 193
workplace. The proposed use of synthetic glasses provided the operator with a clearer and more intuitive knowledge of the operational state of the Main Tunnel Remote Handling (MTRH) system equipment and the hot cell dispo- sal area (Amjad, 2016). Synthetic vision ensures a meaningful evaluation of the usability and effectiveness of the live 3D model of the environment under typical harsh conditions, as the visualized radiation shown in the hot cell window-less in Figure 6.
Figure 6: Working area of as operator wearing Augmented Reality Glasses.
While SLAM captured and maintained the mapping of the robot’s geo- metric features for optimized and localized 3D representation. In addition, the synthetic vision system allowed monitoring for a possible collision of the elbow of a tele-manipulator in 3D without any camera views.
SEQUENCE MAPPING Generated sequences are employed to analyze the Master-Slave motion and kinematics of the manipulator when conducting RH maintenance actions. Sequence mapping is critical to the functionality of the task (Wang, Ma and Cheng, 2015). This temporal component is being considered for each sequ- ence step on a recursive basis, mapping the operator’s state t to time t + δt on the robot. These features starting from the location place are evaluated with specific activities and state’ actions for each step. As a proof of concept of the selected approach, sequence performance steps are customized to a set of pre-defined variable thresholds. A Neural Network (NN) model is imple- mented using variables specific to sequence space analyses as path planning, motion check, collision detection, and RH device weight parameters that are representative of a normal and safer RH task sequence. Compared to other methods (decision trees), building a NN model based on these features
ensures a normal sequence operation and allows to highlight anomalous data and operational changes in the input-output distribution. This approach has been extensively used in industrial applications process monitoring and control (Saucedo-Dorantes, Elvira-Ortiz, Jaen-Cuéllar, and Toledano-Ayala, 2021). For a typical disassembly-and-assembly operation, task stages are fed to the NN training model presented in Figure 7.
Figure 7: Typical RH stages data inputted to a NN algorithm.
The NN model is data-driven and appropriate sensitivity is sought in terms of the appropriate size (number of nodes) used and the level of novelties flagged as quantization errors (QE) by the tool. The declaration of the novelty of any drift and rapid changes indicated in the measurement, exceeding a pre- defined threshold is bounded by a switching point represented by the middle sky cross point between the two centres. The threshold compliance has been defined according to the 3-sigma rule.
CORRECTIVE FEEDBACK At the Super-FRS hot cell, the manipulation learning process relies on the operator ability to anticipate the size, weight, shape and feel of the physical object during manipulation. In this context, visual cues and sensorimotor memories are largely responsible for the propagating torque compensation errors.
These finding are mainly subdued to the dynamic effects of weight com- pensation and force scaling (Schneider, Buckingham and Hermsdörfer, 2020). The predictive scale of the applied force was appraised with a novelty simu- lation approach based on a supervised learning backpropagation algorithm. The trained NN algorithm would ensure a constant and controlled force applied to each manipulative action as a function of the task sequence. Since the state of a task is associated to the state of the components, at every change of state, the assembly and disassembly process of a component is further divi- ded into small subprocesses (Domínguez, 2017). At this stage, each featured subprocess step sequence is divided further as a function of the applied force and projected on a backpropagation algorithm as shown in Figure 8.
Assistive VR Platform Design for Telemanipulation 195
Figure 8: Task sequence and applied force (a), backpropagation principles (b).
The backpropagation algorithm employs a pattern learning rate melding the sensorial force and time sequence. Each subprocess sequence is catego- rized according to the input pattern Y(t). When the manipulation task is performed by a different operator the actual measured force Y’(t) may differ from the desired output Y(t). Therefore, for each task sequence function, a learning rule must be established for adjusting the weights/threshold to get the actual output approximation to the desired Y(t) “feeding force”.
CONCLUSION
The development of concepts and strategies at the Super-FRS is set to conti- nue during the Hardware and Beam commissioning stages, with RH activities set to be integrated right from the beginning and deal with problems as they appear, to avoid increasing costs and time delay until the problem is disco- vered and resolved (Steinhagen, 2018). In this context, assistive modules were formulated to address real-time interventions with improved situati- onal awareness and by blending physical reaction forces with the required manipulation dexterity.
The implementation of direct manipulation contributed to a better under- standing of manipulation planning problems with the creation of active virtual layers for balancing demanding RH requirements with the opera- tor cognitive and physical capacity. The presented ArM concept unfolds distributed intelligence into highly automated RH systems.
ACKNOWLEDGMENT
The authors would like to acknowledge Prof. James C. Bezdek for the use and adaptation of The ABCs structure.
REFERENCES
Almeida, L. Menezes, P. Dias, J. (2020). Interface Transparency Issues in Teleoperation. Applied Sciences,10(18), 6232; https://doi.org/10.3390/app10186232. Amjad F. (2018) Remote handling for the Super-FRS components, availa- ble online https://indico.gsi.de/event/6970/attachments/22589/28345/Amjad_GS I_150318.pdf. (Presentation)
Amjad, F. (2016). Systematic approach for the development of remote handling system concepts for high energy physics research facilities, Thesis for the degree of Doctor of Science in Technology, Tampere University, Tampere. Andrews, C. Endert, A. and North, C. (2010). Space to Think: Large High-Resolution
Displays for Sensemaking, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2010 Proceedings, 55-64. Bezdek J. C. (1994). Computational Intelligence Defined - by Everyone!, in Computational Intelligence: Soft computing and Fuzzy-Neuro Integration with Applica- tions, eds. O. Kaynak, L. A. Zadeh, B. Turksen and I. J. Rudas, NATO ASI series F, v. 162. Bryson, S. (2003) Direct Manipulation in Virtual Reality, available online (https://ntrs.nasa.gov/citations/20030054521) Diggs Bailey, H. Mullaney A.B. Gibney, K.L. and Dowell Kwakye, L. (2018). Audi-
ovisual Integration Varies With Target and Environment Richness in Immersive Virtual Reality. Multisensory Research 31, 7. Domínguez, O.M.L. (2017). Principles and Techniques to Systematically Define, Analyze, and Develop Remote Maintenance Tasks for Machines Operating in Ionizing Radiation Environments. Thesis for the degree of Doctor of Science in Technology, Tampere University. Eswaran, M. Raju Bahubalendruni M.V.A. (2022). Challenges and opportunities on
AR/VR technologies for manufacturing systems in the context of industry 4.0: A state of the art review, Journal of Manufacturing Systems, Volume 65, Pages 260-278, https://doi.org/10.1016/j.jmsy.2022.09.016. Hulin T. Panzirsch M. Singh H. Coelho A. Balachandran R. Pereira A. Weber Bernhard M. Bechtel N. Riecke C. Brunner B. Lii Neal Y. Klodmann J. Hellings A. Hagmann K. Quere G. Bauer Adrian S. Sierotowicz M. Lampariello R. Vogel J. Dietrich A. Leidner D. Ott C. Hirzinger G. Albu-Schäffer A. (2021). Model- Augmented Haptic Telemanipulation: Concept, Retrospective Overview, and Cur- rent Use Cases Frontiers in Robotics and AI, DOI=10.3389/frobt.2021.611251. Hutchins, E. (1995) Cognition in the Wild (Vol. 19). MIT press: Cambridge, MA,
USA. Keller, A. M. Taylor, H. A. and Brunyé, T. T. (2020). Uncertainty promotes information-seeking actions, but what information? Cognitive Res. Princ. Implic. 5 (1), 42. doi:10.1186/s41235-020-00245-2. Kozlowski, S. DeShon, R. (2004) A Psychological Fidelity Approach to Simulation-
Based Training: Theory, Research, and Principles. In Scaled Worlds: Development, Validation, and applications, 1ed,; Ashgate Pub Co. S. G. Schiflett, L. R. Elliot, E. Salas, & M. D. Coovert (Eds.), (pp. 75–99). Krause, U. Schaa V. and Steiner R. (1992). The GSI Control System Available online: https://inis.iaea.org/ (accessed on 18 December 2022). La Valle, S. M. (2019). Virtual Reality. Cambridge University Press. Martin D. Malpica S. Gutierrez D. Masia B. and Serrano A. (2022). Multimo-
dality in VR: A Survey. ACM Comput. Surv. 54, 10s, Article 216, 36 pages. https://doi.org/10.1145/3508361. Nersessian, N. J. (2002). The cognitive basis of model-based reasoning in science In S. S. P. Carruthers & M. Siegal (Eds.). Cam-bridge: Cambridge University Press. Saucedo-Dorantes, J.J. Elvira-Ortiz, D.A. Jaen-Cuéllar, A.Y. and Toledano-Ayala,
M. (2021). Novelty Detection Methodology Based on Self-Organizing Maps for Power Quality Monitoring. In Artificial Intelligence, DOI: 10.5772/intech- open.96145
Assistive VR Platform Design for Telemanipulation 197
Schneider, T.R. Buckingham, G. Hermsdörfer, (2020). J. Visual cues, expectations, and sensorimotor memories in the prediction and perception of object dynamics during manipulation. Exp Brain Res 238, https://doi.org/10.1007/s00221-019- 05711-y. Sheridan, T. B. (1995). Teleoperation, telerobotics and telepresence: A progress
report, Control Engineering Practice, vol. 3, no. 2, pp. 205–214. Shields, G. S. Sazma, M. McCullough, A.M. and Yonelinas, A. P. (2017). The effects of acute stress on episodic memory: A meta-analysis and integrative review. Psychol. Bull. 143 (6), 636–675. doi:10.1037/bul0000100. Steinhagen, R.J. (2018). Fair Commissioning – Concepts and Strategies in View of High-Intensity Operation. 61st ICFA ABDW on High-Intensity and High-Brightness Hadron Beams HB2018, Daejeon, Korea JACoW Pub, doi:10.18429/JACoW-HB2018-TUA2WD01. Szalavári Z. Schmalstieg D. Fuhrmann A. Gervautz M. (1997). “Studierstube”
- An Environment for Collaboration in Augmented Reality, available online (https://www.cg.tuwien.ac.at) Wang, X. Yang, C. Ma, H. and Cheng, L. (2015) Shared control for teleoperation enhanced by autonomous obstacle avoidance 391 of robot manipulator,” in Proc. 2015 IEEE/RSJ Int. Conf. Intell. Robots Syst. IEEE, 2015 Zhang, Z. T. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. (2017)
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation. Zhou, X. Zhang, H. Feng, M. (2018). New remote centre of motion mechanism for robot-assisted minimally invasive surgery. BioMed Eng OnLine 17, 170 https://doi.org/10.1186/s12938-018-0601-6.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Interaction and Emerging Technologies (IHIET-AI 2023): Artificial Intelligence and Future Applications
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-46-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional