David González, Guillermo Aguilera, Jeronimo Rueda Giraldo, Iván Mondragón, Wilson Hernandez, Catalina Alvarado, Diego Mendez, Julian Colorado (2026). Mixed Reality–Driven Rehabilitation Using a Robotic Exoskeleton and an Immersive Game-Based Interface. In: Human Interaction and Emerging Technologies (IHIET-FS 2026): Future Systems and Design Applications. AHFE Open Access, vol. 235, pp. 324–333. https://doi.org/10.54941/ahfe1007286
Stroke is a major cause of long-term disability worldwide and frequently leads to lasting motor deficits that severely compromise upper-limb function and diminish patients’ independence. The World Health Organization reports that every year millions of people live through a stroke but are left with permanent impairments that demand ongoing rehabilitation. However, access to intensive, individualized treatment is often constrained by social, economic, and technological factors, as well as poor adherence to repetitive training routines. To address these issues, this work introduces an interactive Mixed Reality (MR) rehabilitation system aimed at supporting upper-limb motor recovery while enhancing patient motivation. The system targets right-hand bidigital pinch training and offers two complementary operating modes: (i) an assisted therapy mode using a Bluetooth-controlled robotic exoskeleton, which guides users through prescribed therapeutic movements and exercise sequences; and (ii) an immersive, game-like mode that situates rehabilitation tasks within Activities of Daily Living (ADL) contexts. The platform was implemented in Unity and deployed on the Meta Quest 3S headset, delivering real-time visual feedback via an interactive human–computer interface. The system was evaluated through rehabilitation sessions with healthy volunteers, who subsequently assessed their experience using standardized tools, including the System Usability Scale (SUS), the Virtual Reality Sick-ness Questionnaire (VRSQ), and the NASA Task Load Index (NASA-TLX). The findings revealed good usability and manageable workload levels, indicating that the proposed MR-based system is a promising and engaging complement to conventional upper-limb rehabilitation approaches.
Stroke is a leading cause of disability in the whole world, often resulting in significant motor impairments that affect patients’ quality of life. Recent estimates report 11.9 million new cases annually and 93.8 million people living with stroke-related disabilities (Yao et al., 2026). Effective rehabilitation therefore is essential to restore motor function and patient independence through intensive, repetitive, and specific training (Langhorne
Mixed Reality–Driven Rehabilitation Using a Robotic Exoskeleton and an Immersive 325 et al., 2011). However, access remains limited due to economic barriers, lack of infrastructure, and low patient motivation. Although technologies such as robotic systems and virtual reality have been introduced, many lack real-time adaptability and user-centered design, limiting their effectiveness.
This paper presents a Mixed Reality (MR)-based rehabilitation system integrated with a robotic exoskeleton to provide interactive and patient- focused therapy. The system includes two different modes: (i) assisted therapy using the exoskeleton for structured exercises, and (ii) an interactive game environment based on daily activities. By combining physical assistance with immersive interaction, the system aims to improve patient motivation while supporting functionally relevant rehabilitation.
RELATED WORK
Technological advancements have significantly transformed post-stroke neurorehabilitation, particularly in the recovery of upper-limb and hand function. Immersive technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality, along with robotic and exoskeleton-assisted systems, have experimented with increasing integration into therapeutic protocols to enhance motor recovery. These systems provide real-time feedback and allow continuous performance monitoring (Mondragón et al., 2025). They also simulate functional tasks in controlled interactive environments and leverage stereoscopic visualization and motion tracking to promote a sense of presence and engagement, factors that are considered critical for improving adherence and therapeutic outcomes.
Accordingly, several studies have reported practical implementations of integrated rehabilitation systems that combine robotic exoskeletons with computational and immersive environments. For example, as shown in (Falkowski and Gwardecki, 2024), an exoskeleton is simulated using the Robotic Operating System (ROS) and integrated into a VR environment to support telerehabilitation applications, enabling users to interact with the robotic model within an immersive space. Similarly, from in a non- medical context, the BiDexHand project demonstrates real-time interaction between immersive head-mounted displays and a dexterous robotic hand, allowing users to visualize and control complex hand movements in a virtual environment (Weng and contributors, 2024). Within the Colombian context, one representative initiative is the iREHAB project at Pontificia Universidad Javeriana, which proposes a multi-articular robotic exoskeleton supported by analytical and monitoring tools within a unified rehabilitation architecture (Colorado et al., 2022).
Despite these advancements, several challenges remain, including high structural complexity, limited portability, and significant implementation costs, which can hinder large-scale clinical deployment. On the other hand, while improvements in motor involvement and short-term functional outcomes have been reported (Naro and Calabrò, 2021; Díaz-Sáez et al., 2022), standardized validation protocols and long-term studies on persistent recovery are still limited. These limitations emphasize the need for scalable, interoperable, and clinically validated architectures that can effectively integrate immersive environments with robotic assistance technologies.
METHODOLOGY
General System Architecture The proposed solution is a MR system designed to support upper-limb rehabilitation by combining the Meta Quest 3S headset with an optional robotic exoskeleton. As shown in Fig. 1, the architecture integrates three main actors: the user (blue header), the MR application running on the Meta Quest 3S (green area), and the robotic exoskeleton (red area).
Figure 1: High-level architecture. It includes: (i) UI for mode selection and onboarding, (ii) Gesture recognition and an interactive rehab game, (iii) BLE communication with the exo, and (iv) Real-time synch with a digital twin.
A therapy session begins when the user puts on the headset and interacts with the interface located in the light-green user-interface block. From this menu, the user selects one of two modes: (i) Assisted therapy through the exoskeleton (red block), or (ii) Game-based therapy handled entirely inside the MR environment (medium-green block). In assisted-therapy mode, the system launches a guided onboarding tutorial contained within the light- green UI area. After the user completes the setup, movement commands are transmitted via BLE to the exoskeleton’s microcontroller (cream sub-block inside the red area), while the digital-twin module (pale-green block) updates in real time to reflect actuator and joint motion.
In the game-based mode, the user interacts with the rehabilitation game highlighted in the medium-green section, which trains the same bidigital pinch movement through ADL-inspired tasks. Gesture recognition, also represented in this medium-green game block, enables the user to perform all required actions without the physical exoskeleton, supported by a short in-game tutorial.
Overall, the system integrates four subsystems inside the green Meta Quest 3S block: the digital twin (pale green), the BLE communication layer interfacing with the red exoskeleton block, the recognition module embedded in the light-green UI block, and the rehabilitation game (medium green).
Exoeskeleton Digital Twin For the design of the digital twin, the process was based on CAD models of the exoskeleton imported into Unity to define the movement system. Linear actuators were defined as the main elements driving the mechanism, with
Mixed Reality–Driven Rehabilitation Using a Robotic Exoskeleton and an Immersive 327 rigid bases and rods performing linear motion for extension and retraction, linked via a Bluetooth Low Energy (BLE) protocol to synchronize the digital twin with the physical device. Once the actuator behavior was defined, the remaining components were configured following the kinematic chain of the mechanism. The proximal segment was first implemented with a defined pivot point for its rotational axis, followed by the medial segment, which inherits the proximal motion while incorporating its own local rotation with independent limits to match physical constraints. Finally, the distal segment was integrated in the same manner, inheriting motion from the medial segment while defining its corresponding rotational limits to accurately replicate the exoskeleton finger mechanism.
Once the kinematic model was established, the accuracy of the digital twin was validated by comparing its motion with the real movement of the exoskeleton. To achieve this, a biomechanical analysis was performed for both systems: the physical exoskeleton and its digital counterpart. The digital twin model used for this comparison is illustrated in Fig. 2, where the articulated finger mechanism and its kinematic structure implemented in Unity can be observed. The movements of both elements were recorded and analyzed to evaluate the fidelity of the digital representation.
Figure 2: Digital twin model of the hand exoskeleton implemented in Unity, showing the articulated finger mechanism and its kinematic structure.
For the validation process, a comparison was conducted for each finger considering the following kinematic parameters: position, velocity, and acceleration. These metrics allowed the evaluation of the dynamic behavior of the digital twin in relation to the physical system. Additionally, a minimum acceptable similarity threshold was established to ensure that the digital twin accurately replicated the real movement of the exoskeleton and that the system functioned as intended. Fig. 3 presents an example of this comparison for the index finger.
Figure 3: Kinematic analysis of the index finger during an exoskeleton-assisted rehabilitation routine.
HMD-Exoskeleton Bidirectional Communication Based on the exoskeleton design and the standalone communication capabilities of the Meta Quest 3S HMD, the BLE protocol was selected as the primary communication channel. In wearable devices, wireless communication technologies are commonly used to transmit data and enable interaction with external control or monitoring systems. As described in (Roshdy et al., 2019), among the most widely used technologies are Wi-Fi, Bluetooth, and ZigBee, with Bluetooth standing out due to its low power consumption and its suitability for portable and wearable devices such as exoskeletons. As short-range communication is sufficient for this application, Bluetooth is selected as the communication technology.
As a primary outcome, this module enables coordinated interaction between the virtual and physical environment, such as digital twin feedback and exoskeleton routines execution, thereby improving overall system performance and user experience.
Figure 4: BLE hardware connection scheme. The interaction is implemented through five messages enabling (i) the establishment of basic device communication, (ii) the execution of exoskeleton routines, and (iii) the reception of individual actuator data on the VR headset.
Essential communication routines are necessary for the correct functioning of the BLE module, as shown in Fig. 4. To enable this capability, a dedicated software library was initially created to control the headset’s Bluetooth interface. This library is integrated into the Unity environment and managed through three orchestration scripts to ensure smooth coordination between the communication layer and the user interface. Fig. 5 illustrates the high-level architecture of the communication subsystem, highlighting the integration of the Android-based BLE implementation with the Unity application.
Figure 5: BLE software architecture. BLE functionalities were implemented using Android Studio and integrated with Unity to provide a customized control framework for user-interface interactions.
Mixed Reality–Driven Rehabilitation Using a Robotic Exoskeleton and an Immersive 329 Exoskeleton Recognition System
A real-time recognition module was created to identify the exoskeleton parts inside the MR environment and to guide the user through the correct setup procedure. The system detects the Hand, Micro, and Button parts using a lightweight YOLO-based model (Jiang et al., 2022) deployed on-device in the Meta Quest 3S, triggering instructions and synchronizing the kinematic digital twin.
The recognition in Fig. 6 includes passthrough acquisition, preprocessing, YOLO inference via Unity Sentis, and post-processing (NMS and confidence/ IoU thresholds). The results are organized by two layers: (i) guided user interface and (ii) the kinematic digital twin for spatial consistency in MR.
Figure 6: Software architecture for the exoskeleton-recognition pipeline. The diagram shows the four-stage workflow: (i) data collection and annotation, (ii) YOLO model training and validation, (iii) ONNX export and integration into Unity via Sentis, and (iv) real-time deployment and testing on the Meta Quest 3S.
A dataset of 370 images was captured and labelled to train the model, prioritizing variability in viewpoint and lighting. The trained dataset was exported to ONNX and integrated into Unity Sentis for real-time execution (ONNX, 2026) (Unity, 2026). The detection overlays are processed directly in the MR scene and used to drive the interactive guidance process. An example of detection can be seen in Fig. 7. Detection events trigger guidance panels, instructions, animations, and confirmation prompts. The digital twin updates its pose from recognized components, reinforcing spatial understanding during the setup.
Figure 7: Example of a real-time detection of hand, micro, and button in MR.
A three-phase protocol validated the model during training, after ONNX export, and during real-time execution. In-headset evaluation across 20 frames showed reliable detection of Hand and Micro (100% recall), with IoU
values of 0.714 and 0.466 respectively. As expected for a small, embedded component, the Button reached lower IoU (0.042). Overall class-level agreement reached 70%. The results are summarized in Table 1.
Table 1: Summary of real-time exoskeleton-recognition performance on meta quest 3S.
IoU (global, TP only) IoU Hand IoU Micro IoU Button
0.474 0.714 0.466 0.042
Recall Hand Recall Micro Recall Button FP Button
100% 100% 66.7% 0%
In general, the recognition module operates stably in real time in the MR environment, providing reliable detection for Hand and Micro and enabling guided interaction during rehab tasks.
Interactive Game The game levels were designed with a grip detection system for ingredient interaction based on specific hand gestures, implemented through two methods. The first uses Meta’s OVRHand functions in Unity to recognize predefined gestures, such as a closed fist and an index pinch. The second employs a custom script that computes finger pinch strength to classify the hand as open or closed, enabling interaction when values fall within a defined threshold. Both methods incorporate a proximity constraint, requiring the hand to be within a set distance from the ingredient to allow interaction, promoting more realistic behavior in the virtual environment.
The game includes two levels. In the first, users prepare a hamburger by placing ingredients in the correct sequence, each requiring a specific gesture. The second involves preparing a milkshake by placing ingredients into a virtual blender, activating it, and completing the remaining steps using gesture-based interactions. In both levels, performance is evaluated based on task completion time.
RESULTS
Once the system design was completed a validation phase was done to confirm that it operated as expected and remained consistent with the initial objectives. The primary goal at this stage was to verify that the system performed correctly when used by users. To achieve this, several testing sessions were organized with healthy volunteers, who interacted directly with the system and assessed their experience using standardized questionnaires.
Mixed Reality–Driven Rehabilitation Using a Robotic Exoskeleton and an Immersive 331 This process made it possible to evaluate the system’s performance in terms of usability, comfort, workload, and overall user perception.
For this validation phase, the System Usability Scale (SUS), Virtual Reality Sickness Questionnaire (VRSQ), and NASA Task Load Index (NASA- TLX) were used to assess usability and perceived workload. The results are presented below.
Usability Questionnaire Results The results obtained from the applied questionnaires (SUS, VRSUQ) provide a comprehensive assessment of system usability and user comfort. The System Usability Scale (SUS), applied to 22 participants, yielded an average score of 81.25, categorized as highly acceptable and approaching excellent. The distribution included 7 “excellent” ratings (> 85), 12 “acceptable” (65–84), and 3 “not acceptable” (< 65). The highest-rated aspects were ease of use (Q3), functional integration (Q5), learnability (Q7), and user confidence (Q9), with most responses between 4 and 5. Lower scores were associated with perceived complexity (Q2), the need for initial assistance (Q4), and the effort required to learn the system (Q10), indicating that onboarding and interface clarity could be improved. Overall, these results suggest that the system is highly usable and dependable, with minor refinements needed to enhance the initial user experience. These results can be observed on Fig. 8a.
Figure 8: SUS and VRSUQ scores obtained from each participant during the evaluation of the proposed mixed reality rehabilitation system.
In parallel, Fig. 8b presents results of Virtual Reality System Usability Questionnaire (VRSUQ). This questionnaire produced an average score of 81.06, also classified as high–acceptable. Eleven participants scored in the “excellent” range (≥ 85), 8 in the “acceptable” range (65–84), and 3 in the “not acceptable” range (< 65). The best-rated aspects included system responsiveness (Q1), clarity of information (Q4), ease of use and learning (Q5), error recovery (Q6), and overall enjoyment (Q7). Additionally, reversed items indicated a low presence of dizziness or cybersickness (Q8) and low perceived mental workload or time pressure (Q9), supporting the comfort of the immersive experience. Areas for improvement were mainly related to reducing interaction errors (Q3) and improving consistency in feedback and perceived latency (Q1–Q2).
Workload Q Results: The overall workload score was 46.6/100, placing the task within a manageable range. The dimensions contributing most to workload were Performance (23%) and Effort (22%), indicating that users felt motivated to perform well and needed to invest a certain level of effort to achieve the task objectives.
Physical demand (14%) and Frustration (14%) remained at low-to- moderate levels, suggesting that the required gestures were not physically exhausting and that discomfort was occasional rather than persistent. Mental demand (12%) was the lowest dimension, reflecting clear rules and intuitive task flow, while Temporal demand (15%) was moderate, with participants perceiving the timing as reasonable.
Overall, the system was classified as usable without overloading participants, although there is still room to reduce perceived effort and performance-related pressure.
CONCLUSION
This work presented a MR rehabilitation system for the Meta Quest 3S integrating exoskeleton recognition using ONNX/Sentis, bidirectional BLE communication, a kinematic digital twin, and an interactive therapeutic game. Validation with 22 participants showed positive results in usability, comfort, and workload.
The SUS obtained an average score of 81.25, indicating highly acceptable usability. The VRSQ achieved an average of 81.06, suggesting low levels of cybersickness and good visual comfort during the immersive interaction. The NASA-TLX work-load score was 46.6/100, reflecting a manageable task demand with low mental load.
In conclusion, the results indicate that the proposed system is usable and feasible as an interactive rehabilitation platform. Future work will focus on expanding the computer-vision dataset, improving the digital twin for motion analysis, refining the exoskeleton hardware, and conducting clinical evaluations with post-stroke patients.
ACKNOWLEDGMENT
This work was funded by the project “iREHAB: Sistema inteligente de Rehabilitación usando un Exoesqueleto para recuperar Habilidad motora en discapacidades post-ACV, usando se˜nales Biol´ogicas del paciente” sponsored by The Ministry of Science Technology and Innovation (MinCiencias), program 918-2022 under GRANT CTO: 622-2022, Award ID: 91805.
REFERENCES
Colorado, J. D., Alvarado-Rojas, C., Méndez, D., Bernal, I. F. M., Bautista, A. D.
G. and Garcia-Cifuentes, E. (2022). Irehab: Sistema inteligente de rehabilitación usando un exoesqueleto para recuperar habilidad motora en discapacidades post-ACV, usando señales biológicas del paciente. Website: https://osf.io/br37t/ (Accessed: 18 March 2025).
Mixed Reality–Driven Rehabilitation Using a Robotic Exoskeleton and an Immersive 333 Díaz-Sáez, M. C., Navarro-Fernández, G., Fernandez-Carnero, J., Garrigós-Pedrón,
M., Muñoz, J. P. R. and Beltran-Alacreu, H. (2022). Conventional cervical exercises compared with a mixed-reality-based game in asymptomatic subjects: An exploratory crossover pilot study. Applied Sciences, 12(7), 3657. Website: https://doi.org/10.3390/app12073657 Falkowski, P. and Gwardecki, F. (2024). Exoreha exoskeleton digital twin in metaverse for telerehabilitation. In: 2024 13th International Workshop on Metaverse and Robotics (IWMR). Website at: https://ieeexplore.ieee.org/document/10604375 Jiang, P., Ergu, D., Liu, F., Cai, Y. and Ma, B. (2022). A review of YOLO algorithm
developments. Procedia Computer Science, 199, 1066–1073. Website: https://doi. org/10.1016/j.procs.2022.01.135 Langhorne, P., Bernhardt, J. and Kwakkel, G. (2011). Stroke rehabilitation. The Lancet, 377(9778), 1693–1702. Mondragón B., I. F. et al. (2025). Advancements in hand function rehabilitation
for stroke patients: A scoping review of immersive virtual, augmented, and mixed reality technologies and an application study. In: Physical Therapy and Rehabilitation -Research and Clinical Application. IntechOpen. Naro, A. and Calabrò, R. S. (2021). What do we know about the use of virtual reality in the rehabilitation field? A brief overview. Electronics, 10(9), 1042. ONNX (2026). ONNX documentation: Key concepts. Website: https://onnx.ai/
onnx/intro/concepts.html (Accessed: 6 February 2026). Roshdy, A., Al Kork, S., Said, S. and Beyrouthy, T. (2019). A wearable exoskeleton rehabilitation device for paralysis – a comprehensive study. Advances in Science, Technology and Engineering Systems Journal, 4(1), 17–26. Unity (2026). Unity Sentis: Model execution concepts. Website: https://docs.unity3d.
com/Packages/com.unity.sentis@2.0/manual/models-concept.html (Accessed: 6 February 2026). Weng, M. and contributors (2024). BiDexHand: Dexterous robotic hand with immersive interaction. Website: https://github.com/wengmister/BiDexHand (Accessed: 26 February 2026). Yao, M., Wang, M., Ma, Y., Busse, J. W., Hu, X., Liu, Y., Luo, X., Liang, Q., Liang,
X., Zou, K., Li, L. and Sun, X. (2026). Projections of the global, regional and national stroke burden by 2050: A systematic analysis for the Global Burden of Disease Study 2021. European Journal of Epidemiology.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Interaction and Emerging Technologies (IHIET-FS 2026): Future Systems and Design Applications
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 979-8-950676-11-6
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional