Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Evaluating a Refined Augmented Reality Eye-Gaze and Voice Control System for Electric Wheelchairs: A User Study

Hochschule Bremen – University of Applied Sciences, Bremen, 28199, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1006716

Abstract

Self-determined mobility is fundamental for social participation, yet conventional electric wheelchair controls present significant usability challenges for individuals with severe motor impairments like tetraplegia or ALS. While alternative specialized controls exist, they may introduce issues such as fatigue or reduced intuitiveness. This paper reports on the user evaluation of a refined assistive control system, previously introduced as work-in-progress, designed to address these limitations. The system employs a combination of eye-gaze tracking and voice commands, mediated through a Magic Leap 2 Augmented Reality (AR) headset, to operate both a wheelchair simulator and a real electric wheelchair via a Raspberry Pi interface. Initial prototypes highlighted the need for enhancements focused on user experience, control stability, and intentionality, particularly for real-world deployment. The evaluated system incorporates several key refinements: (1) A robust activation mechanism requiring sustained gaze within the interface boundaries for a defined duration to enable control, complemented by a configurable grace period upon gaze exit to prevent unintended deactivation during brief glances away. (2) Advanced stability features, including Kalman filtering or iterative Slerp smoothing of gaze rotation data to mitigate input jitter, alongside head rotation handling that detects high angular velocity to trigger a brief, timed recentering of the joystick input upon cessation, preventing uncontrolled continuation of turns. (3) Fine-tuned control mapping, utilizing non-linear response curves, minimal dead zones, sensitivity scaling, and output signal rate limiting to ensure smooth command delivery, provide fine control near the center, and prevent overly sensitive physical responses. (4) Raycast-based interaction logic for axis-snapping guide cubes and focusable object detection, replacing previous methods.The primary objective of this study was to comprehensively evaluate the usability, effectiveness, perceived workload, and user acceptance of this refined control system. The evaluation was conducted with individuals with spinal cord injuries at the BG Klinikum Hamburg. We aimed to assess the system's potential as a viable supplement or alternative to existing specialized wheelchair controls.A two-phase methodology was employed: participants first engaged with the system in a VR wheelchair simulator for familiarization and baseline assessment, followed by operating a real Ottobock Juvo B5 wheelchair using the AR interface in a controlled clinical environment. Tasks included fundamental maneuvers (activation/deactivation, forward/backward driving, turning in place), navigation along a simple marked path, and precision approach tasks. Data collection utilized a mixed-methods approach, combining objective performance metrics (e.g., task time, errors), standardized subjective questionnaires (System Usability Scale - SUS; NASA Task Load Index - NASA-TLX), and qualitative feedback via semi-structured interviews and direct observation, adhering to approved ethical guidelines.This paper presents the detailed findings from these user tests, focusing on the performance differences between VR and AR conditions, the perceived usability and workload associated with the system, the effectiveness of the specific activation and stability refinements, and overall user acceptance. The results provide empirical insights into the practical application of AR-based eye-gaze and voice control for wheelchairs, identifying strengths, limitations, and crucial areas for future development to enhance autonomous mobility for individuals with severe motor impairments.

© 2025. Published by AHFE Open Access. All rights reserved. 230

Evaluating a Refined Augmented Reality Eye-Gaze and Voice Control System 231

INTRODUCTION

Operating conventional power wheelchairs often present a significant challenge (Frank et al., 2016). While joystick control is the most intuitive method for many users, limitations in arm and hand function often make it unusable. Existing alternative special controls, such as chin or head controls, can lead to fatigue and pain with prolonged use and often require lengthy learning periods (Williams et al., 2015). These systems also reach their limits in progressive diseases such as muscular dystrophy or ALS (Beukelman et al., 2011). Therefore, there is a need for flexible, adaptive, and intuitive control concepts tailored to the individual abilities and needs of the user.

Presented Solution and Preliminary Work In response to these challenges, preliminary work was carried out to develop a prototype system for controlling electric wheelchairs using eye and voice commands (Bulk, 2025). This system is based on the augmented reality (AR) headset Magic Leap 2 (ML2) that uses eye tracking to detect the user’s gaze direction and control a virtual joystick UI in the sight of the user. Voice commands enable additional functions such as turning the joystick on and off, setting the maximum speed and controlling the lights and horn. These functions are also accessible by eye-controlled buttons (see Figure 1) for people who are unable or don’t want to use voice commands. The initial development demonstrated its basic feasibility, but also the need for significant further developments in terms of stability, user-friendliness, and safety for practical use, especially compared to pure simulation. This work focuses on the evaluation of a correspondingly further developed system through user tests with the target group and expert interviews with occupational therapists and wheelchair technicians.

Figure 1: Eye-joystick interface. The blue sphere is the outer border for gaze recognition. The green ball is the controller of the joystick. The inner lines are helping with the gaze aiming.
Original PDF Figure 1:

232 Bulk and Tannert

Related Work

The use of eye tracking as an input method in human-computer interaction is an established field of research (Majaranta et al., 2014), with applications in medical technology (Sqalli et al., 2023) and assistive technologies, among other Challenges often lie in the accuracy of gaze tracking and the handling of natural eye movements such as micro saccades. Various filtering techniques, such as the Simple Moving Average (SMA), used in the initial prototype, or more advanced methods are used for stabilization.

Voice control offers a complementary, hands-free interaction option that is particularly relevant for people with motor disabilities.

In the field of wheelchair control, various approaches exist beyond the standard joystick. Head and chin controls are common but have the aforementioned disadvantages. Tongue controls have also been successfully demonstrated. Direct eye control for wheelchairs is being researched (Araujo et al., 2020) but separating gaze for navigation from gaze for control presents a challenge. Our approach, using a virtual joystick in the AR display, attempts to facilitate this separation.

Objective of This Paper This paper presents the results of a user study evaluating the enhanced eye- voice control system. The main objectives are to assess usability, perceived workload, effectiveness in performing typical driving tasks, and general user acceptance among people with spinal cord injuries. The study examines the practical suitability of the system as a potential complement or alternative to existing special control systems.

FURTHER DEVELOPMENT OF THE CONTROL SYSTEM Based on the findings from previous work (Bulk, 2025) and the specific requirements of real-time wheelchair control, several core areas of the system were revised and expanded to improve stability, intuitiveness, and safety. The technological basis (Magic Leap 2, Unity, OpenXR, OpenVR, Raspberry Pi, ZeroMQ/USB) was largely retained, although the software underwent significant refactoring to increase modularity and maintainability and some design improvements.

For the training phase, the control system was implemented directly into the previously developed wheelchair simulator (Bulk, 2025) to create a coherent user experience. This enabled the use of an eye-tracking VR headset (Pico 4 Enterprise), thus avoiding visual fixation problems that occur when using AR glasses and an external monitor simultaneously. The implementation of the eye-controlled joystick in VR is completely identical to that used on the ML2.

In the following, more improvements to the system will be made.

Activation Mechanism To prevent unintentional control, the system uses a two-stage activation mechanism. The user must maintain gaze within the UI sphere for a defined duration to activate the joystick, which is visually confirmed by a color

Evaluating a Refined Augmented Reality Eye-Gaze and Voice Control System 233

change. If the gaze leaves the sphere, joystick output is immediately zeroed, but control can be instantly resumed if the gaze returns before a short period expires.

Some users noted that the joystick and the speed information it contained were distracting. Therefore, it is now possible to toggle the visibility of the joystick during runtime. A cursor shows the user’s gaze position and can instead be used for aiming within the sphere. Figure 1 shows the joystick with active controller.

Gaze Smoothing To reduce the jitter in the controls and visual feedback that can arise from unstable raw data from the eye tracker, the recorded gaze rotation is smoothed. Instead of a simple SMA filter, an iterative spherical linear interpolation (Slerp) is now used. A parameter controls the strength of the smoothing.

Handling Head Turns To prevent uncontrolled continued rotation after a head movement, the system detects rapid head rotations via angular velocity. When such a rotation ends, a short timer is triggered. During this “braking phase,” gaze input is ignored, and the joystick output is forced to zero. This gives the user time to stabilize their gaze before control is released again.

Control Mapping and Output Processing The translation of the local joystick position into the control signal sent to the wheelchair has been revised to reduce “stuttering” and improve fine control:

Non-linear curve: A power function is applied to the X-axis (rotation). This makes the area near the center less sensitive.

Deadzone: A small deadzone around the center ensures that minimal jitters do not produce any output.

Sensitivity: Global scaling optionally limits the maximum output speed. Rate Limiting: It limits the maximum change rate of the sent command, so that rapid jumps in the calculated target result in a smooth change in the actual output.

Web Interface All variables described above, as well as the button activation time, can be configured via a web interface, provided by the RasPi via WiFi, during and outside of runtime.

In addition to the pure control and fine-tuning variables, it is also possible to customize voice commands, turn the joystick on and off, and activate gamepad control via the web interface.

Further Add-ons To allow the test operator to take control of the wheelchair and lock ML2 inputs at any time, the gamepad control can be activated.

In both applications, VR and AR, a rear-view camera was implemented. The view will be projected on a semi-transparent surface behind the joystick UI, in the sight of the user perspective.

234 Bulk and Tannert

USER TESTING METHOLOGY The goal of the user tests is to comprehensively evaluate the enhanced system in terms of usability (SUS), workload (NASA-TLX), effectiveness (performance metrics), perceived safety and user acceptance, comparing VR simulation and AR real-world applications.

Participants were recruited from the patient population of the Paraplegic center at the BG Klinikum Hamburg. This includes individuals with high spinal cord injuries (tetraplegia) or comparable motor impairments who are eligible for alternative wheelchair control and who are cognitively capable of understanding the tasks and providing feedback.

Study Design and Procedure A two-phase within-subjects design is used:

Phase 1, VR Simulation: Participants initially use the system in a VR environment with the previously developed wheelchair simulator. This serves to ensure safe familiarization with the controls, learn the activation and interaction mechanisms, and collect basic data on performance and subjective perception.

Phase 2, AR Real World: After sufficient familiarization, participants use the Magic Leap 2 to control a real electric wheelchair (Ottobock Juvo B51), in a secure area of the clinic (therapy room).

In both phases, participants complete a series of standardized driving tasks.

Driving Tasks The tasks increase in difficulty and cover essential aspects of wheelchair use:

1. Basics: Activating/deactivating the system based on gaze duration; Driving straight ahead and backward over a defined distance (approx. 5–7 m) with a precise stop at a marker; turning 180 degrees on the spot. 2. Course navigation: Driving along a simple, marked route (e.g., L- or U-shaped) with 90-degree turns and a defined width (approx. 1.5–2 m). 3. Precision approach: Slowly approaching a target object with a precise stop just before it, followed by a 90-degree turn as precisely as possible.

Data Collection A mixed-methods approach is used.

Quantitative performance data: Time required to complete tasks, number of errors (e.g., collisions, line crossings, unintentional activations/deactivations), path deviation (possibly through tracking), stopping accuracy (distance measurement).

1Juvo B5 product page: https://www.ottobock.com/de-de/product/490E75_B5B6, last visit 19.06.2025.

Evaluating a Refined Augmented Reality Eye-Gaze and Voice Control System 235

Subjective assessments: System Usability Scale (SUS): Captures perceived usability after each phase or task. NASA Task Load Index (NASA-TLX): Measures perceived workload (mental, physical, time, effort, performance, frustration) after each task.

Qualitative data: Direct observation:Experimenter’s notes on strategies, difficulties, comments, and nonverbal reactions.

Semi-structured interviews: After completing the VR and AR phases, detailed feedback is collected on specific functions (activation, stability, axis snap, voice commands), comfort, perception of safety.

Both phases take place in a barrier-free, low-disturbance environment. The maximum speed of the real wheelchair is initially limited. The test leader is always present and can intervene if necessary.

EVALUATION OF USER TEST DATA The analysis of data from five participants in one VR and one AR condition shows clear and consistent trends:

Performance (objectively): The controls in the AR condition are objectively superior to the real wheelchair. On average, participants were faster and made significantly fewer errors (approximately 65% fewer unintentional stops and almost 90% fewer path deviations). Accuracy in stopping and turning maneuvers was also significantly higher in AR.

Table 1: Performance metrics (average totals per participant per condition): shows that participants in AR were on average faster and more error-free.

Table available in the original PDF paper.

Std Total

Mean Std Total

Cond. Mean

Std Total Unintended

Total

Path Deviations

Mean Total Path Deviations Stops

Total Unintended

Task Time in s Stops Task Time in s

AR 121.7 14.86 7.6 3.21 0.4 0.55 VR 142.8 24.45 19.4 7.13 4.2 3.63 Usability (subjectively): Both systems are rated as “good” and above average in terms of user-friendliness, with slight advantages for the AR version (SUS score: 74.9 vs. 73.4). However, the difference is small.

Table 2: Subjective questionnaires (mean values per condition): high and similar usability scores. Slightly higher, but also similar workload scores in AR.

Table available in the original PDF paper.

Condition Mean SUS Score

Std SUS Score Mean NASA TLX Overall

Std NASA TLX Overall AR 74.9 13.92 6.50 3.17 VR 73.4 16.44 6.27 4.11

Workload: The perceived overall workload (NASA-TLX) is low and very similar in both conditions (AR: 6.5 vs. VR: 6.3). However, Figure 2 with a

236 Bulk and Tannert

detailed analysis shows that users are more critical of their own performance in AR (higher value for Performance), even though they were objectively better. This indicates greater mental focus and responsibility in the real-world environment.

Figure 2: Detailed NASA-TLX subscales (means per condition): The scores are very similar across most dimensions, except for “performance,” where users rated their performance in VR as better (lower score is better).
Original PDF Figure 2:

Quantitative Feedback: The comments and observations (Figure 3) support the quantitative data. The main problem area in both conditions is stability during turns (“problems with stable turn”, “The smooth turn is difficult”). The AR condition requires more concentration on the environment (“hard to keep track of the environment”, “very high concentration”). Driving by axle fixation proved to be effective, which is reflected in the high ratings (AR average: 4.8/5).

Figure 3: Accuracy & rating metrics (averages per condition): shows higher precision (lower accuracy scores) and better subjective evaluation of features in AR.
Original PDF Figure 3:

RESULT OF THE QUALITATIVE STUDY Turning Challenge: The most consistent theme in both conditions was the difficulty of performing stable and smooth turns. Comments such as

Evaluating a Refined Augmented Reality Eye-Gaze and Voice Control System 237

“problems with stable turns,” “stopping the turn precisely is hard,” and “the smooth turn is difficult” appeared repeatedly. This is the primary area for further improvement.

AR vs. VR Perception: The AR condition appears to require more concentration (“very highly concentrated”), as the real environment must be observed (“hard to keep track of the environment”). This explains why users viewed their performance in AR more critically, even though it was objectively better. The pressure to avoid making mistakes is greater in real space.

Positive Feedback on Features: Driving straight ahead, especially with the axis fixation, was repeatedly described as “very stable” and “smooth.” Axis functionality (1–5) in Figure 3 confirms this.

Learning curve: One participant noted that a technically skilled person would be helpful for the initial use, but that the controls can then be learned independently (“a technically skilled person is necessary for the initial use, but not later”).

EXPERT INTERVIEW WITH ERGO THERAPISTS Following the practical tests, semi-structured expert interviews were conducted with the occupational therapists who are responsible for e.g. teaching the handling of wheelchairs. We captured their professional assessment of the system’s potential and its integration into everyday therapy. The discussions yielded a consistent and positive picture.

The therapists rated the option of virtual training as extremely useful. They emphasized that becoming accustomed to a new, complex control system such as this requires dedicated training. Conducting this training in a safe, virtual environment was considered a significant advantage, as it reduces the need for constant one-on-one supervision and allows patients to practice at their own pace. In particular, the ability of the underlying simulator to virtually recreate the patient’s home environment was perceived as an outstanding feature for everyday-oriented and targeted training. A central point of discussion was the comparison with established special control systems. The therapists estimated the training effort for eye control to be roughly the same or even less intensive than for chin control. They identified two major disadvantages of chin control for which eye control offers a direct solution: First, the frequent slipping of the control element on sudden uneven floors, which can lead to loss of control, and second, the often unergonomic, forced head position, which leads to tension and pain for many users. Eye control eliminates both problems.

The multimodal interaction enabled by voice control was particularly highlighted. The ability to perform discrete actions, such as turning the joystick on or off, via voice command was praised as a key feature for fast and natural operation, as it frees the viewer to focus on its primary task. In the context of everyday therapy, it was also mentioned that the gamepad input method can be useful for certain training scenarios.

In summary, the occupational therapists assessed the presented system as a very promising alternative to established special controls such as chin and mouth control. In particular, the solution to specific problems of existing

238 Bulk and Tannert

systems while maintaining intuitive operation was seen as a major potential benefit for the mobility and quality of life of future users.

EXPERT INTERVIEW WITH OTTO BOCK EMPLOYEES A specialist team from Ottobock, a company who manufactures electric wheelchairs, consisting of an experienced wheelchair technician, field service representatives, and an expert for special controls, provided valuable feedback from a technical and practical perspective following a system presentation. The key findings were:

Practicality and innovation: The control of core functions such as seat adjustment (tilt) and the use of voice commands for secondary actions (e.g., light, activation) were rated as very practical and relevant to everyday use. The combination of eye and voice input was highlighted as an innovative and rare approach in the special controls market.

Simulator as a unique selling point: The system’s connection to a VR training simulator was seen as a significant advantage over existing solutions. The idea of scanning users’ homes via smartphone and uploading them to the simulator for personalized training was proposed.

Future potential: The experts identified significant potential for enhancements. This includes the integration of various wheelchair drive types into the simulator and, most importantly, the addition of voice output for non-speaking users, a feature that is currently in high demand.

Cost-effectiveness: Based on the components used, the system was assessed as potentially significantly more cost-effective than comparable, established special control systems, which could significantly increase its market relevance.

In summary, the expert feedback confirmed the system’s high potential as a promising and cost-effective alternative in the field of special control systems.

CONCLUSION

The user study with people with spinal cord injuries demonstrated that the enhanced AR eye and voice control system represents a functional and highly usable alternative to real-world special wheelchair control control systems. Participants’ objective performance was significantly better in the real-world AR condition (faster, more precise, fewer errors) than in the VR simulation, with the system achieving good usability scores (SUS score > 73) in both contexts.

This potential was confirmed by experts (occupational therapists and wheelchair technicians), who particularly emphasized the solution to specific disadvantages of existing special controls (e.g., posture problems with chin control) and the system’s high cost-effectiveness. The greatest challenge and the key area for future technical improvements remains the stabilization of fluid rotational movements. Building on the positive results, particularly the axis fixation, which was rated as very helpful, the system offers a promising approach to improving autonomous mobility for people with severe motor impairments.

Evaluating a Refined Augmented Reality Eye-Gaze and Voice Control System 239

REFERENCES

Araujo, J., Zhang, G., Hansen, J., Puthusserypady, S. (2020). Exploring Eye- Gaze Wheelchair Control. In ACM Symposium on Eye Tracking Research and Applications (ETRA ‘20 Adjunct). Association for Computing Machinery, New York, NY, USA, Article 16, 1–8. https://doi.org/10.1145/3379157.3388933 Beukelman, D. R., Fager, S., Nordness, A. (2011) Communication support for

people with ALS. Neurology Research International. https://doi.org/10.1155/ 2011/714693 Bulk, J. (2025). An eye-controlled joystick with voice control for operating a simulated and real electric wheelchair. Extended Reality, International Conference, XRSolento 2025, Otranto, Italy, June 17–20, Proceedings, Part III, Volume LNCS 15739, Springer, pp. 59–69. Bulk, J., Tannert, B. (2024). Development of a Training Simulator and Testing

Environment for Electric Wheelchair Controls. In: Tareq Ahram and Waldemar Karwowski (eds) Human Factors in Design, Engineering, and Computing. AHFE (2024) International Conference. AHFE Open Access, Volume 159. AHFE International, USA. http://doi.org/10.54941/ahfe1005712 Frank, A. and Souza, L. (2016). Problematic clinical features of children and adults with cerebral palsy who use electric powered indoor/outdoor wheelchairs: A cross- sectional study. Assistive Technology, 29(2), pp. 68–75. Majaranta, P., Bulling, A. (2014). Eye Tracking and Eye-Based Human–Computer

Interaction. In: Advances in Physiological Computing. Human–Computer Interaction Series. Springer, London, pp. 39–65. Sqalli, M. T., Aslonov, B., Gafurov, M., Mukhammadiev, N., Houssaini, Y. S.

(2023). Eye tracking technology in medical practice: A perspective on its diverse applications. Frontiers in Medical Technology, p. 5. Williams, M. and Kirsch, R. (2015). Evaluation of head orientation and neck muscle EMG signals as three-dimensional command sources. Journal of Neuroengineering and Rehabilitation, 12(1), p. 25.

Cover of the proceedings volume Human Interaction and Emerging Technologies (IHIET 2025)

Explore the complete digital volume

Human Interaction and Emerging Technologies (IHIET 2025)

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-73-1

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →