Romy Alm, Simon König, Claus Backhaus (2026). Usability evaluation of a MR system for remote support during ECMO within clinical environment. In: Usability and User Experience. AHFE Open Access, vol. 232, pp. 242–248. https://doi.org/10.54941/ahfe1008052
Intensive care treatment entails significant risks for patient health. For troubleshooting extracorporeal membrane oxygenation (ECMO), an essential intervention for supporting cardiac and pulmonary function, often perfusionists must be present on-site. This causes delays in treatment. Mixed reality (MR)–based remote support may help bridge these delays; however, high system usability is essential to ensure its contribution to patient safety. Therefore, an MR system for remote support is evaluated within a clinical environment through a usability test followed by a user survey. During the test, participants resolved two simulated ECMO malfunctions while receiving remote guidance via the MR system. A test manager assessed the subtasks and recorded processing times. Finally, user acceptance and experience were evaluated using the System Usability Scale (SUS) and User Experience Questionnaire (UEQ). The usability test was performed in two U.S. hospitals involving 19 intensive care nurses. In both scenarios high troubleshooting performance was observed: success rates reached 90% each. Average solution times were 4:10 (±1:16) minutes and 3:32 (±0:44) minutes. The system reached a SUS score of 83±12, indicating good usability. Consistent with this result, UEQ measured excellent scores in every usability and user experience aspect. These findings demonstrate that MR-based remote support can effectively assist intensive care nurses in resolving ECMO incidents, which may contribute to improved patient safety. However, the simulated conditions of the usability tests and the novelty of the technology may have positively biased ratings. Future research should investigate performance during real emergency use and evaluate long-term acceptance after repeated exposure.
SITUATION Intensive care treatment entails risks, including incidents that may significantly impact patient health. Extracorporeal membrane oxygenation (ECMO) is an essential intervention for supporting cardiac and pulmonary function. Complications need perfusionists often to be present on-site to troubleshoot. Since continuous handling of the device is not necessary, perfusionists are
Usability Evaluation of a MR System for Remote Support During ECMO 243 often at home on call. Due to the physical distance, this results in waiting times that delay treatment (Al Disi et al., 2019).
Mixed reality (MR)–based remote support may help bridge these delays by providing immediate audiovisual guidance. MR expands the real world by virtual content via a display. Especially MR-glasses such as the HoloLens 2 (HL2) developed by Microsoft Corp. (Redmond, WA, USA) are frequently used, as they ensure hands-free operation via gesture and voice control. This is particularly beneficial for continuous interaction with the environment.
Remote assistance with MR is already used in various industrial sectors, e.g. during maintenance and assembly work (Miranda et al., 2025; Solomashenko, 2025; Vorraber et al., 2020). Specifically in medicine, it is primarily tested as a supporting tool in the planning and training of operations, while remote assistance during medical interventions has been insufficiently researched so far (Vostars, 2023). Systems in medical technology are strictly regulated to protect patient safety. MR applications thus need to be thoroughly tested and must demonstrate high level of usability to ensure its improvement to the medical treatment.
Therefore, the aim of this study is to evaluate a user-centredly developed MR system for remote support during ECMO within a clinical environment.
METHODS
For the evaluation the MR system was assessed through a usability test followed by a user survey. A usability test is a user-based method, where a user is watched solving predefined tasks to evaluate usability by factors such as action competence and implementation success (DIN EN ISO 9241-11, 2018).
The usability test was conducted in a clinical setting, where the participants were stated in an ICU room with the ECMO (Cardiohelp, Getinge AB, Getinge, Sweden) and the MR system. The study focused on the participants as the users interacting directly with the technically novel aspects of the MR system. A test manager, located in a separate area, guided them after answering the call via the MR system on a tablet (see Figure 1). During the call, holograms were actively used to support the guidance, ensuring that instructions were not only provided verbally but also visually embedded in the user’s field of view.
Figure 1: Schematic setup of the test environment in the usability test.
The participants had to solve two test scenarios in the usability test. Therefore, the ECMO malfunctions of the sensor alarm and pump failure were simulated. The sensor alarm malfunction was simulated by setting up the flow sensor in the wrong direction and pump failure by shutting down the ECMO.
To assess the action competence, the scenarios were divided into subtasks via task analysis. This allows step-by-step documentation and evaluation. The test manager observed the procedure and evaluated the individual subtasks (Diaper and Stanton, 1989) using a three-level performance scale according to DIN EN 614-1 (2009). Factors relating to duration, safety, and completion of the subtasks are considered for the rating (see Table 1).
Table 1: Three-level performance scale according to DIN EN 614-1 (2009).
Table available in the original PDF paper.
Rating Description Good (G) The task is carried out quickly and without mistakes.
Medium (M) The task is performed with a delay, or an operation mistake occurs, which is corrected independently.
Poor (P) The task is not performed, is performed after assistance, or an operation error occurs that is not corrected independently.
These served as the base for calculating success rates according to Nielsen and Budiu (2021) for the whole scenario as well as for each subtask. In addition, processing times were recorded to assess system efficiency. Specifically, the times needed to solve the problem and to actually terminate the MR connection were recorded.
Finally, to record the participants perspective and experiences, user acceptance and experience were evaluated using standardized questionnaires. The System Usability Scale (SUS) is used for the quantitative evaluation of the user acceptance of hardware and software products (Brooke, 1995). It is a 5-point Likert Scale, which is then used to calculate an overall score. The calculated SUS score is interpreted as stated in Brooke (2013) in relation to the results of other applications and technologies to improve comparability and interpretability.
User Experience Questionnaire (UEQ) is used for measuring usability and user experience of technical systems. It is a 7-point Likert scale which measures both classic usability aspects such as efficiency, perspicuity, and dependability, as well as hedonic aspects such as attractiveness, novelty and stimulation (Schrepp, 2023). A universal analysis tool is available. This tool categorizes the results from poor to excellent based on the evaluation data collected for each aspect examined. Based on these results, corresponding scores were derived and subsequently interpreted in relation to established benchmark values.
RESULTS
The usability test was performed in two U.S. hospitals involving 19 intensive care nurses as participants. The participant cohort consisted of 9 male and 10 female intensive care nurses (mean age 36 ± 12 years) with an average
Usability Evaluation of a MR System for Remote Support During ECMO 245 professional experience of 10 ± 10 years. While prior MR experience was generally reported as low, most participants reported moderate to high experience with ECMO. Despite this low familiarity with MR, high troubleshooting performance was observed: success rates reached 90% for correcting the sensor alarm as well as 90% for resolving pump failure. The success rates of each scenario and subtask can be seen in Figure 2. The lowest success rates are shown in the subtask to end the calls with 21% in the first scenario and 55% in the second scenario. With 76% the subtask “insert the oxygenator into the hand-crank drive” of the scenario of the pump failure is also to highlight.
Figure 2: Mean results of the subtasks in the scenarios of sensor alarm malfunction and pump failure.
Average solution times were 3:32 ± 0:44 minutes (sensor alarm malfunction) and 4:10 ± 1:16 minutes (pump failure). Including the final system interaction required to terminate the MR connection (i.e., ending the call via the user interface), total scenario durations were tracked as 4:10 ± 0:38 minutes for sensor alarm and 5:16 ± 1:16 minutes for pump failure.
The system reached a SUS score of 83 ± 12 indicating good usability (see Figure 3).
best imaginable
AR-Application
Figure 3: SUS value rated on the evaluation scale according to Brooke (2013).
Figure available in the original PDF paper.
Consistent with this result, the UEQ measured excellent scores in usability aspects (efficiency, perspicuity, dependability) as well as hedonic aspects (attractiveness, novelty and stimulation).
DISCUSSION
These findings demonstrate that MR-based remote support can effectively assist intensive care nurses in managing ECMO-related incidents. Participants completed troubleshooting tasks quickly despite limited prior exposure to MR technology. This highlights the system’s intuitive design, high learnability, and overall clinical suitability. Given the high user acceptance and experience scores, MR technology may provide valuable bridging functionality during critical waiting periods. This may contribute to improved patient safety.
Nevertheless, few areas for improvement were identified. The main challenges observed during the usability test involved inserting the oxygenator into the hand-crank drive in the pump failure scenario and interacting with the MR user interface.
Difficulties related to oxygenator placement were primarily caused by the complexity of the instructions and the additional requirement to mark small device components. Marking semi-transparent parts located at the edges of medical devices occasionally exceeded the technical limits of the HoloLens 2 depth-sensing system. Furthermore, accurate placement of markings requires a very steady head position, as holograms are applied in real time. Rapid head movements significantly reduce precision. This can be especially challenging in a fast-paced clinical environment.
Challenges in MR interface interaction can be attributed to the novelty of MR tools, which may not be fully intuitive during first-time use. The subtask of ending the call cannot be replaced by supplementary voice recognition for safety reasons. Incorrect recognition of commands such as “end” or “cancel” could unintentionally terminate a call during critical situations and compromise patient safety. However, once users became familiar with the interaction concept, performance improved. This is reflected in the increase in success rate for the “end call” subtask from 21% in Scenario 1 to 55% in Scenario 2. Furthermore, the subtask “end call” itself is not safety- critical, as the call can alternatively be terminated by the perfusionist in case of user difficulties. As aggregated success rates combine heterogeneous subtasks with differing interaction demands and safety relevance, secondary interaction problems – such as difficulties in ending the call – may obscure the interpretation of overall usability if not analysed separately. Therefore, future evaluations and comparative analyses may benefit from a phase-based assessment of the interaction process, distinguishing between call initiation, execution of the core troubleshooting task, and call termination. Alternatively, non-critical termination steps could be excluded from overall usability ratings when the research focus lies on safety-relevant system performance. Such an approach would allow for a more precise interpretation of usability outcomes and support more targeted design improvements. Nevertheless, a structured technical introduction prior to clinical deployment of the HoloLens 2 is recommended to reduce initial usability barriers. The benefits of introducing
Usability Evaluation of a MR System for Remote Support During ECMO 247 users to MR interactions, particularly through repeated practice, have been demonstrated by Benedict et al. (2019). Familiarity with device operation enables users to apply the system more effectively in emergency situations.
Beyond these task-specific challenges, few study-level limitations must be considered when interpreting the results. Despite conducting the user study in clinical setting with the target group, it was a simulated condition rather than direct patient care. As a result, cognitive workload, time pressure, and environmental distractions may have been underestimated, potentially influencing task performance and perceived usability. In addition, the system was perceived as highly novel, which may have positively biased user ratings due to a novelty effect (Pisapia et al., 1993). These factors may have contributed to the high acceptance scores observed in the study. Future research should therefore evaluate system performance during real emergency use, assess long-term user acceptance after repeated exposure, and examine the scalability of MR-based remote support in routine clinical practice. A particular focus should be placed on time-sensitive scenarios or those with limited on-site expertise.
REFERENCES
Al Disi, M., Alsalemi, A., Alhomsi, Y., Bensaali, F., Amira, A., Alinier, G. (2019) Extracorporeal membrane oxygenation simulated-based training: methods, drawbacks and a novel solution. Perfusion 34(3), 183–194, https://doi. org/10.1177/0267659118802749 Benedict, Jacob D. et al. (2019). “The Intuitiveness of Gesture Control with a Mixed
Reality Device”. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 63.1, S. 1435–1439. issn: 2169-5067. doi: 10.1177/1071181319631403. Brooke, John (1995). “SUS - A quick and dirty usability scale”, S. 0–8. Brooke, John (2013). “JUS Journal Template: SUS: A Retrospective”, S. 29–40. Diaper, D. and Stanton, Neville A. (1989). The handbook of task analysis for human computer interaction. Mahwah NJ: Lawrence Erlbaum. isbn: 0805844325. DIN EN 614-1 (2009).“Sicherheit von Maschinen – Ergonomische Gestaltungsgrundsätze: Teil 1: Begriffe und allgemeine Leitsätze: DIN EN 614-1“. DIN EN ISO 9241-11 (2018). “Ergonomie der Mensch-System-Interaktion: Teil 11:
Gebrauchstauglichkeit: Begriffe und Konzepte: DIN EN ISO 9241-11”. Miranda, A., Vallejo, A. M., Ayala, P., Garcia, M. V., & Naranjo, J. E. (2025).
Enhancing Industrial Processes Through Augmented Reality: A Scoping Review. Future Internet, 17(8), 358. https://doi.org/10.3390/fi17080358 Nielsen, Jakob and Budiu, Raluca (2021). Success Rate: The Simplest Usability Metric. url: https: //www .nngroup. com/articles /success- rate - the - simplestusabilitymetric/. Pisapia, John et al. (1993). “Learning Technologies in the Classwoom: Review of the
Literature”. Schrepp, Martin (2023). “User Experience Questionnaire Handbook: All you need to know to apply the UEQ successfully in your projects”, S. 1–16. Solomashenko, A., Afanaseva, O., Shishova, M. et al. A systematic review of assisted
Vorraber, Wolfgang et al. (2020). “Assessing augmented reality in production: remoteassisted maintenance with HoloLens”. Procedia CIRP 88, S. 139–144. issn: 22128271. doi: 10.1016/j.procir.2020.05.025. Vostars: Video and Optical See Through Augmented Reality Surgical Systems. https://
www.vostars.eu, last accessed 2023/08/06.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Usability and User Experience
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 979-8-950676-08-6
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional