Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

BIOFEE: Biomedical Framework for Enhanced Experimentation

Publication year
Digital Object Identifier10.54941/ahfe1003451

Abstract

Biomedical Framework for Enhanced Experimentation, based on Firebase Real-Time Database, helps in designing and testing various multimodal solutions for patients suffering from progressive diseases. It allows interacting with objects and robots to test various multimodal solutions, such as touch, voice, gesture... to perform tasks according to the patient's pathology. We tested BIOFEE with Unity3D and Webots virtual worlds to compare different ways to interact for particular tasks (switch lamps, pick and place...).

INTRODUCTION

In the domain of health informatics and biomedical engineering applications, we have to design and implement multimodal solutions adaptable to patients, carers, health professionals and social context. But desi- gning and developing such medical applications for people with speci- fic needs might prove very difficult, time consuming and expensive for multiple reasons: – Unique and particular needs, according to cognitive or physical specific disabilities;

– Health situation and mental state changing from one session to another; – Calibration and sharp synchronization of sensors and effectors needed; – Requirement to test various input and output solutions... In a previous work, we have introduced a Multimodal Interaction Frame- work based on Firebase Real-Time Database (Guedira and Rouillard 2021). The interest of such intelligent systems built on a multimodal basis lies in the fact that a decision is made through several independent information chan- nels with the subsequent aggregation of these decisions (Filist et al. 2022).

In this paper, we introduce the BIOFEE project: Biomedical Framework for Enhanced Experimentation. The article is organised as the following: part 2 presents the related work, part 3 describes the BIOFEE frame- work architecture and part 4 shows some case studies using Unity3D (Unity 2023) and Webots (Webots 2023), before the conclusion and perspectives part.

© 2023. Published by AHFE Open Access. All rights reserved. 22

BIOFEE: Biomedical Framework for Enhanced Experimentation 23

RELATED WORK

Worldwide, the number of people with significant disabilities is approxima- tely 1.3 billion, or 16% of the world’s population. Among them, one in five people has a so-called severe disability (WHO 2022). In countries where life expectancy exceeds 70 years, each individual will spend an average of 11.5% of his life with a disability (United Nations 2023). Today, assistive technolo- gies compensate, to some extent, the user for physical, functional, cognitive or mental disabilities.

Unfortunately, assistive technologies are often expensive, designed for a specific disability (for paraplegic people, blind people, deaf people, autistic people, elderly people, etc.) and must be configured according to each user. Nevertheless, assistive technologies offering multimodal interfaces can adapt to a greater number of users than those offering traditional interfaces (Ovi- att 2003). To be fully inclusive, multimodal interfaces must be tailored to adapt to the needs, capabilities and environment of users. A deficient user in one modality can then compensate for his handicap by alternative modali- ties without limiting the system functionality. The simultaneous provision of several communication channels makes it possible to choose the most practical/fast/intuitive channel according to the handicap, state health and context.

Disabilities can be motor or sensory, mental or psychic or both in case of multiple disabilities. The scientific literature presents various multimodal assistive technology systems for many disabilities. (Bissoli et al. 2015) inte- grate, in an intelligent environment, a wheelchair specially equipped with a multimodal communication system. A contextual menu (depending on the rooms visited and the connected devices) is displayed on a screen integrated into the wheelchair. The menu item selection is based on either frontal muscle activity or brain activity. A robotic arm equipped with a tablet is used by (Brunete et al. 2021) in a smart environment. The user can interact with the robot and with the various IoT (Internet of Things) systems by gesture, voice, a touch interface or augmented reality. (Sahadat et al. 2018) enable computer access by tracking head movements, voice recognition and tongue movements instead of mouse and keyboard. (Argyropoulos et al. 2007) have develo- ped a collaborative treasure hunt game between blind players and deaf-mute players integrating voice and haptic feedback on the one hand and vision and sign language on the other. An Aphasia rehabilitation support system is presented by (Mabutchi et al. 2015) including reading, listening, writing and speaking. Multimodal communication is implemented thanks to a sim- ple commercial tablet equipped with a microphone, a loudspeaker, a touch screen and a camera. This list, far from being exhaustive, presents a variety of multimodal assistive technology aids designed for different disabilities.

THE BIOFEE FRAMEWORK The Figure 1 describes the BIOFEE modular architecture. The user’s devices (PC, smartphones…) are connected to a Firebase Real-Time Database. This allows each component of the system to be notified instantly when a modi- fication occurs in the database. The various input signals are then treated by

Figure 1: Architecture of our multimodal BIOFEE framework based on Firebase Realtime Database.
Original PDF Figure 1:

the multimodal engine to synthesize one user command. Then, the system reacts accordingly. A Wizard of Oz module (Hoffman 2016) is also availa- ble in order to let a human (the wizard) inject those user commands when the machines are not able to do it by themselves (cf. complex or ambiguous multimodal request, for example).

Other users (family, medical staff, etc.) can directly interact with the system thanks to conventional inputs/outputs, such as mouse, keyboard, remote control, speakers… whereas the main user often has equipment that is parti- cularly adapted to the disability and to the evolution of the disease. In order to compare different solutions, the system is able to generate a random cycle of tasks to perform by the patients, and generate automatically the prompts to diffuse on their connected devices.

CASE STUDIES a) Pick and Place (Unity 3D application)

A first case study is presented to show the implementation of components that the experimenter can test, such as ASR (Automatic Speech Recogni- tion), TTS (Text To Speech), gaze detection, touch screens, optical hand tracking modules, EMG (electromyography) for the detection of muscles activities, EEG (electroencephalography) for the detection of intention to perform actions or in reaction to certain stimuli applied on the skin, etc.

Let’s imagine that an experimenter would like to test a ≪1 Degree Of Fre- edom ≫solution with a patient with DMD (Duchenne Muscular Dystrophy) or ALS (Amyotrophic Lateral Sclerosis) disease, for instance. The goal, here, is to detect some residual activities on separate or combined channels (Sharma and al. 1998), more or less available, across different sessions (cf. task to perform, user fatigue, device usable...).

In a first session, the patient could touch a button on the screen “A” with the left hand to choose a command (5 choices offered in an infinite circular loop: Up, Right, Down, Left, Robot). The icon above the button is chan- ging as a feedback given to the user. With the right hand, the patient could touch a button on the screen “B” to effectively send to the system the desired command, previously chosen with the left hand (see Figure 2).

BIOFEE: Biomedical Framework for Enhanced Experimentation 25

Figure 2: Mobile interface used to interact with the pick and place application (see Fig 3).

Figure available in the original PDF paper.

Figure 3: Pick and place example, performed in a Unity application connected to Firebase Database and piloted by ROS and MoveIt in a Docker container.
Original PDF Figure 3:

With such a mechanism, certainly longer than a classical use, a physically disabled person can control the movement of a target object, step by step, on a graphical interface, programmed with Unity 3D, for instance (see Figure 3). Some auditory and haptic feedbacks are also proposed on the smartphone. Then a “Pick and Place” command can be requested thanks to the “Robot” button. This allows a virtual robotic arm to plan and perform trajectories and movements, thanks to Firebase Real-time Database, via ROS (Robot Operating System) and MoveIt protocols, in a Docker container.

Now, let’s imagine that, unfortunately, this DMD patient is no more able to click with a finger on a smartphone. Which other modalities could be used by this specific user? We would like to test if a residual muscular activity is detectable with an optical hand tracking module, such as a LeapMotion (UltraLeap 2023) for instance. In other sessions, the experimenter will pro- pose EMG and EEG, ASR and TTS, gaze detection, etc. in order to determine the most suitable modalities usable by this patient, depending on the context. The BIOFEE framework is thus used to more easily design and test means of interaction, both for input and output.

For example, with less force in the fingers, is an indirect interaction still possible, using an optical device? A Leap Motion Controller (LMC) is used when it becomes difficult for a person to press a push button or for hygiene issues. The LMC is a device that uses infrared cameras and hand tracking technology to allow users to interact with computers and other devices

through gestures made with their hands. The controller is often used in Vir- tual Reality (VR) and Augmented Reality (AR) applications, but it can also be used with other types of software especially with disabled people. Gesture control is an area of research as the controller has the potential to serve as an alternative input method for individuals with physical disabilities enabling users to control devices and interact with computers using gestures which can be an alternative for individuals who are unable to use traditional input methods such as keyboard or mouse.

Although different studies and projects have been developed to use hand tracking technology with disabled people, such as a hand rehabilitation system to assist in developing muscle tonus and increase precision in gestures (Alimanova et al. 2017, Cortés-Pérez et al. 2021), for sign languages recogni- tion (Galván-Ruiz et al. 2020), in music therapy sessions (Baratè et al. 2018), it is not a common or a widely used technology.

Some limitations of the device include:

  • a limited tracking range (10 to 70 cm) and a restricted angular view (120 to 140
  • ) (Ultraleap 2023),
  • a possibility of bad tracking and accuracy lack when hands or fingers are obscured by other objects. It then needs a clear line of sight (Potter et al. 2013), (Vannobel et al. 2022),
  • a limited number of supported gestures. It may not be able to recognize complex hand and finger movements like those used in sign languages (Leap Motion 2023),
  • an interior use. The controller may not work properly in bright sunlight or other harsh lighting conditions (Insani et al. 2019),
  • a use limited to desktop computers. The LMC is not compatible with mobile devices (Ultraleap 2023),
  • a high cost comparatively to a mouse, a keyboard or a joystick. It may not be accessible to all users.

On the other hand, benefits include:

  • a high accuracy especially for hand palms, thumb and index fingers tips making the device well-suited for use with individuals with limited movements abilities,
  • a natural user interface because the users can interact with their computers using hand and finger gestures as in the physical world,
  • a portable use since the controller is small, lightweight and easy to set up,
  • the compatibility with a wide range of operating systems and program- ming languages making it easy to integrate into existing software systems or as a pointing device (Bachmann et al. 2014),
  • a large developer community especially for VR and AR applications,
  • an alternative way to traditional input devices such as mouse, keybo- ard and joystick, which can be beneficial for users with certain types of physical or cognitive impairments.

In our case, the LMC is used as a complement or replacement for touch screens. Indeed, these require the ability to raise and lower the fingertip,

BIOFEE: Biomedical Framework for Enhanced Experimentation 27

Figure 4: Leap Motion used to interact with a virtual robot without touching the smartphone.
Original PDF Figure 4:

which is not always easy to achieve for the users we consider. The LMC thus allows relying on slight flexions of the index finger which are more comfor- table to perform than a finger lift with DMD patients. It can be used to drive GUI oriented software instead of using a mouse or to move a robotic arm.

In the figure 4, for instance, a user is requesting a Niryo Ned (Niryo 2023) robotic arm to perform a pick and place task. A little movement of the index finger is detected by a LMC and instantly transmitted to Webots, via Firebase Real-time database.

b) Switch on/off Lamp (Webots application) In this second case study, we are considering a virtual world created with the Webots application (Webots 2023).

When this virtual world is launched, a connection, programmed in Python and JavaScript, is established with our Firebase Real-time database. Our supervisor declares into our Firebase database all the interactive objects (example: Lamp_1 to Lamp_5), and their values, as shown on Figure 5 (left).

The experimenter can now consider, for instance, a voice interaction in order to switch on/off a particular lamp of this virtual world. Thus, an Android mobile application, created with AppInventor (AppInventor 2023) is used to send commands to Webots. The “Number_task_todo”variable can easily be modified by the experimenter to generate automatically a cycle of random tasks to be performed by the user (ex: “Please switch on the light 3”). This number is decremented when the system detects that the task is succes- sfully achieved by the user. A simple voice command such as “switch lamp 1” allows the user to switch (on or off, according to the current state) a lamp. This voice interaction was designed, in an infinite loop, and do not require direct muscular force, like a push on a button, to be activated.

But, as we can see on Figure 5 (right), interactive objects of this virtual world can be also selected with a slight movement of the index finger, when a voice or a simple touch on a smartphone screen are no more usable by

Figure 5: Firebase Database (left) containing interactive objects declared by Webots (right), where a leap motion is used to change the selected object with an index finger movement.
Original PDF Figure 5:

the patient. The Leap Motion Controller is used to change the selected object with an index finger movement from the left hand. The right hand movement allows to interact with the selected object (switch on/off for instance). Later, when those left and right commands would not be detectable anymore with muscular activities, the BIOFEE framework will be usable across commands detected on C3 and C4 electrodes on an EEG cap, related to right and left intention to move a hand. In a close future, with the use of non-invasive BCI based on ultra-high-density electroencephalography (Lee et al. 2022) it will be possible to detect individual finger movement or intention of movement.

CONCLUSION

Since the beginnings of multimodal interfaces (Bolt 1980), assistive techno- logy systems have continued to evolve alongside the development of new technologies. Today, multimodality is found in the design of smartphones, voice assistance systems, watches, and other connected objects via the IoT. This makes assistive technology systems less stigmatizing. Who is surprised today to hear someone talking to a machine? What gamer hasn’t used or dreamed of using a virtual reality headset?

Biomedical Framework for Enhanced Experimentation helps in designing and testing various multimodal solutions for patients suffering from progres- sive diseases. The tools used for interfacing are daily live objects (telephone, tablet, PC) or inexpensive/free technological tools/software (Leap Motion Controller, AppInventor, Firebase Real-Time Database, Webots, etc.). It is a rapid prototyping framework that is set to evolve: for the control of real robots, to enable the study and comparison of the different modalities (or combinations of modalities) chosen in terms of efficiency, usability and speed.

One of our prospective short-term works will be to verify the usability of our tools when patients will have to communicate with robots like the PR2, visible in figure 6. We will also further improve the capabilities offered

BIOFEE: Biomedical Framework for Enhanced Experimentation 29

Figure 6: Webots virtual world with interactive objects (lamp, curtain, bed) and PR2 Robot.
Original PDF Figure 6:

by BIOFEE by integrating automatic data recording (score, time elapsed to complete a task, etc.) in order to better compare the different modalities that can be used by patients. Finally, in the medium and long term, we are plan- ning to add fusion and fission functionalities to our framework, in order to support more natural interactions for patients.

REFERENCES

Alimanova, M., Borambayeva, S., Kozhamzharova, D., Kurmangaiyeva, N., Ospanova, D., Tyulepberdinova, G. and Kassenkhan, A. (2017). Gamification of hand rehabilitation process using virtual reality tools: Using leap motion for hand rehabilitation. In 2017 First IEEE International Conference on Robotic Computing (IRC), 336–339. doi: 10.1109/IRC.2017.76. AppInventor 2023, Accessed on Jan. 31, 2023. http://ai2.appinventor.mit.edu/ Argyropoulos, S., Karpov, A., Moustakas, K., and Aran, O. (2007). A multi-

modal framework for the communication of the disabled. Proceedings of the eNTERFACE’07 Workshop on Multimodal Interfaces, Istanbul, Turkey, 27-36. Bachmann, D., Weichert, F. and Rinkenauer, G. (2014). Evaluation of the leap motion controller as a new contact-free pointing device. Sensors, 15(1): 214–233. doi: 10.3390/s150100214 . Baratè, A., Elia, A., Ludovico, L. A. and Oriolo, E. (2018). The leap motion controller

in clinical music therapy: a computer-based approach to intellectual and motor disabilities. In International Conference on Computer Supported Education, 461– 469. Bissoli, A., Ferrara, F., Sime, M., and Bastos, T. (2015). A Multimodal Assistive System to Operate a Smart Environment. Proceedings of the 1st International Workshop on Assistive Technology, IWAT 2015, Victoria, Brazil, 103–106. Brunete, A., Gambao, E., Hernando, M. and Cedazo, R. (2021). Smart Assistive

Architecture for the Integration of IoT Devices, Robotic Systems, and Multi- modal Interfaces. In Healthcare Environments. Sensors 2021, 21, 2212. doi: 10.3390/s21062212.

Cortés-Pérez I; and al., “Leap Motion Controller Video Game-Based The- rapy for Upper Extremity Motor Recovery in Patients with Central Nervous System Diseases. A Systematic Review with Meta-Analysis,” Sensors 2021, vol. 21(6):2065, 2021. de Los Reyes-Guzmán, A., Lozano-Berrio, V., Alvarez-Rodríguez, M., López-Dolado, E., Ceruelo-Abajo, S., Talavera-Díaz, F., Gil-Agudo, Á., RehabHand: Oriented-tasks serious games for upper limb rehabilitation by using leap motion controller and target population in spinal cord injury. NeuroRehabilitation, vol. 48, no. 3, pp.365–373, 2021, DOI: 10.3233/NRE-201598 Filist, S. A., Al-Kasasbeh, R. T., Shatalova, O. V., Aikeyeva, A. A., Al-Habahbeh, O. M., Alshamasin, M. S., Alekseevich, K. N., Khrisat, M., Myasnyan- kin, M. B. and Ilyash, M. (2022). Classifier for the functional state of the respiratory system via descriptors determined by using multimodal techno- logy. Comput Methods Biomech Biomed Engin. 2022 Oct 28, 1-19. doi: 10.1080/10255842.2022.2117551. Epub ahead of print. PMID: 36305552. Galván-Ruiz, J., Travieso-González, C. M., Tejera-Fettmilch, A., Pinan-Roescher,

A., Esteban-Hernández, L and Domínguez-Quintana, L. (2020). Perspective and evolution of gesture recognition for sign language: A review. Sensors, 20(12), 3571. doi: 10.3390/s20123571 Guedira, Y. and Rouillard, J. (2021). Multimodal Interaction Framework Based on Firebase Real-Time Database. Human-Computer Interaction International (HCII 2021), Jul 2021, Washington DC, United States. 367-384. doi: 10.1007/978-3- 030-78095-1_27. ⟨hal-03277571⟩. Hoffman, G. (2028). OpenWoZ : a runtime-configurable Wizard-Of-Oz framework

for human-robot interaction. In AAAI Spring Symposium on Enabling Computing Research in Socially Intelligent Human-Robot Interaction, Palo Alto, CA Insani, C. N, Nurtanio I. and Ilham A. A., “The effect of light on Leap Motion Controller in the classification of Sign Language Translator System,” 2019 International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), Yogyakarta, Indonesia, 2019, pp.296–300, doi: 10.1109/IS- RITI48646.2019.9034602. Leap Motion JavaScript SDK v2.3 Documentation: https://developer-archive.leap

motion.com/documentation/v2/javascript/devguide/Leap_Overview.html (acces- sed on 31 January 2023). Lee, H. S., Schreiner, L., Jo, S. H., Sieghartsleitner, S,. Jordan, M., Pretl, H., Guger, C. and Park, H. S. (2022). Individual finger movement decoding using a novel ultra-high-density electroencephalography-based brain-computer interface system. Front. Neurosci. 16:1009878. doi: 10.3389/fnins.2022.1009878. Mabutchi, T., Takeda, T., Kubota, N. and Matuda, T. (2015). Aphasia Rehabilitation

Support System by Using Multimodal Interface Device. 2015 IEEE International Conference on Systems, Man and Cybernetics, Hong Kong, China, 1433–1438. doi: 10.1109/SMC.2015.254. Niryo Inc, Niryo - robotic arm. Accessed on Jan. 31, 2023. https://niryo.com Oviatt, S. (2003). Multimodal Interfaces. Jacko, J. A. and Sears, A., eds. The Human- Computer Interaction Handbook. Lawrence Erlbaum Associates, Mahawah, New Jersey, USA, 286–301. Potter, L. E., Araullo J. and Carter, L., The leap motion controller: A view on

sign language, in 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, (OzCHI 2013), pp. 175– 178, November 2013.

BIOFEE: Biomedical Framework for Enhanced Experimentation 31

Richard A. Bolt. 1980. “Put-that-there”: Voice and gesture at the graphics interface. SIGGRAPH Comput. Graph. 14, 3 (July 1980), 262–270. doi: 10.1145/965105.807503 Sahadat, M. N., Alreja, A. and Ghovanloo, M. (2018). Simultaneous Multimodal PC Access for People With Disabilities by Integrating Head Tracking, Speech Reco- gnition, and Tongue Motion. in IEEE Transactions on Biomedical Circuits and Systems, vol.12, no.1, 192–201, Feb. 2018. doi: 10.1109/TBCAS.2017.2771235. Sharma, R., Pavlovic, V. I. and Huang, T. S. (1998). Toward multimodal human-

computer interface. In Proceedings of the IEEE, vol. 86, no. 5, 853-869, May 1998, doi: 10.1109/5.664275. Ultraleap Inc, Ultraleap - hand tracking module. Accessed on Jan. 31, 2023. https: //www.ultraleap.com/ United Nations, Département des affaires économiques et sociales. Faits et chiffres

sur le handicap. Accessed on Jan. 31, 2023. https://www.un.org/development/des a/disabilities-fr/faits-et-chiffres-sur-le-handicap.html Unity, Unity Inc, Accessed on Jan. 31, 2023. https://unity.com/ Vannobel, J.-M., Bekaert, M.-H., Baumann, J., Le Leap Motion Controller: de la souris gestuelle à la commande gestuelle, Deux études de faisabilité, in Proceedings of the Handicap 2022 Conference, June 2022, Paris. Webots, Webots, Open-source Mobile Robot Simulation Software. Accessed on Jan.

31, 2023 http://www.cyberbotics.com. World Health Organization, Rapport mondial sur l’équité en santé pour les personnes handicapées : résumé d’orientation. 02/12/ 2022. OMS EDs, ISBN: 9789240064539. view on sign language, in 25th Australian Computer- Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, (OzCHI 2013), pp. 175–178, November 2013.

Cover of the proceedings volume Health Informatics and Biomedical Engineering Applications

Explore the complete digital volume

Health Informatics and Biomedical Engineering Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-54-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →