Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

A pilot study for a more Immersive Virtual Reality Brain-Computer Interface

Publication year
Digital Object Identifier10.54941/ahfe1002055

Abstract

We are presenting a pilot study for a more Immersive Virtual Reality (IVR) Brain-Computer Interface (BCI). The originality of our approach lies in the fact of recording, thanks to physical VR trackers, the real movements made by users when they are asked to make feet movements, and to reproduce them precisely, through a virtual agent, when asked to imagine mentally reproducing the same movements. We are showing the technical feasibility of this approach and explain how BCIs based on motor imagery can benefit from these advances in order to better involve the user in the interaction loop with the computer system.

INTRODUCTION

In the context of Brain-Computer Interface (BCI), Motor Execution (ME) is a voluntary body movement (foot, arm, hand…) while Motor Imagery (MI) is defined as the imagination of a kinesthetic body movement (Pfurtscheller and Neuper 2001), (Han et al. 2020). MI can be seen as a mental process by which an individual rehearses or simulates a given action. ME and MI invo- lve the same regions and activation patterns in the brain. Thus, imagination of feet movement, for instance, can be detected by an electroencephalogram (EEG) in the same area of the brain as if the user really performed the move- ment (Leeb & al. 2006). Unfortunately, many difficulties arise when dealing with the detection of imaginary movements. Firstly, the noisy nature of EEG recordings as well as the lower amplitude of brain signals related to MI com- pared to muscle activities requires the use of often complex signal processing methods to increase the Signal-to-Noise Ratio (SNR) and highlight the appro- priate brain features. Secondly, the inner nature of the task renders it difficult for users to practically use a MI-based BCI. As it is very difficult to imagine moving a limb while restraining oneself to actually perform the movement, using an MI-based BCI has been labelled as a skill to be trained (Lotte et al. 2013). An important dimension of this training has been shown to be related to the nature of the feedback provided to users when performing the ima- gination task, which can significantly improve how well users can perform MI (Jeunet et al. 2016). Building upon these findings, many researchers have investigated different feedback modalities to increase the training process for MI-BCI (Rimbert et al. 2017), (Roc et al. 2021).

© 2022. Published by AHFE Open Access. All rights reserved. 8

A Pilot Study for a More Immersive Virtual Reality Brain-Computer Interface 9

On particular modality that is currently being largely investigated is Vir- tual Reality (VR). VR enables to immerse users in 3D computer-generated environments which can be similar to or completely different from their real world. These simulated environments enable for a large degree of control over the parameters that are displayed to the users. In particular, it has been shown that VR can elicit a high sense of embodiment in virtual avatars, which in turn has been shown to potentially improve the engagement in several tasks. We also know that “The performance in 3-D virtual reality environment is con- siderably higher compared to the 2D screen” (Abbasi-Asl et al. 2019), but it is not so easy to reproduce outside laboratories, in the everyday life, for impaired users, that really need them, for example.

Due to this malleability and the engagement VR can elicit, several works have investigated the combination of BCI and VR (Lotte et al. 2013). Some work brings evidences that it is possible to interact in a virtual environment thanks to VR-BCI (Leeb 2008) and to “walk forward in a virtual street”with- out muscle activity. VR-BCI systems can be based upon various paradigms (P300, SSVEP, MI, ME, Gaze tracking...) and use various visual displays (CS: Computer Screen; VST: Video See-Through; HMD: Head Mounted Display; OST: Optical See-Through, glasses, smartphones...) (Si-Mohammed 2019).

A recent study just mentioned that “There appears to be an increase in performance while switching from the control stimulus to the VR stimu- lus in the group that has already had some experience with the SMR-based (SensoriMotor Rhythm) BCI task in the past.” (Coogan and He 2018). To go further than these initial encouraging results, which also indicated that “users performed no worse when in an immersive, virtual reality, BCI expe- riment, indicating that the immersion effect of a virtual environment does not impede performance.”, we are working on more Immersive Virtual Reality Brain-Computer Interface. In order to be really worthwhile for traditional users and being potentially used by impaired patients, those systems should send commands anytime at user’s will, and not when the machine decides it. It’s the principle of the so called self-paced (a.k.a. asynchronous) BCI (Lotte et al. 2013).

As explained by Alimardani and colleagues, an important problem lies in the fact that most users cannot visualize a realistic picture of their move- ments and its kinesthetic experience: “the mental rehearsal of a movement without actually performing it, is a counterintuitive task for the majority of individuals.” (Alimardani et al. 2018).

In this paper, we propose the design and implementation of a pilot study to investigate the effect of high fidelity movement feedback in VR, on the training in of MI skills. The first goal is to determine the feasibility, and then to determine the time, cost, risks and plan before carrying out a similar large scale project.

PILOT STUDY In our pilot study, we first ask users, seated on a chair, to execute feet move- ments (ME) of their own choice (always the same), while we are recording the performed trajectories (see figure 1). Indeed, VR trackers positioned at

Figure 1: An avatar representing the user in an immersive virtual world. The experi- menter can see the external view (left) while the user (right) see the virtual world at the first person.
Original PDF Figure 1:

the feet of the users enable us to capture, record and render on the VR ava- tar, the real movements made, in terms both of direction and amplitude, by the participants. This first step is designated as the warmup phase. In a second time, we ask participants to perform feet-MI task, by imagining the same movement as in the warmup phase. During this task, our final study will aim at comparing between three conditions: (1) Displaying the feedback using a state of the art modality, namely the so-called Graz visualization; (2) Displaying an immersive feedback on the virtual avatar, unrelated with the previously performed movement; (3) Displaying an immersive feedback on the virtual avatar, with a high fidelity to the previously performed and the imagined movement.

Our hypothesis is that the high-fidelity nature of the rendered feedback will increase the sense of agency over the avatar, and thus help in the training and appropriation of the MI task, as suggested by recent studies: “Leeb et al. also compared the influence of feedback types on the motor imagery performance and BCI classification accuracy. They found that immersive feedback (wal- king inside a VR environment) resulted in a better task performance by the subjects than a simple BCI feedback (bar presented on a computer screen)” cited by (Alimardani et al. 2018).

Materials, Softwares and Methods Various avatars were used during the prototyping phase. For instance, the figure 2 shows an early version of our work, with a robot avatar seen by behind. This version was used to check the communication layer between software.

An EEG detects the beta rebound measured with a Laplacian on 5 electro- des (around Cz in a 10/20 international protocol). This trigger is used as a switch for the avatar’s walk in our Unity3D application. The signal treatment is made with OpenViBE, a software platform that enables to design, test and use Brain-Computer Interfaces (BCIs) and also that can be used as a generic real-time EEG acquisition, processing and visualization system. We also use Unity3D, a cross-platform game engine developed by Unity Technologies. It supports a variety of desktop, mobile, console and virtual reality platforms. In

A Pilot Study for a More Immersive Virtual Reality Brain-Computer Interface 11

Figure 2: Communication between Unity 3D (left) and OpenViBE (right) across Lab- Streaming Layer protocol.
Original PDF Figure 2:
Figure 3: The user performs a foot movment while a VR tracker is positioned on his shoe (left). The user can see this exact movment reproduced by his avatar (right).
Original PDF Figure 3:

our prototype, the communication between OpenViBE and Unity is assumed by LSL (LabStreaming Layer) protocol.

At this point, two solutions were discussed: (a) the avatar walk is triggered at a particular threshold and as long as brain activity exceeds this threshold, the walk continues, or (b) the avatar walk is triggered at a particular thre- shold and continues just during a few seconds before stopping. After several tries, we found that it was more ecological for the user and more relevant for the signal treatment (see beta rebound detected after the end of the muscle activity) to exploit the second solution.

Then, we decided to implement a more realistic avatar. It was created in Unity3D with tools allowing calculating Inverse Kinematics (Gonzalez- Franco et al. 2020). This experiment part is conducted with a “ Valve Index” headset, for the VR part, and a Unicorn gTec cap, for the BCI part. VR tra- ckers are positioned on the user’s feet. Thanks to Inverse Kinematics (IK), it is possible, in a very realistic way, to reproduce in a virtual world the moves made by the user, even with few sensors. For instance, in our study, only one sensor was positioned on each foot, but we can see the knees of the avatar going up accordingly, when the user decides to lift his foot off the ground (see figure 3).

Rationale Lotte et al. mentioned eleven studies involving BCI-based VR applications using Motor Imagery (Lotte et al. 2013). Brain-computer interfaces and vir- tual reality for neurorehabilitation were also studied together (Vourvopoulos and Bermúdez i Badia 2016), (Achanccaray et al. 2018), (Leeb and Perez- Marcos 2020), (Georgiev et al. 2021). But, in the literature, few papers deal with an approach where the avatar that the user see in the VR headset repre- sents exactly his/her own body with a very realistic sensation that moving a part of the body in the real world performs the same movement in the virtual world, without constraints (speed, amplitude, repetition, etc.).

The originality of our proposition lies in the fact of recording, thanks to physical VR trackers, the real movements made by users, at the warmup phase, when they are asked to make feet movements. In a second time, we are able to reproduce those moves precisely, through a virtual agent, when the user is asked to imagine mentally reproducing the same movement. Our hypothesis is that using a Brain-Computer Interfaces with an Immersive Vir- tual Reality will lead users to better understand how the BCI reacts. Instead of a simple bar visualization on the screen, we assume that seeing the exact movements, previously recorded, with the same directions and amplitudes will bring a better feedback and hopefully, will lead to a better interaction loop.

In the final study, the true positive rate (TPR) and false positive rate (FPR) on Event-Related (De)Synchronizations (ERD/ERS) will be considered as indicators to play the sequence associated to feet ME during MI.

A System usability scale (SUS) questionnaire with assertions such as “I find using a BCI in immersive 3D more efficient than in 2D” with a 5 levels Likert scale (1. Totally agree, 2. Agree, 3. Neither disagree nor agree, 4. Disagree, 5. Strongly disagree) will be proposed to users after the experiment (Tcha-Tokey et al. 2016).

These two kinds of results (measured data and personal feeling) will confirm or invalidate our scientific working hypothesis, namely: (1) the immersion of a user in a 3D environment, thanks to a virtual reality headset, improves his/her performance for a BCI based on feet motor imagination and (2) the fact the avatar reproduce the exact moves made by the user increases the incarnation feeling and thus the efficiency of the BCI.

CONCLUSION

We have shown through our pilot study the technical feasibility consisting in reproducing precisely in an immersive virtual world, the feet movements of an avatar, recorded previously on the user’s real movements. We asked a few beta testers to move their feet at their convenience with no constraints of directions and amplitudes (heel, toe, knee...). Then, we asked them to ima- gine moving their feet without really performing that movement. The EEG treatment allows to detect this intention and this is considered as a trigger to replay the sequence recorded previsoulsy, given so the sensation to the user that his/her mental will in the real world is performed in the virtual world. This allows the user to visualize his/her own avatar performing previously

A Pilot Study for a More Immersive Virtual Reality Brain-Computer Interface 13

recorded foot movements (during the warmup phase), while he tries to per- form this movement mentally. We are now planning to recruit participants for a larger experiment campaign, approved by our research ethics commit- tee. In our next experiments, conducted on the basis of this preliminary work, we hope to obtain results on a representative number of users that will show, after the technical feasibility demonstrated in the paper, the relevance and interest of the approach to better involve them in the BCI interaction loop. For future work, it seems possible to us to imagine making a movement of the feet (walking, bicycling, kicking, etc.) in order to virtually actuate a mecha- nism in the 3D scene. This could be, for example, a wheel or a pulley, pointing on various potential actions (switch on/off TV, fan, light…) which one could look at, straight ahead, and interact with, thinking about the movement of the feet learned previously.

REFERENCES

Abbasi-Asl, R., Keshavarzi, M. and Chan, D. Y. (2019). Brain-Computer Interface in Virtual Reality, 9th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 1220–1224. https://doi.org/10.1109/NER.2019.8717158 Achanccaray, D., Pacheco, K., Carranza E., and Hayashibe, M. (2018). Immersive

Virtual Reality Feedback in a Brain Computer Interface for Upper Limb Rehabili- tation, IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1006–1010. https://doi.org/10.1109/SMC.2018.00179 Alimardani, M., Nishio, S., Ishiguro, H., (2018). Brain-Computer Interface and Motor Imagery Training: The Role of Visual Feedback and Embodiment, in D. Larrivee (ed.), Evolving BCI Therapy - Engaging Brain State Dynamics, IntechOpen, London. https://doi.org/10.5772/intechopen.78695 Coogan, C. G. and He B. (2018). Brain-computer interface control in a virtual reality

environment and applications for the internet of things, in IEEE Access, vol. 6, pp. 10840–10849. https://doi.org/10.1109/ACCESS.2018.2809453 Georgiev, D.D., Georgieva I., Gong, Z., Nanjappan, V., Georgiev G.V. (2021).

Virtual Reality for Neurorehabilitation and Cognitive Enhancement. Brain Sci. 11;11(2):221. https://doi.org/10.3390/brainsci11020221 Gonzalez-Franco M, Ofek E, Pan Y, Antley A, Steed A, Spanlang B, Maselli A, Banakou D, Pelechano N, Orts-Escolano S, Orvalho V, Trutoiu L, Wojcik M, Sanchez-Vives MV, Bailenson J, Slater M and Lanier J, (2020) The Rocketbox Library and the Utility of Freely Available Rigged Avatars. Front. Virtual Real. 1:561558. https://doi.org/10.3389/frvir.2020.561558 Han C-H, Müller K-R, Hwang H-J. (2020). Brain-Switches for Asynchronous Brain–

Computer Interfaces: A Systematic Review. Electronics. 9(3):422. https://doi.org/ 10.3390/electronics9030422 Jeunet, C., Jahanpour, E., and Lotte, F. (2016). Why standard brain-computer interface (BCI) training protocols should be changed: an experimental study. Journal of neural engineering, vol. 13, no. 3, p. 036024. Leeb, R., Keinrath, C., Friedman, D., Guger, C., Scherer, R., Neuper, C., Garau, M.,

Antley, A., Steed, A., Slater, M., Pfurtscheller, G. (2006). Walking by thinking: the brainwaves are crucial, not the muscles! Presence: Teleoperators and Virtual Environments, 15, pp. 500–514. Leeb, R. (2008). Brain-computer communication: the motivation, aim, and impact of virtual feedback. Ph.D. thesis, Graz University of Technology.

Leeb, R.; Perez-Marcos, D. (2020). Brain-computer interfaces and virtual reality for neurorehabilitation. In Handbook of Clinical Neurology; Elsevier: Amster- dam, The Netherlands, 2020; Volume 168, pp. 183–197. https://doi.org/10.1016/ B978-0-444-63934-9.~00014--7 Lotte, F., Faller, J., Guger, C., Renard, Y., Pfurtscheller, G., et al. (2013). Combining

BCI with Virtual Reality: Towards New Applications and Improved BCI. Allison, Brendan Z. and Dunne, Stephen and Leeb, Robert and Millán, José Del R. and Nijholt, Anton. Towards Practical Brain-Computer Interfaces:, Springer, 2013. ⟨hal-00735932⟩. Pfurtscheller, G. and Neuper, C. (2001). Motor Imagery and Direct Brain-Computer Communication. in Proceedings of the IEEE, vol. 89, no. 7, pp. 1123–1134. https: //doi.org/10.1109/5.939829 Rimbert, S., Bougrain, L., Orhand, R., Nex, J., Gaborit, S., & Fleck, S. (2017).

Grasp’it: a brain-computer interface for improving the kinesthetic motor ima- gery learning, 29ème conférence francophone sur l’Interaction Homme-Machine, Poitiers, France, hal-01568588v2. Roc, A., Pillette, L., Mladenovic, J., Benaroch, C., N’Kaoua, B., Jeunet, C., & Lotte, F. (2021). A review of user training methods in brain computer interfaces based on mental tasks. Journal of Neural Engineering, 18(1), 011002. Si-Mohammed, H. (2019) Design and Study of Interactive Systems based on Brain-

Computer Interfaces and Augmented Reality, PhD thesis, INSA de Rennes, France. Tcha-Tokey, K., Christmann, O., Loup-Escande, E., Richir, S. (2016). Proposition and Validation of a Questionnaire to Measure the User Experience in Immersive Virtual Environments. International Journal of Virtual Reality, IPI Press, 2016, 16 (1), pp. 33–48. ⟨hal-01404497⟩ Vourvopoulos A. and Bermúdez i Badia, S. (2016). Motor priming in virtual reality

can augment motor-imagery training efficacy in restorative brain-computer intera- ction: a within-subject analysis, Journal of NeuroEngineering and Rehabilitation 13:69. https://doi.org/10.1186/s12984-016-0173-2

Cover of the proceedings volume Human Factors in Virtual Environments and Game Design

Explore the complete digital volume

Human Factors in Virtual Environments and Game Design

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-26-1

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →