Ki Hwan Kim, Eun Sung Choi, Young Jin Kang, Seok Chan Jeong (2024). Cyber Rehabilitation for the Elderly with Mild Cognitive Impairment. In: Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems. AHFE Open Access, vol. 119, pp. 303–313. https://doi.org/10.54941/ahfe1004494
The World Health Organization (WHO) predicts that by 2050, the proportion of people over the age of 60 will double. As a result, there is a growing need for digital therapy for geriatric patients with Mild Cognitive Impairment (MCI). This paper introduces a platform designed to facilitate MCI examination, user response judgment, and real-time content control for the elderly with MCI. This paper employing techniques such as You Only Live Once (YOLO) and eye tracking to minimize user intervention, this system has demonstrated a user response judgment accuracy of 90% or higher for 100 elderly individuals with MCI. Additionally, the platform achieved a content control function accuracy of 95% based on reaction judgment and a content matching suitability rate exceeding 90%.
The World Health Organization (WHO) anticipates that the global propor- tion of people aged 60 years and over will nearly double from 12% to 22% by 2050 (World Health Organization, 2015). Demographic shifts are occur- ring in all countries, resulting in various recommendations and responses to these changes. The Alzheimer’s Association has reported that approximately 12–18% of the population over the age of 60 in the United States suffers from Mild Cognitive Impairments (MCI) (Anderson, 2019). Moreover, a study involving 3,496 people aged 65 and older in the United States found that approximately one-third exhibited dementia or MCI. The prevalence of these conditions was similar across genders but differed by age, education, race, and ethnicity (Manly et al., 2022).
The US National Institute on Aging (NIH) indicates that an estimated 10–15% of high-risk patients with predementia will progress to Alzheimer’s disease (MCI, 2021). Patients with Alzheimer’s dementia exhibit more severe impairments in memory, language, and orientation measures compared to those with Parkinson’s disease (Huber et al., 1989). Early detection and consistent treatment of long-term cognitive impairment can reduce the risk
of symptom deterioration. Delaying dementia onset by 2 years has been found to reduce dementia prevalence to 80% after 20 years, and delaying it by 5 years has been shown to reduce prevalence to 56% (Hira, 2020). While elderly individuals unfamiliar with computers may find learning to use tablets and other devices time-consuming and challenging, various attempts at cybergame rehabilitation have been made for cognitive rehabilitation (Lee et al.; Hwang et al., 2022; Redmon et al., 2016). Jodderll et al. (Manca et al., 2021) reported that people with dementia can effectively use touchscreen devices. Additionally, Manca et al. (Manca et al., 2021) demonstrated that a music-based memory game increased participation among 14 older adults with cognitive impairment.
This paper describes the design and testing of a platform intended to assist cyber-rehabilitation therapy for MCI patients by incorporating computer vision technology into MCI rehabilitation and minimizing user control. For the primary experiments outlined in this paper, we implemented simple MCI diagnosis, user state monitoring during MCI therapeutic content viewing, and real-time viewing state feedback.
RELATED WORKS Medically, there are various diagnostic methods for MCI, including neu- rological examinations, MRI brain imaging, and mental state tests (MCI). Neurological examinations encompass methods of testing reflexes and motor skills. Virtual Reality (VR) dominates visual and auditory senses, providing users with a sense of immersion similar to reality.
The research area combining MCI and VR is steadily growing. Using a VR device (Data Glove), symptom relief can be facilitated by observing a patient with neglect syndrome reaching for and grabbing a real object while simulta- neously grabbing a virtual object in the virtual environment using their virtual hand (Ansuini et al., 2006; Sofroniew et al., 2015). Additionally, reports indi- cate that utilizing VR could assist in MCI patient treatment regarding spatial memory (Allison et al., 2016), activities of daily living (Buss, 2009), language (Montenegro et al., 2017), executive function (Tost et al., 2015), short- term working memory (Burdea et al., 2013), attention (Kalova et al., 2005), and movement and balance (Wen et al., 2018). Patients with various stages of cognitive impairment, including MCI (PDCIP), could potentially benefit from alleviating symptoms through VR, as spatial memory impairment is a common symptom (Wen et al., 2018).
Game-based methods for assessing MCI among the elderly can be useful tools. Attention improvement has been achieved through virtual environ- ments and tasks (Tost et al., 2014) involving 3D families, as well as puzzles (Manera et al., 2016). Spatial memory impairment assessment examples include tablet PC-based supermarket games (Stelios et al., 2017) and vir- tual road exploration (Lesk et al., 2014). Language disorders have diagnostic methods in VR such as linguistic data related to arbitrary landscapes (Wid- mann, 2012) and language expression and comprehension in virtual environ- ments (Montenegro and Argyriou, 2017). Intermittent memory impairment ability has the potential to be evaluated through virtual alleys (Bellassen et al.,
Cyber Rehabilitation Study for the Elderly With Mild Cognitive Impairment 305
2012) and virtual driving (Plancher et al., 2012), indicating that unconscious insensitivity to temporary memory impairment and spatial memory impair- ment could be biomarkers for determining latent Alzheimer’s (Silvia and Riva, 2016). VR-based neuropsychological evaluations and research are limited by the subjective judgment and explanations of experimenters and researchers, as well as patient and family feedback. Such methods can be difficult to use as scientific judgment criteria due to the inclusion of subjective interpreta- tions and real-time diagnosis challenges. Therefore, a quantitative evaluation method that minimizes human judgment is necessary for ideal MCI diagnosis and rehabilitation.
IMPLEMENTATION AND ANALYSIS OF A COGNITIVE REHABILITATION PLATFORM FOR MILD COGNITIVE IMPAIRMENT AND ELDERLY USERS The proposed technology has implemented a cognitive rehabilitation plat- form that allows users with MCI and the elderly to receive cognitive rehabil- itation and systematic management online. The platform has experimented with the Cognitive Impairment Screening Test (CIST) (World Health Orga- nization, 2015; Anderson 2019), real-time content control based on user responses, and monitoring of cognitive function changes according to learn- ing outcomes. Considering the convenience of manipulation for the elderly, an Android app with an intuitive interface was chosen. Data collection and evaluation were conducted with 100 elderly individuals aged 65 and above with low MCI symptoms, and all operations were performed together with a safety manager to ensure safety.
The tablets used in the experiment were Samsung Galaxy Tab S7 and a PC web camera, as shown in Fig. 1, which recorded in Full HD resolution. Table 1 is a content addition classification table defined for the elderly’s CIST. The content consisted of basic common-sense videos and multiple-choice questions. MCI diagnostic item states were divided into High, Medium, and Low. The diagnostic results were designed to be judged based on the six elements of CIST, and the MCI content type was implemented to act as a personalized service by referring to recommendations and monitoring.
Figure 1: An illustration of key components and relationships in system dynamics model.
As a result of collecting and analyzing the reactions of the elderly, the most common reactions were “gaze instability,” “drowsiness,” and “leav- ing the seat” in that order. It is difficult to check the concentration status in real-time and provide feedback for content constituted in a virtual environ- ment. This paper monitors cases where users are not in a normal viewing state while playing content, as shown in Fig 3. Real-time viewing status feed- back was added with new content playback, pause, and other content control functions. User statuses were defined as normal, unable to concentrate, drowsiness, and leaving the seat, as shown in Fig. 2.
Table 1. Overview of essential data format and factors.
Table available in the original PDF paper.
Class Points Disorientation High 4∼5 Medium 2∼3 Low 0∼1 Attention deficit High 3 Medium 1∼2 Low 0 Spatial Temporal Reasoning High 2 Medium 1 Low 0 Executive function High 5∼6 Medium 3∼4 Low 0∼2 Amnesia High 7∼10 Medium 3∼6 Low 0∼2 Language disorder High 3∼4 Medium 1∼2 Low 0
Figure 2: Identification of four categories of abnormal user behavior.
In order to minimize the computational load on the user’s device, com- plicated computations such as viewing status determination and content recommendation are all performed on the server. The viewing status
Cyber Rehabilitation Study for the Elderly With Mild Cognitive Impairment 307
determination used YOLOv5 (Manly et al., 2022) to extract the user’s face area and detect the pupils and posture in the facial area through the eyes to determine where the gaze was directed using Media pipe and Eye tracking. It consisted of normal, gaze instability, drowsiness, and leaving the seat. If the front of the face was detected normally, it was judged as normal (face). Cases with no detected objects were considered as leaving the seat, bowing head as drowsiness (sleep), and turning head sideways as gaze instability (side). At this time, if the detected pupils were biased to the left or right, it was treated as gaze instability. Hands were recognized as ‘hand’ and cases where the Bounding Box(bbox) of the hand overlapped with that of face, sleep, and side were considered as drowsy.
In order to minimize user manipulation, new content recommendations and video control were configured to be performed through pre-conducted MCI screening results and user response determination, as shown in Fig 3.
Figure 3: Comprehensive visualization of the integrated system.
New content recommendations created an appropriate playlist by referring to the previously tested MCI test results. The device receives the video list from the server and displays the playlist. Video control automatically moves on to the next video when the currently playing content ends and forcibly pauses the content when the user’s status is gaze instability or leaving the seat. When the user’s status is identified as drowsy, the currently playing content is stopped, an active content is randomly selected, and played immediately.
EXPERIMENTS A total of 10,000 original training data images were collected, and data aug- mentation processes such as horizontal flipping, brightness adjustments, and random cropping were performed to obtain the training dataset shown in Fig. 4.
The elements for evaluating a classification model can be defined through the relationship between the actual answers and the predicted results of the trained model, as shown in Fig. 5. The components of the evaluation metric
are divided into true or false for the actual answers and positive or negative for the predicted results.
Figure 4: Training datasets examples.Figure 5: Confusion matrix.
The evaluation metrics consist of True Positive (TP), False Positive (FP), False Negative (FN), and True Negative (TN). FN and FP are defined as Type 1 error and Type 2 error in hypothesis testing. Type 1 error occurs when the null hypothesis is rejected when it should not be, and Type 2 error occurs when the null hypothesis is not rejected when it should be. These errors are used as elements for calculating precision and recall.
Precision, as shown in (1), is the ratio of the actual True cases among those classified as True by the model. For example, it can be considered as a metric to look at whether the actual weather was clear when the weather prediction model predicted it would be clear. Recall, as shown in (2), is the ratio of the cases predicted by the model as True among the actual True cases. For example, predicting only clear weather when the weather is indeed clear can be considered effective.
Equation reproduced from the original PDF.
Cyber Rehabilitation Study for the Elderly With Mild Cognitive Impairment 309
The Average Precision (AP) curve is a comprehensive evaluation metric that considers both precision and recall. In fact, AP is the average precision for every corresponding recall between 0 and 1. Since 2010, the calcula- tion method has changed in the PASCAL VOC challenge, and now 11-point interpolation or interpolating all data is used (Everingham et al., 2010). Com- paring the same plots is not easy because the curve has severe ups and downs and often intersects.
There are various versions of the YOLOv5 model that recognize human face areas on the server-side. Considering the image size and recognition efficiency, we targeted YOLOv5m, YOLOv5l, and YOLOv5x for the cog- nition rehabilitation platform and randomly divided 150,000 datasets into train:val:test at a ratio of 8:1:1. The evaluation metrics for this study were chosen as precision, recall, mAP@0.5, and mAP@0.5:0.95. As shown in Fig 6, we performed 150 epochs of training, and the models were trained to a level where overfitting would not occur. The validation results, there was no signif- icant difference in performance among the models, so we chose YOLOv5m, which has the lowest parameter requirements.
Figure 6: YOLOv5m training and validation results.
YOLOv5m’s confusion matrix, as shown in Fig 7, could appropriately dis- tinguish the facial areas, and most of them met at least 0.94, while sleep had a slightly lower accuracy of 0.87. To compensate for this, we changed the criteria to judge sleep when facial recognition was possible, but eye-tracking was not.
During the 20 validation processes, we tested user response judgment, con- tent control function based on response judgment, and content matching with one appropriate function for each reaction in the video data per mea- surement. User response judgment accuracy was achieved at 90% or higher, content control function accuracy was 95%, and content matching suitability was 90% or higher. However, most elderly people needed assistance due to their unfamiliarity with tablet PC operation. Nevertheless, some elderly peo- ple learned the tablet PC operation method and even progressed to operate smartphones independently. The elderly who could operate smartphones by themselves experienced a significant increase in life satisfaction.
Figure 7: Visualizing the multiclass confusion matrix for enhanced model assessment.
CONCLUSION
The WHO predicts that the proportion of the elderly aged 60 and over will double from 12% to 22% by 2050, and the Alzheimer’s Association in the United States reports that 12-18% of the population aged 60 and over suffer
Cyber Rehabilitation Study for the Elderly With Mild Cognitive Impairment 311
from MCI. In relation to this, the NIH reports that 10-15% of MCI patients progress to Alzheimer’s disease dementia. Computer-based game rehabilita- tion therapy has been proven to help cognitive rehabilitation and has been successfully studied. This study designed and tested a cyber rehabilitation therapy platform applying computer vision technology for MCI patients.
The proposed cognitive rehabilitation platform collected data from 100 subjects aged 65 and over, targeting mild cognitive impairment and elderly users. After analyzing the users’ responses, the most common reactions were gaze instability, drowsiness, and absence. The platform was designed to min- imize user manipulation and provide personalized services using automatic content recommendation and control functions.
Through this study, it was confirmed that tablet PCs can have a positive effect on elderly MCI patients and can serve as an opportunity to teach elderly individuals who are inexperienced in operating tablet PCs or smartphones how to use them.
ACKNOWLEDGMENT
This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the Grand Information Technology Research Center sup- port program (IITP-2024-2020-0-01791) supervised by the IITP(Institute for Information & communications Technology Planning & Evaluation.
REFERENCES
“Analysis of treatment status for dementia and mild cognitive impairment on the occasion of ‘Dementia Overcoming Day’,” Korea HEALTH INSUR- ANCE REVIEW & ASSESSMENT SERVICE, vol. 21, 2020. Source: https://www.hira.or.kr/bbsDummy.do?brdBltNo=10146&brdScnBltNo=4&page Index=1&pgmid=HIRAA020041000100 “Mild cognitive impairment (MCI),” MAYO CLINIC, Source: https: //www.mayoclinic.org/diseases-conditions/mild-cognitive-impairment/diagn osis-treatment/drc-20354583 “What Is Mild Cognitive Impairment?,” National Insitute on Aging, 2021. Source:
https://www.nia.nih.gov/health/what-mild-cognitive-impairment B. Buss, “Virtual reality training system for patients with dementia,” ETH, Swiss Federal Institute of Technology, Institute of Neuroinformatics, MS thesis, 2009. C. Ansuini, A. C. Pierno, D. Lusher, and U. Castiello, “Virtual reality applica-
tions for the remapping of space in neglect patients.” Restorative Neurology and Neuroscience, vol 24, no. 4-6, pp. 431–441, 2006. C. T. Lin, H. Z. Lin, T. W. Chiu, C. F. Chao, Y. C. Chen, S. F. Liang, and L. W. Ko, “Distraction-related EEG dynamics in virtual reality driving simulation.” 2008 IEEE International Symposium on Circuits and Systems (ISCAS), IEEE, 2008. Catherine N. Widmann, Ulrike Beinhoff and Matthias W. Riepe, “Everyday memory
deficits in very mild Alzheimer’s disease,” ELSEVIER Neurobiology of aging, vol. 33, no. 2, pp. 297–393, February 2012. D. Tost, A. V. Barnnekow, E. Felix, S. Pazzi, S. Puricelli, and S. Bottiroli, “Early detection of cognitive impairments with the smart ageing serious game,” ICTs for Improving Patients Rehabilitation Research Techniques: Second International
Workshop, REHAB 2014, Oldenburg, Germany, May 20–23 2014, Revised Selected Papers 2. Springer Berlin Heidelberg, 2015. D. Y. Hwang, S. H. Ryu, K. H. Kwon, C. R. Choi, and S. A. Kim, “Correlation Between Cognitive Impairment Screening Test (CIST), Korean-Mini Mental State Examination, (K-MMSE∼2) and Clinical Dementia Rating (CDR) of Patients with Stroke,” Therapeutic Science for Rehabilitation, vol. 11, no. 2, pp. 53–62, 2022. Dani Tost, Ariel von Barnekow, Eloy Felix, Stefania Pazzi, Stefano Puricelli and
Sara Bottiroli, “Early detection of cognitive impairments with the smart age- ing serious game,” Communications in Computer and Information Science book series, vol. 515, pp. 183–195, May 2014. Dong Wen, Xifa Lan, Yanhong Zhou, Guolin Li, Sheng-Hsiou Hsu and Tzyy-Ping Jung, “The study of evaluation and rehabilitation of patients with different cog- nitive impairment phases based on virtual reality and EEG,” Frontiers in aging neuroscience, vol. 10, pp. 1–6, April 2018. E. Kalova, K. Vlcek, E. Jarolimova, and J. Bures, “Allothetic orientation and
sequential ordering of places is impaired in early stages of Alzheimer’s disease: corresponding results in real space tests and computer tests,” Behavioural brain research, vol. 159, no. 2, pp. 175–186, 2005. G. Burdea, B. Rabin, D. Rethage, F. Damiani, J. S. Hundal, and C. Fitzpatrick, “BrightArm™therapy for patients with advanced dementia: a feasibility study,” 2013 International Conference on Virtual Rehabilitation (ICVR). IEEE, 2013. G. Plancher, A. Tirard, V. Gyselinck, S. Nicolas and P. Piolino, “Using virtual reality
to characterize episodic memory profiles in amnestic mild cognitive impairment and Alzheimer’s disease: influence of active and passive encoding,” ELSEVIER Neuropsychologia, vol. 50, Issue 5, pp. 592–602, April 2012. H. Sauzeon, B. N. Kaoua, P. A. Pala, M. Taillade, S. Auriacombe, and P. Guitton, “Everyday-like memory for objects in ageing and A lzheimer’s disease assessed in a visually complex environment: The role of executive functioning and episodic memory.” Journal of neuropsychology, vol. 10, no. 1, pp. 33–58, 2016. J. J. Manly, R. N. Jones, K. M Langa, L. H Ryan, D. A. Levine, R. M. Cammon,
S. G. Heeringa, and D. Weir, “Estimating the prevalence of dementia and mild cognitive impairment in the US: the 2016 Health and Retirement Study harmo- nized cognitive assessment protocol project,” JAMA neurology, vol. 79, no. 12, pp. 1242–1249, 2022. J. M. F. Montenegro, and V. Argyriou, “Cognitive evaluation for the diagnosis of Alzheimer’s disease based on turing test and virtual environments,” Physiology & behavior, vol. 173, pp. 42–51, 2017. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified,
Real-Time Object Detection,” Computer Vision and Pattern Recognition (CoRR), 2016. Juan Manuel Fernandez Montenegro and Vasileios Argyriou, “ Cognitive evaluation for the diagnosis of Alzheimer’s disease based on turing test and virtual environments,” ELSEVIER Physiology & behavior, vol. 173, no. 1, pp. 42–51, May 2017. M. Manca, F. Paterno, C. Santoro, E. Zedda, C. Braschi, R. Franco, and A. Sale,
“The impact of serious games with humanoid robots on mild cognitive impair- ment older adults,” International Journal of Human-Computer Studies vol. 145, no. 102509, 2021.
Cyber Rehabilitation Study for the Elderly With Mild Cognitive Impairment 313
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn and Andrew Zisserman, “The pascal visual object classes (voc) challenge,” Interna- tional journal of computer vision, vol. 88, pp. 303–338, 2010. N. D. Anderson, “State of the science on mild cognitive impairment (MCI),” CNS
spectrums, vol. 24, no. 1, pp. 78–87, 2019. N. J. Sofroniew, Y. A. Vlasov, S. A. Hires, J. Freeman, and K. Svoboda, “Neural coding in barrel cortex during whisker-guided locomotion.” Elife 4, vol. e12559, 2015. S. Baumann, C. Neff, S. Fetzick, G. Stangl, L. Basler, R. Vereneck, and W. Schneider,
“A virtual reality system for neurobehavioral and functional MRI studies,” CyberPsychology & Behavior, vol. 6, no. 3, pp. 259–266, 2003. S. J. Huber, E. C. Shuttleworth, and D. L. Freidenberg, “Neuropsychological differences between the dementias of Alzheimer’s and Parkinson’s diseases,” Archives of Neurology, vol. 46, no. 12, pp. 1287–1291, 1989. S. L. Allison, A. M. Fagan, J. C. Morris, and D. Head, “Spatial navigation in preclini-
cal Alzheimer’s disease,” Journal of Alzheimer’s Disease, vol. 52, no. 1, pp. 77–90, 2016. S. M. Lee, J. H. Chae, L. S. Sung, S. J. Lee, S. B. Moon, D. H. Park, and S. H.
Park, “A study on the perception of occupational therapy majors on Cognitive Impairment Screening Test (CIST),” Journal of Korean Clinical Health Science, 9(2), 1493–1501. Serino, Silvia, and Giuseppe Riva, “The proactive self in space: How egocentric and allocentric spatial impairments contribute to anosognosia in Alzheimer’s disease,” Journal of Alzheimer’s Disease, vol. 55, no. 3, pp. 881–892, December 2016. Valeria Manera, Emmanuelle Chapoulie, Jérémy Bourgeois, Rachid Guerchouche,
Renaud David, Jan Ondrej, George Drettakis and Philippe Robert, “A feasibility study with image-based rendered virtual reality in patients with mild cognitive impairment and dementia,” PLOS One, vol. 11, no. 3, March 2016. Valerie E. Lesk, Syadiah Nor Wan Shamsuddin, Elizabeth R. Walters and Hassan Ugail, “Using a virtual environment to assess cognition in the elderly,” Springer Virtual Reality, vol. 18, pp. 271–279, September 2014. Virginie Bellassen, Kinga Iglói, Leonardo Cruz de Souza, Bruno Dubois and Laure
Rondi-Reig, “Temporal order memory assessed during spatiotemporal navigation as a behavioral cognitive marker for differential Alzheimer’s disease diagnosis,” ournal of Neuroscience, vol. 32, no. 6, pp. 1942–1952, February 2012. World Health Organization, “World report on ageing and health,” World Health Organization, 2015. Source: https://www.who.int/news-room/fact-sheets/detail/a geing-and-health Zygouris Stelios, Ntovas Konstantinos, Giakoumis, Dimitrio, Votis Konstantinos,
Doumpoulakis Stefanos, Segkouli Sofia, Karagiannidis Charalampos, Tzovaras Dimitrios and Tsolaki Magda, “A preliminary study on the feasibility of using a virtual reality cognitive training application for remote detection of mild cognitive impairment,” Journal of Alzheimer’s Disease, vol. 56, no. 2, pp. 619–627, January 2017.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-95-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional