Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Automated vision-based toilet assistance for people with dementia

Computer Vision Lab, TU Wien Vienna, 1040, Austria ↗
Publication year
Digital Object Identifier10.54941/ahfe1001638

Abstract

As the disease progresses, people with dementia increasingly require assistance with Activities of Daily Living (ADLs), including hygiene and toileting. In long-term care facilities, caregivers are responsible for accompanying residents to the toilet, which is not only time-consuming but not pleasant for either party. This paper presents an automated step-by-step system to guide people with mild dementia in using the toilet. The user's actions are detected and then compared to a model that specifies the correct toilet procedure, which includes toilet bowl actions and hand-washing. When inconsistencies are detected between the user's actions and the model, interaction with the user is initiated. A multimodal approach and feedback from healthcare professionals are used to tailor the interaction of the system specifically for users with dementia. Experimental results validate the functionality of the system to recognize actions performed by the user and provide suitable instructions to the user or alert a caregiver.

INTRODUCTION

Dementia progresses from a milder form, when the person’s functioning is just beginning to be affected, to the most severe stage, when the person is dependent on others to perform ADLs, including personal hygiene and toi- leting1. In long-term care facilities, residents are accompanied to the toilet by caregivers, who wait inside or in front of the toilet to intervene if neces- sary. According to a survey of 1181 dementia caregivers (Cheng, 2017), 68% identified assistance with ADLs (e.g., bathing or toileting) as one of the most burdensome activities for caregivers. In addition, Drennan et al. (2011) report a lack of acceptance by people with dementia to be accompanied to the toi- let by their caregivers, as this causes them embarrassment and discomfort. Since this is a stressful procedure for both parties that may have to be repea- ted several times throughout the day (Drennan et al., 2011), systems to assist people with dementia to use the toilet independently can be of benefit to both caregivers and the patients.

In this work, we present a vision-based system for automated toilet assista- nce for people with mild dementia. By analyzing the user’s actions on the toilet 1National Institute on Aging’s ADEAR Center https://www.nia.nih.gov/health/what-dementia-symptom s-types-and-diagnosis (accessed January 24, 2022)

© 2022. Published by AHFE Open Access. All rights reserved. 21

22 Ballester and Kampel

against a predefined model for toilet-going, the system detects abnormal beh- avior and, in case of problems, prompts the user or alerts a caregiver. Through the use of a 3D sensor, the user’s privacy is protected, and no identification of the user is possible.

RELATED WORK

Assistive Technologies for Cognition (ATC) are defined by Gillespie et al. (2012) as “any technology which assists cognitive function during task per- formance”. These authors review ATC solutions and argue that such soluti- ons can alleviate interpersonal tensions between caregivers and care recipients while increasing care recipients’ independence, their self-confidence, and the cost-effectiveness of care.

As an example of a personal hygiene ATC system for people with demen- tia, Mihailidis et al. (2007) present a system for verbally guiding people with dementia through the procedure of washing their hands. Along this line of work, Labelle et al. (2006) analyze the effectiveness of interaction modali- ties to guide people with dementia in the task of hand-washing. The study reports that participants seem to prefer the presence of the caregiver due to the sense of reassurance it provides and that the inclusion of an additional acknowledgment message (e.g., “that’s right”) when the patient performs a task correctly might have reduced the frequency of caregiver intervention. The idea of incorporating feedback into the interaction of automated systems with people with dementia is also presented by Stara et al. (2019) and by König et al. (2016). In particular, König et al. (2016) suggest avoiding excla- mations such as, “Good job!” and, instead, maintaining an acknowledging tone.

With regards to ATC specifically dedicated to toilet assistance, Lumetzber- ger et al. (2021) propose a preliminary version of a vision-based system to assist cognitively impaired users with the usage of the toilet. Ballester et al. (2021) discuss the efficacy of different interaction modalities of an automa- tic toilet assistant, based on data gathered from focus groups with healthcare professionals. Building on these works, we present a system validated by 10 users, which automatically guides the user through the entire process of using the toilet, including hand-washing.

DESCRIPTION OF THE SYSTEM The main components of the proposed system are the cogvisAI sensor, the interaction module with its corresponding model to capture the toileting process, and the selected interaction modalities.

CogvisAI Sensor The cogvis AI sensor uses Microsoft’s KINECT technology, with only the 3D sensor part and no built-in operational RGB camera. The generation of depth data is similar for all commercially available sensors (Schöning, 2016). The operation is based on the Time-Of-Flight (TOF) principle: an infrared light is emitted and the device measures for 512 × 484 pixels individually

Automated Vision-Based Toilet Assistance for People with Dementia 23

Figure 1: Samples of depth images captured in the laboratory toilet room.
Original PDF Figure 1:

the distance the light traveled before being reflected by an object back to the sensor - similar to an echo sounder. The TOF method with continuous light signal (“continuous wave”) calculates the distance of a point by the phase shift between the reflected and transmitted signal. The output of the sensor is a depth image in which each pixel is assigned a depth value. In a depth image, distances to the sensor are measured and displayed in a variety of gray tones using the captured distance data. Figure 1 shows samples of depth images captured during the experimental validation in the toilet.

Privacy and Identification Voydock et al. (1983) describe identification features as the attributes that enable individuals to be identified, and are classified as primary or secondary. Primary identification features of a person are those attributes with the help of which a person can be clearly identified. To make these attributes irretri- evable, they must either be encrypted or deleted (Shadlau, 2009). Secondary identification features are those attributes that, on their own, do not enable a person to be uniquely identified, but when combined, can lead to a clear recognition of an individual. For the identification of a person from images, the primary attributes would be their faces and the secondary attributes com- prise height, weight, and dimensions of body parts. The cogvisAI system is not capable of capturing primary identification features, as the depth sensor only provides 3D data and no brightness information, so the texture of the obje- cts in the image is not captured. Only secondary identification features are present, as depth images can be used to measure height and volume (Kouno, 2012). However, the discriminating power and calculation of, for example, weight, are not accurate for unambiguous and error-free identification.

Interaction Module The interaction module is responsible for determining which type of intera- ction should be initiated. Figure 2 shows the flow chart of the interaction module.

The process begins with an MQTT message as input from which the user’s status and action are inferred. In the event of a fall, a caregiver is alerted directly and a reassuring message is displayed to the user while the caregiver is on the way. The next step is to verify that the user’s status and action are consistent with the predefined model. In case of deviation, the corresponding instruction is given to the user. If the instruction is successful and the user performs the action requested by the system, an acknowledgment message

24 Ballester and Kampel

Figure 2: Flowchart of the interaction module.
Original PDF Figure 2:

is displayed to reassure the user that he/she has done well. Otherwise, the instruction is repeated. If after repeating the instruction Nrepeat times, the user still does not perform the expected action, an alert is sent to the caregiver informing them that the resident is having trouble using the toilet. As in the case of a fall, a reassurance message is displayed to the user to help him/her remain calm until the caregiver arrives.

Model for the Toilet-Going Procedure The model defined to represent the correct toilet procedure consists of a sequence of steps (see Figure 3) and time thresholds for the duration of the different steps. The steps in the model must be performed in this specific order and within a specified time frame in order for a user’s actions to be considered correct. Any deviation from this sequence will be interpreted as anomalous behavior. The time thresholds have been set heuristically for development purposes and should be investigated further in the future. The status (shown inside the boxes in Figure 3) is determined based on the last message publi- shed by the cogvisAI sensor, which contains information about the posture of the person (standing, sitting or fallen), the distance from the toilet, dtoi- let, and the distance from the basin, dbasin. These distances are compared to distance thresholds (Dtoilet, Dbasin) to determine if the user is at the expe- cted point, and in combination with the user’s body posture, infer the user’s status. The conditions associated with each status are listed to the left of the flow diagram in Figure 3. By combining the user’s status with a record of pre- vious actions, the system determines which step of the procedure the user is in and what the next action should be (shown on the right side of the flowchart in Figure 3).

Interaction Modalities The effectiveness of an automated assistance system ultimately depends on its ability to interact with the user. As a means of overcoming the barrier caused

Automated Vision-Based Toilet Assistance for People with Dementia 25

Figure 3: Model for the toilet-going procedure.
Original PDF Figure 3:

by the communication impairments in people with dementia, a multimodal approach is employed, as proposed by Smith et al., (2011) and Fried-Oken et al. (2015). Instructions are given verbally and are accompanied by visual support. Verbal messages are designed to be as short and clear as possible, maintaining a courteous tone, and are recorded by a female voice. In future implementations, it is envisioned that messages will be recorded by a care- giver at the care facility where the system is to be installed, as suggested by Orpwood et al. (2005). As visual support, simplified animations and icons are displayed on a screen. Figure 4 shows four samples of the visual support used in the instructions. As well as conveying the message in a complemen- tary format, visual support helps the user identify the audio source, which is a desirable feature according to König (2016).

EXPERIMENTS Experiments are conducted to validate the system’s capability to 1) recognize user actions and 2) provide suitable instructions. The former allows us to determine the reliability and accuracy of the modules for person tracking and action detection. The latter gives information on the actual capabilities

26 Ballester and Kampel

Figure 4: Samples of the icons designed as visual support to give instructions to the user.
Original PDF Figure 4:
Table 1. Description of scenarios for functional testing.

Table available in the original PDF paper.

Input Expected output No interaction from the system.

1. The user enters the room, uses the toilet, washes his/her hands and leaves the room.

No interaction from the system.

2. The user enters the room, washes his/her hands and leaves the room.

One instruction and acknowledgment message for each step (5, in total).

3. The user enters the room and delays his actions until he is guided through all the steps (approaching the toilet, sitting on the toilet, getting up from the toilet, approaching the basin and washing his hands, leaving the room).

One instruction and acknowledgment message for each step (2, in total).

4. The user enters the room and delays his actions until he is guided through all the steps (approaching the basin and washing his hands, leaving the room).

5. The user enters the room, sits on the toilet but is unable to stand up.

No interaction until the “get up” step, where 2 instructions are repeated. No acknowledgement message. A reassurance message is given. 6. Open scenario N/A

of the model to capture the different steps during the process of using the toilet.

According to Orpwood et al. (2005), only validated prototypes should be tested with people with dementia, as early-stage devices could provoke anxiety and future rejection in these users. In total, 10 cognitively healthy users participated in the tests.

Methodology

Validation of the system is conducted in a laboratory toilet room under controlled conditions. Test participants are requested to perform five fixed scenarios (as described in Table 1) and one open scenario, in which the par- ticipant is free to act as he/she wishes, to test how the system reacts to unforeseen situations. For each participant and scenario, 1 point is awar- ded for each correct recognized action and each correct interaction, and 0 points in case of error.

Automated Vision-Based Toilet Assistance for People with Dementia 27

Table 2. Average scores for action and scenario.

Table available in the original PDF paper.

Standing Next to Sit Stand Next to Hands Left the toilet down up the basin washed room 1 1 0.9 1 1 0.9 1 1 2 1 - - - 1 1 1 3 1 0.9 1 1 0.9 1 1 4 1 - - - 1 1 1 5 1 1 1 - - - - Mean 1 0.93 1 1 0.95 1 1

Results

Table 2 shows the accuracy of recognition obtained for each of the actions. The actions are correctly recognized (100% accuracy) in 5 out of 7 cases, except for “Next to the toilet” and “Next to the basin” actions (93 and 95% success, respectively). The reason for this difference is that, unlike the other actions, the recognition of these two actions is based on the user being located within the distance range.

Regarding the results of the interaction module, all interactions were correct in the 5 fixed scenarios, for all 10 participants, for the 3 types of messages: instructions, acknowledgment and reaffirmation. As for the open scenario, Table 3 shows a description of the actions performed by the participants and of system responses.

The interaction is correct in 8 of the 10 unplanned situations. No inte- raction is initiated in cases that were not planned but are not anomalies, such as drinking water from the tap (P3). Instructions are given when the user behaves erratically (P5, P7, P8, P9 and P10) and their action is ackno- wledged when they follow the instruction given. In 6 of the 10 cases, a caregiver is alerted. Of these 6, 3 are emergencies (P1, P4 and P6) while the other three are due to the user not following instructions (P2, P7 and P10). Although the intervention is correct for P7 and P10, the care- giver alert should differentiate between emergencies and erratic behaviors so as not to overload the staff. Finally, P4 is in a dangerous situation, so the system should not wait to give the instruction two times to call a caregiver.

CONCLUSION

In this work, we have presented a system to automatically guide people with mild dementia in the toilet. It is a privacy-aware system, which prevents user identification, and whose interaction has been specifically designed for people with dementia. Experimental results validated the system’s ability to detect user actions with 98% accuracy and interact accordingly with the user or alert a caregiver in 100% of fixed scenarios and 80% of unplanned sce- narios. Since this prototype has been successfully validated with cognitively healthy people, future work will involve testing the different components of the system with users with dementia.

28 Ballester and Kampel

Table 3. Description of participant’s actions and system’s interactions for the open scenario.

Table available in the original PDF paper.

P Actions Interaction OK?

No

1 The user sits down on the floor. Fall detected, caregiver alerted. Yes 2 The user sits on the toilet, stands up and sits on the toilet until the end of the test.

“Stand up”(x2) + caregiver alerted when the user is seated the second time.

Yes

3 The user drinks water from the basin faucet.

No interaction (detected as hand-washing).

Yes

“Approach toilet/basin” + “Acknowledgment message” + no interaction.

4 The user climbs onto the toilet. “Sit down” (x2), caregiver alerted. No 5 The user stands still in the center of the room, follows the instruction, gets close to the toilet, goes to the basin, washes her hands and leaves.

Yes

“Wash your hands” (x2), caregiver alerted.

6 The user sits down on the floor. Fall detected, caregiver alerted. Yes 7 The user gets close to the toilet, does not sit, approaches the basin, and goes to the corner of the room.

“Sit down” + no interaction. Yes

8 The user stands closer to the toilet, goes away and leaves.

Yes

9 The user goes to the corner of the room, follows instructions, washes his hands, stands still and after 2 instructions leaves the room.

“Approach the toilet/basin” + “Acknowledgment message” + “Leave the room” (x2) + “Acknowledgment message”.

Yes

10 The user wanders around the room. “Approach the toilet/basin” (x2), caregiver alerted.

ACKNOWLEDGMENT

This work is partially funded by JP-AAL under grant number AAL-2019–6- 150-C for the DIANA project and by the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 861091 for the visuAAL project.

REFERENCES

Ballester, I., Mujirishvili, T., & Kampel, M. (2021). “RITA: A privacy-aware toileting assistance designed for people with dementia” Proceedings of the 15th EAI International Conference on Pervasive Computing Technologies for Healthcare Cheng, S. T. (2017). “Dementia caregiver burden: a research update and critical

analysis.” Current psychiatry reports, 19(9), 1–8. Drennan, V., Cole, L. and Iliffe, S., (2011). “A taboo within a stigma? a qualitative study of managing incontinence with people with dementia living at home.” BMC Geriatrics, 11(1). Fried-Oken, M., Mooney, A., & Peters, B. (2015). Supporting communication for

patients with neurodegenerative disease. NeuroRehabilitation, 37(1), 69–87. Gillespie, A., Best, C., & O’Neill, B. (2012). Cognitive function and assistive technology for cognition: A systematic review. Journal of the International Neuropsychological Society, 18(1), 1–19.

Automated Vision-Based Toilet Assistance for People with Dementia 29

Kouno, D., Shimada, K., & Endo, T. (2012). Person identification using top-view image with depth information. 13th ACIS international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (pp. 140–145). IEEE. König, A., Malhotra, A., Hoey, J., & Francis, L. E. (2016) “Designing personalized

prompts for a virtual assistant to support elderly care home residents.” Pervasive- Health. (pp. 278–282). Labelle, K., and Mihailidis, A. (2006) “The use of automated prompting to facilitate handwashing in persons with dementia.” American Journal of Occupational Therapy 60.4: 442-450. Lumetzberger, J., Ginzinger, F., & Kampel, M. (2021). Sensor-Based Toilet Instru-

ctions for People with Dementia. International Conference on Applied Human Factors and Ergonomics (pp. 101–108). Springer, Cham. Mihailidis, A., Boger, J., Canido, M., & Hoey, J. (2007). The use of an intelligent prompting system for people with dementia. interactions, 14(4), 34–37. Orpwood, R., Gibbs, C., Adlam, T., Faulkner, R., & Meegahawatte, D. (2005). The

design of smart homes for people with dementia—user-interface aspects. Universal Access in the information society, 4(2), 156–164. Schöning, J. and Heidemann, G. (2016). Taxonomy of 3D Sensors - A Survey of State-of-the-Art Consumer 3D-Reconstruction Sensors and their Field of Applica- tions. Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP, (VISIGRAPP 2016) (pp 192-197) DOI: 10.5220/0005784801920197 Shadlau, P. (2009). Entwurf und Entwicklung einer dynamischen Prüfkompo-

nente für den Pseudonymisierungsgrad zum Zweck des Peer-to-Peer Austauschs medizinischer Daten in Forschung und Lehre (Doctoral dissertation). Smith, E. R., Broughton, M., Baker, R., Pachana, N. A., Angwin, A. J., Humphreys, M. S., ... & Chenery, H. J. (2011). Memory and communication support in demen- tia: research-based strategies for caregivers. International Psychogeriatrics, 23(2), 256–263. Stara, V., de Jong, M., Felici, E., Bolliger, D., Birrer, E., von Döllen, V., ... &

Heerink, M. (2019). “The design adaptation of the Virtual Assistant Anne for moderate dementia patients and their formal caregivers in protected environ- ment tests.” International Conference on Applied Human Factors and Ergono- mics (pp. 270–279). Springer, Cham. Voydock, V. L., & Kent, S. T. (1983). Security mechanisms in high-level network protocols. ACM Computing Surveys (CSUR), 15(2), 135–171.

Cover of the proceedings volume Human Factors in Accessibility and Assistive Technology

Explore the complete digital volume

Human Factors in Accessibility and Assistive Technology

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-13-1

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →