Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Segmentation of Augmented Reality 3D Meshes to Discover In Home Safe Walking Spaces for Older Adults

Department of Computer Science, Gonzaga University, Spokane, WA, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1005711

Abstract

Falling continues to be a significant risk factor for older adults and other mobility limited individuals. Around 16% of all US adults 65 and over fell at least once in a 3 month window, and falling affects 50% of adults over the age of 80. These falls account for 0.1% of all US healthcare costs, or about $4 billion per year. Falls are the number one reason older adults move from independent living to long term care, so any reduction in their frequency has significant benefits to our communities. Around 55% of all fall related injuries occur within the home, and 35% to 40% of those from environmental conditions such as clutter, slippery floors, narrow walkways, and poor lighting. Monitoring and maintaining clear, tripping hazard free pathways in living spaces is invaluable in helping people live independently and safely in their home. This paper proposes and demonstrates a space segmentation approach to discovering which areas of the home are classified as ‘walkable’ in a safe manner. The goal is to then provide residents and caregivers guidance about reducing in-home environmental risk factors for falls. The system leverages 3D mesh-based maps of the home gathered by the Microsoft HoloLens v1's sensors. The algorithm was designed to work with many different kinds of sensor platforms that provide mesh-style maps of spaces, such as LiDAR, vision-based solutions, and other Augmented Reality platforms. The collection of algorithms needed for the system were collected into the Walking Spaces Algorithm (WSA) package. These algorithms first used a denoising floor detection algorithm and a waterfall-based furniture and clutter detection to handle the noisy raw mesh data to help identify uncluttered floor spaces. The resulting reduced noise 3D model of the home was then processed using a segmentation algorithm to find pathways that would be considered safe under the United States Occupational Safety and Health Administration (OSHA) guidelines. The home's safe and unsafe spaces were then visualized for users to see and understand where the home is safe and accessible or not. To test the WSA algorithm, data was collected from both lab and apartment spaces for testing. All data collected was stored on a database backend with a web frontend to see real time updates of the home as the resident carried the AR headset through the space, with final images being generated on a daily basis for long term analysis as the home's space changed due to clutter or furniture movements. The long term goals of these technologies are to monitor the living space’s clutter and clear walkways over time. The information it generates shall then be provided to the residents and their caregivers during environmental home assessments. It informs them about how well the home is being maintained so proactive interventions may be taken before a fall occurs. Overall, the results from the WSA algorithm in the testing spaces were successful at identifying open vs. cluttered spaces and allowed the algorithm to demonstrate where residents had OSHA-rated clear walking spaces in their home.

INTRODUCTION

Seniors most frequently encounter falls within their own residences, with approximately 55% of all fall-related injuries transpiring at home (Pynoos, Steinman, & Nguyen, 2010). Notably, between 35% and 40% of these incidents stem directly from environmental conditions. Therefore, the medical field places significant emphasis on innovating strategies to prevent, identify, and aid in the management of falls.

During prior work gathering clinical needs for senior care focused smart home technologies, nurses often cited the need for help with several common topics. These areas included medicine compliance, falls, and diet (Zulas & Crandall, 2014), (Zulas & Crandall, 2017). When it came to falls, nurses normally focused on two major targets: clutter on the floors and hoarding.

Moreover, relatives of elderly individuals recounted addressing stumbling risks, ensuring safe indoor mobility, and maintaining household tidiness (Zulas & Crandall, 2014). Enhancing our understanding of seniors’ living

© 2024. Published by AHFE Open Access. All rights reserved. 1434

Segmentation of Augmented Reality 3D Meshes to Discover In-Home 1435

conditions constitutes a pivotal aspect of furnishing high-caliber, enduring in-place aging care (Kaye et al., 2007). Consequently, there arises a possibility for intelligent home and sensor technologies to amass, scrutinize, conceptualize, and convey data to caregivers regarding a senior’s well-being and condition. The ultimate objective of gerontechnological tools within home settings is to supply actionable insights to caregivers, enhancing both the seniors’ autonomy and the quality of caregiving.

While the field of fall prevention is well studied, most of the medical focus has been on clinical, medicine, and assistive tools. These are well summarized by the American and British Geriatrics Society’s works on the topic (American Geriatrics Society, 2001). The area that receives a relatively small amount of work is in tools for facilitated environmental home assessments. A home assessment is performed by a caregiver, clinician, or care facility manager to identify tripping hazards such as loose rugs, lack of grab bars in high-risk locations, and overly narrow walking paths throughout the home. In an ideal world, an assessment would be performed regularly, though that is often cost prohibitive.

To address the problem of clutter and safe walking spaces, the authors implemented a system leveraging in-home 3D mesh models built by a HoloLens augmented reality headset (Weinmann, Wursthorn, Weinmann, & Hübner, 2021). The 3D mesh built by a HoloLens provides a spatial mapping of a home through a combination of infrared, vision, and depth mapping algorithms. Under good conditions, the HoloLens mesh is accurate to centimeters allowing the detection of objects that could be classified as tripping hazards around the room (Khoshelham, Tran, & Acharya, 2019).

This work summarizes the implementation of a system centered around identifying safe walking spaces in homes. It uses several algorithms to handle the complexity and noise in the HoloLens 3D mesh output to segment the floor into categories. The results are analyzed for safe paths based around the US Occupational Safety and Health Administration suggested safe walking path regulations (United States Department of Labor, 2022). In this case, it means that a path needs to be at least 0.46m (18in) wide for safe passage. This value can easily be changed to analyze the room for the 0.92m (36in) required by the Americans with Disabilities Act if the older adult is more mobility impaired or needs more room for assistive tools such as a walker or wheelchair. When performed iteratively over time, the net result creates a history of how clean and orderly the home’s walking spaces being maintained. This information could be leveraged by caregivers to determine if interventions are needed to help remove tripping hazards from an older adult’s home (Fritz, Schmitter-Edgecombe, Crandall, & Cook, 2017) to help foster successful aging in place for older adults.

RELATED WORKS Addressing the issues of falls and falling has been an ongoing effort in the research community. The range of approaches includes critical situation detection, early detection, and prevention. All of these works have a place during the arc of care.

1436 Crandall

A similar field to this work is the one of room classification and segmentation to identify room uses and objects. For example, Guo and Fan (Guo & Fan, 2022) use pixel-level labeling of in-home images to identify the use of objects and rooms. Other algorithms do the room classification using just the home’s floor plan (Paudel, Dhakal, & Bhattarai, 2021) which could be generated automatically from the HoloLens’ 3D mesh, but their algorithms would not take into account the current clutter or furniture placements.

There is a relative dearth of work on automatic environmental home assessment, though there are semi-autonomous smart home approaches such as clinician-in-the-loop (Fritz, Schmitter-Edgecombe, Crandall, & Cook, 2017). The technologies required to perform 3D model-based assessments are very new. This work addresses part of this needed area of information gathering, analysis, and presentation surrounding home safety.

METHODS

To build and evaluate a system for finding safe walking spaces, several system components were designed and used. These components included a sensor package on a HoloLens version 1, an augmented reality interface, and a Linux server with a database and web server.

The sensor source was a Microsoft HoloLens version 1. This device runs a Unity AR application to start, stop, and upload mesh data from the home. The application copies the collected data to a Linux server for storage. The server can be broken down into three subsystems: data storage, Walking Spaces Algorithm (WSA), and web services. The data storage subsystem manages all of the data movement and user notifications. The WSA tools do the primary parsing, analysis, and generation of information for each 3D mesh uploaded by the HoloLens. The website offers a way for the end-user to view and interact with the data.

Figure 1: HoloLens mesh visualization and in-lab data collection. (a) Mesh over objects. (b) HoloLens worn for data collection.
Original PDF Figure 1:

For this project, all 3D meshes are stored in the Wavefront object file format (.OBJ), which is readily usable by Unity on the HoloLens. This

Segmentation of Augmented Reality 3D Meshes to Discover In-Home 1437

data provides vertex data from the estimated surfaces discovered by the HoloLens as it scanned the home. The values include geometric vertices, texture vertices, vertex normals, and parameter space vertices. Together, these points provide a set of triangular surfaces that roughly correspond to the 3D space. An example of the 3D mesh overlayed on some objects scanned by the HoloLens is shown in Figure 1a. The room used to perform testing of the WSA segmentation algorithm and how the HoloLens looks when worn by a user is shown in Figure 1b.

The WSA algorithms subdivide the complete 3D mesh. Each of these meshes are designed to identify particular pieces of interest to a user. Notably, outputs include level walking spaces (floors), raised working spaces like tables, and identified clutter on the floors.

Based on these mesh outputs we are able to identify a subset of the total estimated floor space that pass OSHA requirements as a safe walkway. These regions can then be highlighted with various colors to show degrees of walkable spaces. The highlighted images are stored on the web server for the user to access and review at a later time.

Processing the initial OBJ file has several steps to reduce noise and unnecessary information. A first step is to remove the ceiling and all mesh above 2m.

To identify the likely floor, all triangles are analyzed for their normal to the Z (vertical) axis. Triangles with a normal with a divergence greater than 1 degree (min2) are removed. Of the remaining set, the triangles are clustered into groups representing likely bands of true floor height. The triangles in the lowest height group are averaged and any triangles below that average are considered noisy and removed. Of the remaining triangles, they represent a highly accurate elevation for the primary floor height within the room.

Given the likely floor height, a ground plane is established and the mesh can be subdivided into open floor spaces and non-open spaces. This is done by a waterfall algorithm (Hanbury & Marcotegui, 2006). “Droplets” of simulated water are dropped from the highest elevation in the 3D space occupied by the mesh. Any droplets that reach the ground level without touching a triangle on route are considered to have reached a clear floor space. The triangles in that droplet’s landing zone are considered a clear walking space and included in the clear walking space mesh. Any droplets that stop before reaching the ground plane add the triangle they impacted into the “other objects” category.

δ2 = abs(floor_angle_neighbor_angle) (1) neighbor_class = clutter, min2 < δ2 < max2 furniture, otherwise. (2)

1438 Crandall

Figure 2: Visualization of target segmentation behavior. (a) Segmentation parameters and settings. (b) Expected segmentation behavior in a simple room.
Original PDF Figure 2:

The edges of the walking spaces are also classified as either clutter or larger objects by comparing all triangles who are members of the floor group with their neighbors. The class of a given neighbor is determined by first calculating the δ2 between floor and neighbor as shown in Equation 1. The neighboring tile is then classified according to Equation 2. These additional classifications are kept in the OBJ file for rendering in the resulting images for the user.

Post WSA algorithm completion, the server has multiple OBJ files, each with subsets of the original 3D mesh. These subsets are viewable individually to see the calculated walking spaces, raised work surfaces, calculated clutter in the room, and other objects. The resulting walking spaces mesh is then analyzed for valid OSHA walking space open areas via the WSA space segmentation algorithm. For this work, each 0.15m2 of floor space is processed as a potential open and safe walking space. Each 0.15m2 of space is considered the center of a circle with a radius of 0.46m (18in), and if that point’s circle is entirely on the flat floor area, then the 0.15m2 square is colored green in the final visual renderings. Any location that is on the floor, but does not have a full 0.46m radius circle around it of open space is colored yellow, and if it only has 10cm or less of flat floor area, it is colored red. These distance settings and rough expected buffer zones from walls and objects are shown in Figure 2a.

To demonstrate the way a room should be segmented around placed objects, Figure 2b shows a room furnished simply with a chair, TV/table, and a couch. The room is segmented and colored in with the expected open green OSHA-safe rated walking areas, red 10cm edges, and yellow buffer areas. This quickly highlights to a person viewing maps of the home what is an OSHA approved pathway or not. It also shows that even though areas might be open, that openness might not be contiguous. The residents should be able to safely move around in the area above and to the left of the chair, but not transition without hazard to the main area in front of the couch because the gap between chair and the TV/table is too narrow. Future work should start using these kinds of processed maps in conjunction with pathing algorithms to build connectivity maps based on the safe walking segments of the home.

The Walking Spaces system was evaluated in two homes, a lab at a university, and a student apartment. In every case the 3D mesh was collected

Segmentation of Augmented Reality 3D Meshes to Discover In-Home 1439

and archived for later processing. The results were reasonable in each case, though the HoloLens has significant trouble with darker surfaces, windows, mirrors, and small objects under about 7.5cm (3in) in height. Future systems, both through sensor and 3D mesh creation improvements, should reduce these limitations.

This approach has several limitations and assumptions. The current algorithms process a home as a single unit, so individual rooms are not segmented for notation or exclusion by the user if they’re not to be included in an analysis. Stairs are not considered since all homes and spaces used were single story locations. The Unity application used to do the collection was designed for simple sampling and not continuous collection.

Evaluation of the algorithms used was done by inspection by a researcher at all stages of processing. The researcher used photos taken from the homes as ground truth. The researcher was able to inspect the resulting mesh’s match to floor, clutter, raised surfaces, and walking spaces for accuracy.

The data collection process also measured open floor spaces where a simple map was drawn of the space and main walking space widths were marked. These marked up maps allowed the researcher to identify where the algorithms were reasonably correct or not.

RESULTS

The WSA algorithm and resulting visualizations were presented to the researchers and users via a web interface that pulled the drawings from the MongoDB back end. These were able to show important details about the living spaces already scanned and processed. A caregiver, nurse, family member, or the residents themselves can readily see if there are safe walking paths through the home. The resulting 3D meshes should also be analyzable by a simple shortest path algorithm such as A* to determine if the resident is able to safely move from key locations such as the bedroom to the bathroom through OSHA or ADA compliant routes.

Figure 3: Home #1 detected floor space from WSA algorithm.
Original PDF Figure 3:

1440 Crandall

Data was collected from several homes, apartments, and labs on campus. The WSA algorithm was used to denoise, detect objects, and segment the floor. The results from each location were very similar.

Figure 4: WSA floor clutter output for Home #1.
Original PDF Figure 4:

WSA Floor Mesh Output: The mesh shown in Figure 3 is the floor of Home #1. These are the OBJ file triangles found to be oriented normally to the ground plane with noisy triangles removed. The large open areas missing from the mesh are furniture, cabinets, walls, clutter, and a few areas not detected well by the HoloLens.

WSA Clutter Mesh Output: The clutter objects discovered by the WSA are shown in Figure 4. In this home, there were several piles of objects around the edges of many rooms, and especially the entry hallway on the right. These piles were made of books, clothes, and hobby items. The WSA did identify many of these piles properly, though there were several regions of false positives.

Figure 5: WSA OSHA walking space edges found for Home #1.
Original PDF Figure 5:

Segmentation of Augmented Reality 3D Meshes to Discover In-Home 1441

WSA OSHA Walking Space Outlines from Home #1: The final output of the WSA tools is to generate the edges of the OSHA width walking spaces found. The outlines found for Home #1 are shown in Figure 5. These areas are outlines of the locations that had under 0.46m radius of open spaces. The enclosed sections are then spaces considered “safe” walking areas by the WSA algorithm.

When reviewing the ground truth photos and measurement sketches taken during the data collection, the researcher felt that the WSA output was relatively accurate for the homes measured. The open areas were often narrower than expected, though the open routes were correct. More work would be needed to generate quantified values for the accuracy of the Walking Spaces tool when measuring home living spaces.

Segmented Walking Spaces Visualization: After the WSA tools process the mesh for walking spaces, clutter, and raised surfaces the resulting meshes are used to generate user-facing visualizations. These renderings use the various WSA segmentation-based classifications to color code a 3D model of the home. The renderings are centered around the availability of different segments of the walking space categories. These renderings shall be presented to caregivers and clinicians responsible for older adults aging in place. A longer-term goal would be to feed the segmentation data back to the HoloLens to provide real time safe vs unsafe walking spaces, or even suggestions to make the room safer for daily mobility-based activities.

Figure 6: Visualization of discovered and OSHA-rated walking spaces for Home #1.
Original PDF Figure 6:

In Figure 6, the various open, OSHA width, walking spaces and the raised work surfaces are colored together. Green shows where a person would have a full OSHA-qualified walking space to move in. The yellow show the floor where it is not a full OSHA width, and a final red band around the edge where only about 10cm of floor space is visible. Orange and purple colors show where clutter is resting on the floor or the “other” objects up high.

1442 Crandall

Figure 7: Visualization of discovered and OSHA-rated walking spaces for Apartment #1.
Original PDF Figure 7:

In Home #1 there are some relatively good walking paths, though there are objects making narrow points in the room. Notably, the orange-colored object is a beanbag chair sitting near a couch with makes the routes in the room too narrow for OSHA standards. An environmental home assessment would strongly suggest removing the chair in the room to open up safer paths through the space. The other suggestions would include re-arranging the dining room table behind the couch to open up the walking space from the front door to the middle of the living space. These are the kinds of objects and furniture arrangements that contribute to the hazards noted in Pynoos et al.’s summary of environmental hazards (Pynoos, Steinman, & Nguyen, 2010).

In contrast to Home #1, Apartment #1’s walking spaces were less constricted. The final visualization, as shown in Figure 7, has a notably larger green area for clear OSHA width walking spaces. Residents of the apartment are able to enter through a nearly fully compliant doorway on the right, and reach most of the living space without hazards.

Overall, the WSA tools, WSA denoising and segmentation algorithms were able to generate accurate visualizations of safe vs. unsafe walking spaces. This system could be done in real time with more work to run the segmentation algorithm on the HoloLens itself, though at a significant cost to battery life. There were notable limitations based around the 3D mesh noise in some lighting and texture situations in the homes, but the overall results were clean and usable as a prototype for future work.

CONCLUSION

The combination of the 3D mesh generated by the HoloLens v1 and the WSA tools was able to generate accurate annotated visualizations of homes and work spaces. These tools were able to create images that accurately represented open walking spaces and cluttered areas. The researcher was able

Segmentation of Augmented Reality 3D Meshes to Discover In-Home 1443

to review the resulting images and map them onto the living spaces they were collected from The algorithms worked effectively in homes, apartments, and workspaces. Objects larger than about 35cm in any dimension were sometimes detected, and all non-black colorful objects over 50cm were reflected in the mesh. This should be sufficient for most kinds of clutter detection in day to day living spaces.

Future work would include adding annotations of the visualizations for connected spaces, more direct connection between target segmentation and generated segmented maps, locations of notable clutter, and alerts when an open path becomes occluded over time.

ACKNOWLEDGMENT

The authors thank Don McMahon from the Washington State University Assistive Technology Research and Development Lab (ATR&D Lab) for the use of HoloLens equipment. Special thanks to Team Hekate for their hard work: Konstatin Shevdov, Jacob Stocklass, Jarred Eagley, and Austin Craigie.

REFERENCES

American Geriatrics Society. (2001). Guideline for the Prevention of Falls in Older Persons. Journal of the American Geriatrics Society, 49(5), 664–672. Fritz, R., Schmitter-Edgecombe, M., Crandall, A. S., & Cook, D. J. (2017). Health-

assistive smart homes with a clinician-in-the-loop. Innovation in Aging, 1. Guo, L., & Fan, G. (2022). Holistic indoor scene understanding by contextsupported instance segmentation. Multimedia Tools and Applications, 81(25), 35751–35773. Hanbury, A., & Marcotegui, B. (2006). Waterfall segmentation of complex scenes.

Asian Conference on Computer Vision (pp. 888–897). Springer. Kaye, J., Hayes, T., Zitzelberger, T., Yeargers, J., Pavel, M., Jimison, H., et al. (2007).

Deploying wide-scale in-home assessment technology. Technology and aging, 21, 19–26. Khoshelham, K., Tran, H., & Acharya, D. (2019). Indoor mapping eyewear: geometric evaluation of spatial mapping capability of HoloLens. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 42, 805–810. Paudel, A., Dhakal, R., & Bhattarai, S. (2021). Room classification on floor plan

graphs using graph neural networks. arXiv preprint arXiv:2108.05947. Pynoos, J., Steinman, B. A., & Nguyen, A. Q. (2010, November). Environmental Assessment and Modification as Fall-Prevention Strategies for Older Adults. Clinics in Geriatric Medicine, 633–644. United States Department of Labor. (2022). Occupational safety and health

administration (OSHA) law and regulations. Retrieved from OSHA: https://ww w.osha.gov/lawsregs Weinmann, M., Wursthorn, S., Weinmann, M., & Hübner, P. (2021). Efficient 3D Mapping and Modelling of Indoor Scenes with the Microsoft HoloLens: A Survey. PFG – Journal of Photogrammetry, Remote Sensing and Geoinformation Science, 89, 319–333.

1444 Crandall

Zulas, A., & Crandall, A. S. (2014). Caregiver Needs from Elder Care Assistive Smart Homes: Children of Elder Adults Assessment. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 58, pp. 634–638: Sage Journals. Zulas, A., & Crandall, A. S. (2017). Assessing professional caregiver needs in assistive

smart homes. Handbook of Smart Homes, Health Care and Well-Being (pp.121– 131). New York, NY: Springer International Publishing.

Cover of the proceedings volume Human Factors in Design, Engineering, and Computing

Explore the complete digital volume

Human Factors in Design, Engineering, and Computing

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-35-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →