Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

SeeBeyond: An AI-Powered Mobile AR System for Context-Aware Color Assistance

School of New Media Art and Design, Beihang University, Beijing 100191, P. R. China ↗
Publication year
Digital Object Identifier10.54941/ahfe1007293

Abstract

People with color vision deficiency (CVD) face challenges in perceiving and distinguishing colors. Existing assistive approaches primarily rely on technical color correction and visual feature substitution. However, these methods often lack contextual awareness, intuitiveness, and universality, making it difficult to effectively support users' cognition and decision-making in everyday life. To address this issue, this study investigates the real-world needs of people with CVD through questionnaire surveys, semi-structured interviews, and situational simulations. Based on our findings, we propose a Context-Aware Color Interpretation Framework. This framework categorizes daily situations into three hierarchical levels based on task urgency and response requirements: instant decision-making, daily perception, and experience enhancement. Guided by this framework, we designed SeeBeyond, an AI-powered mobile and augmented reality (AR) system. The system integrates real-time color recognition with multi-modal feedback, providing personalized interaction adaptations tailored to the three contextual levels. By instantiating this framework through SeeBeyond, we demonstrate the feasibility of delivering context-aware, multi-modal assistance in everyday scenarios. This work shifts the focus of CVD assistive technologies from mere visual correction to holistic, context-driven cognitive support, providing a novel design paradigm for accessible interaction.

INTRODUCTION

Color vision deficiency (CVD) is one of the most common inherited visual impairments, with a prevalence of approximately 8% in males and 0.5% in females. It is primarily characterized by a reduced ability to perceive and discriminate colors. Previous research has shown that individuals with CVD are at a significant disadvantage in visual tasks that rely on color information (Simunovic, 2010). In everyday life, color is not only a visual attribute, but also an important carrier of information, widely applied across various contexts, such as daily activities, work, and transportation. For example, people use color to distinguish note contents, recognize traffic signals, assess food ripeness, and coordinate clothing. These color-based cues implicitly form a fundamental basis for everyday cognition. However, for individuals with CVD, such color-reliant information is often difficult to perceive accurately, which in turn affects their understanding of the environment.

Although various assistive approaches have been proposed- focusing on global screen color correction or visual pattern substitution, user needs vary significantly across different real-life scenarios. Such methods often treat color interpretation in isolation, overlooking the dynamic nature of human-environment interactions. Consequently, existing methods still face limitations in terms of contextualization, intuitiveness, and universality. Therefore, it is necessary to systematically investigate the needs of people with CVD from the perspective of real usage contexts and to explore more targeted assistive design strategies.

Motivated by these challenges, this study focuses on how individuals with CVD use color information in everyday life and proposes the following research questions: RQ1: What are the key user needs of people with color vision deficiency across different scenarios? RQ2: How can assistive interactions be effectively matched to different scenarios to support color-based tasks?

RELATED WORK

Assistive Methodology for People With Color Vision Deficiency Current assistive design research for people with CVD can be divided into three main directions: task-oriented design mechanisms, visual feature substitution methods, and emotional experience optimization.

Early studies proposed task-oriented design as a key approach. Cole (1972) introduced a four-level classification of color-related tasks and showed that different tasks vary in function and importance. Li and Tang (2013) further explained that user needs change across different contexts and levels of urgency. These studies suggest that assistive design should be based on task context and user needs.

Treisman and Gelade (1980) found that visual features such as shape and texture can be processed quickly through preattentive mechanisms, faster than text that requires cognitive effort. This supports the use of visual feature substitution. For example, ColourMix encodes color information into textures so that users can recognize it more easily. In situations with low urgency, semantic information can provide more precise descriptions. Applications such as Colorrect present color names and numerical values to help users identify colors in real environments.However, Zheng (2016b) pointed out that too much information can increase cognitive load. Recent studies therefore use multimodal feedback to reduce visual fatigue and improve efficiency. Visual substitution design should balance clarity and cognitive load based on task context.

In terms of emotional experience, assistive design is moving from purely functional support to a balance between usability and user experience. Norman (2004) emphasized that design should address both function and experience. Flatla and Gutwin (2013) also noted that improving color distinguishability should not remove the natural perception of color.

SeeBeyond: An AI-Powered Mobile AR System for Context-Aware Color Assistance 23 Geddes et al. (2025) reviewed existing accessibility design resources and tools and found that most work focuses on contrast enhancement and recoloring techniques. There is still limited research on context adaptation and multimodal feedback. This suggests a need for a Context-Aware Color Interpretation Framework that integrates visual substitution and emotional experience optimization.

Technology-Enhanced Assistive Approaches While existing research offers various technical assistive approaches, most remain highly function-specific and lack seamless real-world integration. Consequently, recent efforts have shifted toward developing context-aware systems designed for practical, everyday use.

Within the realm of wearable devices, AR and MR are widely used to enhance visual perception. Studies improve HMD visual systems for better color and environment recognition (Melillo et al., 2017), while systems like Vuzix Blade 2 integrate object detection, distance estimation, and OCR with auditory feedback for navigation (Kumari and Hammady, 2026). These approaches highlight the potential of AI-powered wearables to enhance independence and mobility.

Recent AI advances have shifted the focus from low-level image processing to intelligent interactive systems. While DNN-based color processing techniques provide a solid technical foundation (Afifi and Brown, 2020), they primarily emphasize algorithmic accuracy and visual enhancement, often overlooking user-centered interaction and contextual adaptability.

Furthermore, multimodal interaction integrating visual, auditory, and haptic modalities has become a key trend. In autonomous driving contexts, such designs enhance safety perception in unfamiliar environments (Meinhardt et al., 2025), improving accessibility through redundancy and cross-sensory compensation.

Some assistive technologies are already embedded in consumer products. Smartphone color filters (e.g. iPhone) can enhance color perception, while AI devices like Envision Glasses offer real-time recognition of objects, text, and scenes, along with features such as scanning, export, and video calls. By converting visual information into speech, these systems help users better understand their surroundings.

Although existing technologies have improved visual accessibility, they still have clear limitations. First, in terms of context awareness, most systems focus on single functions such as color recognition or visual enhancement, lacking support for specific real-life situations and cannot effectively assist decision-making in complex environments; Second, in terms of intuitiveness, some approaches rely on computational processes,making it difficult to provide stable and real-time feedback in time-sensitive tasks; Third, in terms of general usability, head-mounted devices have high costs and usage barriers. As a result, they are not suitable for a wide range of users and scenarios. Therefore, it remains an open challenge to develop assistive solutions that work effectively while balancing context awareness, intuitiveness, and universality.

Figure 1: Research process.
Original PDF Figure 1:

RESEARCH DESIGN This study adopts a user-centered approach to systematically explore the real needs of individuals with CVD. Mixed methods including questionnaire surveys, semi-structured interviews and situational simulation are applied (see Figure 1). The analysis is conducted across three dimensions—perceptual experience, behavioral patterns, and underlying needs, providing a foundation for the accessible design of interfaces and services.

Respondents and Recruitment Participants were recruited through online searching and social media recruitment to ensure sample diversity. A total of 40 valid questionnaires were collected, covering different types of color vision deficiency: 42.5% people with red-green CVD, 40.0% people with blue-yellow CVD, and 17.5% achromatopsia. To gain in-depth insights, this study also conducted semi- structured interviews with 17 people with CVD. The participants ranged in age from university students to working professionals in their 30s, with occupations spanning education, administration, engineering, and design. They were drawn from major metropolitan areas to rural settings, ensuring that the study reflects experiences across different levels of infrastructure and accessibility conditions.

Data Collection The situational questionnaire survey, semi-structured interview, and simulation experience were conducted in 2025 through a combination of online and offline methods. The situational simulation experience involved a two-week “life simulation” using a color vision simulator (Zheng, 2016a), during which researchers recorded difficulties arising from color perception in daily life. The questionnaire survey focused on everyday life scenarios and digital experiences, collecting participants’ evaluations of challenges in color recognition, and coping strategies. The semi-structured interview centered on core difficulty scenarios, daily coping strategies, and underlying needs, with each interview lasting approximately 30 minutes and audio recorded with participants’ informed consent.

Data Analysis During the situational simulation phase, researchers conducted immersive observations of typical visual interfaces and daily life scenarios, such as subway maps, online shopping, and festive environments, using a color vision

SeeBeyond: An AI-Powered Mobile AR System for Context-Aware Color Assistance 25 simulator (see Figure 2). This process provided a preliminary framework for designing the questionnaire and interview.

Figure 2: In scenario simulation, researchers use a color vision simulator to simulate the daily life experiences of people with CVD.
Original PDF Figure 2:
Figure 3: Survey results.
Original PDF Figure 3:

The quantitative data from the questionnaire were analyzed using descriptive statistics to identify high-frequency challenge scenarios. For the qualitative data gathered from the semi-structured interviews, all audio recordings were transcribed verbatim. We employed a thematic analysis approach to process the transcripts. Two researchers independently reviewed the texts, generated initial codes related to color-based challenges and user expectations, and subsequently grouped these codes into broader themes based on task urgency and emotional needs.

Questionnaire results (see Figure 3) show that in daily life, the main scenarios with color recognition difficulties are food assessment, transportation, and shopping selection. Users primarily rely on memory and seeking help from others, indicating that current support methods remain largely non-systematic. In app usage, applications related to daily services, entertainment, and social interaction exhibit significant color dependency issues, particularly in areas such as e-commerce product color identification, interface button, and faction identification in games. Users mainly cope by switching to assistive modes or using color-based alternatives, highlighting a clear lack of systematic accessibility support. Overall, people with CVD demonstrate a high demand for color-friendly modes, suggesting that such products have clear practical value.

According to semi-structured interviews, transportation was identified as the most high-risk scenario. Primary challenges include traffic light recognition, sign comprehension, and navigation interpretation. Notably, there is a significant regional disparity in accessible infrastructure: while metropolitan areas frequently deploy auditory or graphical traffic aids, such features are available at fewer than 30% of intersections in smaller cities and rural regions. Since these scenarios directly affect safety, participants expressed a strong need for immediate and clear assistive guidance. Food- related judgments and work/study scenarios ranked next, mainly involving the assessment of food ripeness and interpretation of chart information, highlighting a need for accuracy and efficiency in daily decision-making. In shopping and entertainment scenarios, such as coordinating outfits or experiencing festive atmospheres, users were more concerned with optimizing their overall experience and hoped to enhance their quality of life through assistive support.

Regarding coping strategies, assistance from others still dominates most offline scenarios, while users have gradually developed compensatory strategies based on memory, though these have limited applicability. One participant mentioned, “I put small labels on my clothes or use clips of different shapes to distinguish colors, so it is less likely to wear something incorrectly.” In terms of technological support, AI tools offer certain advantages in efficiency and personalization, but still face accuracy issues in complex lighting or multi-color environments. One participant noted, “Sometimes the color indicated by the app is completely indistinguishable, so I don’t know whether to trust it or rely on my own judgment.” Multi- modal assistance, such as haptic feedback, was also mentioned, providing potential directions for future design.

Findings Based on the preliminary study, people with color vision deficiency show different levels of concern and needs regarding color recognition challenges in public social and personal daily contexts. By analyzing these challenges through the dual dimensions of task urgency and user need type, we derived a framework comprising: instant decision-making, daily perception, and experience enhancement (see Figure 4).

Instant Decision refers to high-urgency scenarios that demand rapid action and possess an extremely low tolerance for error. Preliminary survey results highlight safety-critical tasks, such as traffic signal recognition, as the most stressful situations for users. Driven by a fear of making mistakes, individuals are acutely sensitive to tasks requiring split-second judgments. Consequently, success in these contexts hinges on the user’s ability to extract key information with minimal cognitive load. The primary design goal is to deliver immediate and actionable interpretation. By leveraging preattentive visual processing features (Treisman and Gelade, 1980), the system can support swift and accurate decision-making without overwhelming the user.

Figure 4: Context-aware color interpretation framework.
Original PDF Figure 4:

Daily Perception involves low-urgency routines where users need continuously process color information. Preliminary results reveal that users primarily struggle with integrating and synthesizing information from multiple color-coded sources. For instance, linking color-coded office notes to their specific meanings. Therefore, the design objective in these contexts shifts from enabling immediate action to delivering sustained perceptual assistance, prioritizing visual distinctiveness and cognitive comprehension.

Experience Enhancement involves low-stress scenarios focus on enriching users’ emotional journey. In these scenarios, the emphasis is on sensing and engaging with the broader atmosphere. Preliminary results reveal that the primary barrier for users is the inability to perceive these atmospheric nuances. Consequently, the core task is to facilitate effective immersion into a multisensory emotional experience. Because design in this context actively shapes user emotion rather than merely fulfilling functional goals (Norman, 2004), an experience-first, multimodal strategy is essential. By integrating visual, auditory, and haptic feedback, the system can profoundly strengthen the user’s emotional bond with their environment.

DESIGN PRACTICE - SEEBEYOND The core design of SeeBeyond operationalizes the Context-Aware Color Interpretation Framework to provide tailored feedback across different urgency levels. Currently, SeeBeyond is implemented as a high-fidelity interactive prototype to validate the design framework. Future iterations will integrate YOLOv8 for real-time object detection and ARCore for spatial anchoring to fully realize the proposed multi-modal interactions.

Interface Design Based on Task Levels For high-urgency scenarios such as traffic signals, brake lights, and medication identification, the design focuses on “ow cognitive load” and “rapid decision feedback.” In these tasks, SeeBeyond uses large text and prominent icons

visually, repeats voice instructions aurally, and uses vibrations to enhance haptic perception. For example, in the interface for temporary traffic lights (see Figure 5a), the system displays the large text “Proceed straight” and overlays icons on the lights, paired with voice prompts and vibrations synced with the green light rhythm. Users can rely on this intuitive results provided by the system in emergency situations to quickly make action decisions.

Figure 5: User interfaces of SeeBeyond across three contextual levels. (a) Instant Decision: High-contrast text and icon overlay for traffic lights. (b) Daily Perception: Detailed semantic labeling for cosmetics. (c) Experience Enhancement: Immersive AR effects synced with holiday audio.
Original PDF Figure 5:

For scenarios requiring diverse information, such as finding color-coded folders, matching outfits, or judging food ripeness, the design of the system aims to assist users in complex decision-making. SeeBeyond employs highlighting, graphic indicators, and personalized suggestions; additionally, users can ask for detailed color information through dialogue. For example, when judging cosmetic shades, the system provides labels and suggestions to support refined choices (see Figure 5b). This design translates subjective psychological perceptions into concrete semantic labels.

In scenarios where users prioritize emotional value and imagery resonance, such as holiday celebrations or natural landscapes, the system design aims to address the lack of atmospheric perception. By using scene recognition technology, the system triggers graphics, visual effects, and audio feedback that match the environment’s tone. For example, in a Christmas-themed square (see Figure 5c), the system plays classic holiday music and overlays stylized snowflake effects on the screen. This creates an immersive emotional experience, effectively bridging the aesthetic gaps and emotional barriers caused by limited color perception.

Multidimensional Assistive Ecosystem SeeBeyond builds a multi-modal assistive ecosystem, providing color vision assistance on the mobile interface to solve color recognition challenges in digital scenarios, such as online shopping. The system also supports

SeeBeyond: An AI-Powered Mobile AR System for Context-Aware Color Assistance 29 connectivity with mobile devices like smart glasses to enable a hands-free, continuous perception experience. Furthermore, the built-in label library enables users to digitally store personal color memories and share them with others, effectively compensating for individual perceptual limitations and forming a cohesive assistive ecosystem.

CONCLUSION AND FUTURE WORK People with CVD have significantly different assistive needs depending on their environmental contexts. This study clarifies the task characteristics and design considerations across varied daily scenarios. Based on these insights, we propose a Context-Aware Color Interpretation Framework, categorizing user needs into three hierarchical levels: instant decisions, daily perception, and experience enhancement. Guided by the framework, we constructed the SeeBeyond interactive prototype. Rather than relying on a one-size-fits- all color discrimination approach, SeeBeyond demonstrates how assistive strategies can be dynamically configured to provide context-aware, real-time, and multi-modal support.

While this study provides a theoretical foundation, it has limitations. The current sample size is limited and may not fully represent the diverse CVD demographic across different ages and regions. Furthermore, as SeeBeyond is currently a high-fidelity prototype, our immediate future work involves integrating wearable AR devices and conducting in-the-wild evaluations to assess its real-world efficacy. Ultimately, we hope this context-driven framework offers a novel reference paradigm for accessible interaction design.

REFERENCES

Afifi, M. and Brown, M. S. (2020), Deep white-balance editing, in ‘2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)’, pp. 1394–1403. Cole, B. L. (1972), ‘The handicap of abnormal colour vision’, Clinical and Experimental

Optometry 55(8), 304–310. Flatla, D. R. and Gutwin, C. (2013), Sprinkles: transforming software to improve accessibility for users with individual differences, in ‘Proceedings of the 15th international ACM SIGACCESS conference on Computers and accessibility (ASSETS ’13)’, pp. 1–8. Geddes, J. et al. (2025), ‘A comprehensive review of accessibility design resources

and tools for color vision deficiency’, Journal of Accessibility and Design for All 15(1), 45–72. Kumari, P. and Hammady, R. (2026), ‘Assisting blind people with ai and audio using smart glasses: System design with yolov8 variants comparisons’, Multimedia Systems 32(1), 73. Li, Y.-C. and Tang, H.-H. (2013), User experience design and research for the daily

life of people with color vision deficiencies, in ‘The First International Symposium of Chinese CHI’, Paris, France. Meinhardt, L.-M., Weilke, L. M., Elhaidary, M., von Abel, J., Fink, P. D. S., Rietzler, M., Colley, M. and Rukzio, E. (2025), Light my way: Developing and exploring a multimodal interface to assist people with visual impairments to exit highly automated vehicles, in ‘Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems’, pp. 1–20.

Melillo, P., Riccio, D., Di Perna, L., Sanniti Di Baja, G., De Nino, M., Rossi, S., Testa, F., Simonelli, F. and Frucci, M. (2017), ‘Wearable improved vision system for color vision deficiency correction’, IEEE Journal of Translational Engineering in Health and Medicine 5, 1–7. Norman, D. A. (2004), Emotional Design: Why We Love (or Hate) Everyday Things,

Basic Books, New York. Simunovic, M. P. (2010), ‘Colour vision deficiency’, Eye 24(5), 747–755. Treisman, A. M. and Gelade, G. (1980), ‘A feature-integration theory of attention’, Cognitive Psychology 12(1), 97–136. Zheng, L. (2016a), ‘User experience design for color blind groups in games’, Design

(15), 24–25. Zheng, R. Z. (2016b), Handbook of Research on Modality in Multimedia Learning, IGI Global, Hershey, PA.

Cover of the proceedings volume Accessibility, Assistive Technology and Digital Environments

Explore the complete digital volume

Accessibility, Assistive Technology and Digital Environments

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-78-6

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →