Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Beyond Vision: Non-Visual Sensory Channels Assist Blind Passengers in Understanding Driving Environments and Vehicle Behavior

Publication year
Digital Object Identifier10.54941/ahfe1008044

Abstract

In the context of the development of autonomous driving technology, the intelligent cockpit's ability to process information has gradually increased, becoming an important carrier for obtaining driving information and understanding the traffic environment. Unlike traditional human-computer interaction methods dominated by vision, this study focuses on how to provide blind people with relevant information on vehicle status and environmental changes through sensory channels such as touch and hearing during autonomous driving, in order to support their formation of basic cognition of driving situations. The research aims to explore the role of multimodal interaction methods in enhancing the perception ability and riding safety of blind people. The experiment focuses on constructing typical usage scenarios for 10 common scene elements in autonomous driving, using interactive methods such as tactile gloves and car machine prompts as the main sensory channels for transmitting information. By collecting data on users' situational perception ability, scene judgment accuracy, judgment grasp degree, etc., evaluate users' perception ability under different sensory channels. The results indicate that multimodal information feedback can significantly enhance the perception of blind people during autonomous driving, help them better understand vehicle behavior and driving status, and enhance their predictability of the system operation process, providing a reference design basis for barrier free interaction design for blind people in autonomous driving scenarios.

INTRODUCTION

With the rapid development of autonomous driving technology, vehicles are gradually transforming from traditional modes of transportation to highly intelligent mobile platforms that can perceive, process, and transmit a large amount of environmental and operational information (Fagnant & Kockelman, 2015; Hussain & Zeadally, 2019). This transformation not only reshapes driving behavior, but also fundamentally changes the roles of passengers in the car. In a highly automated driving environment, passengers no longer need to continuously control the vehicle, but rely on the system to

152 Zhi-Yang and Dan-Hua complete most driving tasks, and their role shifts to supervising, interpreting, and trusting system behavior (Banks et al., 2018). For visually impaired people, autonomous vehicle provide a promising opportunity to overcome long- standing travel obstacles and gain greater independence in daily travel (Harper et al., 2016). However, not requiring active driving does not necessarily mean that one can automatically obtain a safe and comfortable travel experience.

Unlike people with normal vision, blind passengers cannot rely on visual cues to understand the vehicle’s status, surrounding traffic conditions, or upcoming driving operations. Even if the auto drive system itself operates reliably, this lack of information about the surrounding environment will also lead to a reduction in uncertainty, anxiety and security (Loomis et al., 1998). Situational awareness is widely recognized as a key factor influencing user safety, trust, and decision-making in complex systems. It involves perceiving relevant elements in the environment, understanding their meanings, and predicting their future states (Endsley, 2017). In autonomous driving, passengers’ situational awareness is closely related to their understanding of what the vehicle is doing, why it is doing so, and what will happen next (Banks et al., 2018). For blind users, this process must be supported through non visual information channels. However, currently most in car human-machine interfaces are still primarily designed around visual displays, which creates barrier free access for visually impaired users. To address this challenge, researchers have explored sensory substitution and multimodal interaction methods, utilizing auditory and tactile feedback to convey spatial and environmental information (Kaczmarek et al., 2002; Jones & Sarter, 2008). Previous studies have shown that auditory displays can effectively assist blind users in navigation and environmental understanding, while tactile feedback can provide spatial clues and directional information through vibration tactile patterns (Zhao et al., 2018). These methods have been widely applied in navigation assistance systems, virtual environments, and assistive devices (Maidenbaum & Amedi, 2015). However, empirical research on how such multimodal feedback supports situational awareness and safety in autonomous driving environments is still limited.

Existing research often focuses on isolated interaction technologies or single sensory channels, without systematically comparing different interaction methods or their combinations in typical driving scenarios. At present, there is a lack of experimental evidence to demonstrate how auditory feedback, tactile feedback, and their integration affect blind users’ understanding of vehicle behavior, accuracy of scene interpretation, confidence in judgment, and emotional reactions during autonomous driving. In order to make up for these research gaps, this study explored how auditory and tactile feedback, when used alone or in combination, affects blind passengers’ perception of driving environment and vehicle behavior when riding autonomous vehicle. Our research question is as follows: RQ1: Can non visual sensory channels help blind passengers perceive the

driving environment and vehicle behavior during autonomous driving? And how it has an impact. RQ2: If it can have an impact, what are the reasons for the impact? In what way can barrier free design be carried out in the context of autonomous vehicle design?

RELATED RESEARCH The accessibility research for visually impaired individuals has long focused on how to utilize non visual sensory channels to assist their perception, navigation, and interaction with the environment. Early research on sensory substitution suggests that information typically transmitted through visual cues can be partially transferred to other sensory modes, such as auditory and tactile, enabling blind users to construct psychological representations of space and events (Bach-y-Rita et al., 1969). These studies laid the theoretical foundation for later auxiliary technologies that convert visual information into auditory or tactile signals. The auditory interface has been widely used in blind navigation systems because sound can effectively convey spatial direction, distance, and event information. Research has shown that auditory navigation systems can assist users in route following, obstacle avoidance, and environmental perception in indoor and outdoor environments (Katz et al., 2012). However, auditory channels also have some limitations, such as information overload, environmental noise masking, and increased cognitive load when presenting complex or continuous information (Maidenbaum & Amedi, 2015). Therefore, tactile and vibration tactile feedback have been explored as a complementary or alternative way of perception. Wearable tactile displays, such as belts, gloves, or vests, can transmit directional cues, distance information, and alerts through spatially distributed vibration patterns (Yang et al., 2021). Compared to auditory feedback, tactile cues can provide a more direct spatial mapping and may reduce interference from environmental sounds. Research in the fields of virtual reality and navigation assistance has shown that multimodal sensory substitution can improve task performance, user confidence, and satisfaction compared to single modal solutions (Zhao et al., 2018). Despite these advances, most accessibility research still focuses on pedestrian navigation, virtual environments, or independent assistive devices, with relatively limited attention to in vehicle environments and autonomous driving scenarios.

With the improvement of vehicle automation, the role of passengers in the car is gradually shifting from active driving to passive or supervised driving. This shift has raised new questions about how passengers understand vehicle behavior and how to maintain appropriate awareness and trust in the system (Goodall, 2014). Even if the system performance is objectively safe, insufficient understanding of autonomous vehicle behavior (such as sudden braking, lane changing, or route deviation) can lead to discomfort, anxiety, and decreased trust. For visually impaired passengers, maintaining their perception of the surrounding environment faces additional challenges. Due to the inability to obtain visual clues such as road layout, traffic signals, or nearby vehicles, blind passengers must rely entirely on other sources of information to understand the situation during their journey. Research on non driving related tasks and passenger experience has shown that perceived safety and comfort are closely related to the predictability and transparency of vehicle behavior (Banks et al., 2018). However, most existing research assumes that passengers can see the display screen or external environment, without explicitly focusing on the needs of blind users. Some studies have explored improving passengers’ understanding of autonomous vehicle

154 Zhi-Yang and Dan-Hua decision-making through auditory interpretation or system state information. Although these methods can improve transparency, they typically focus on general system states rather than detailed environmental backgrounds, and their effectiveness for blind passengers still needs to be studied. In addition, there are few studies that systematically evaluate how different sensory modes affect passengers’ situational perception, confidence, and emotional responses in real driving scenarios.

SIMULATED AUTONOMOUS DRIVING EXPERIMENT FOR BLIND PEOPLE Scene and Task Design

In order to explore how auditory and tactile feedback can help blind people understand the driving environment and vehicle status in an autonomous driving environment, we conducted an experiment using a simulated autonomous driving platform. Each participant experienced a series of autonomous driving scenarios under multiple sensory conditions, enabling them to systematically evaluate how non visual information channels affect understanding of vehicle behavior and surrounding traffic conditions. The participants’ task is to experience simulated autonomous driving and describe the current driving scenario, as well as their level of understanding of whether the description is correct.

Figure 1: Experimental site layout and display.
Original PDF Figure 1:

Beyond Vision: Non-Visual Sensory Channels Assist Blind Passengers 155 Independent Variable

Different sensory channels: This study employed an in subject design. We evaluated the sensory channels that can replace vision in the process of autonomous driving, and selected two sensory channels based on spatial contact points during the autonomous driving process. The sensory channels involve hearing and touch, and are presented in the form of tactile gloves and car machine prompts.

Based on the literature review of vibration feedback patterns in driving environments (Spence & Ho, 2008), the tactile glove vibration patterns we designed include five different modes: single-sided short shock once (when there is an object in that direction), single-sided long shock once (when turning or changing lanes in that direction), single-sided long shock twice (when making a U-turn in that direction), double-sided simultaneous short shock once (when decelerating or braking), double-sided two long shocks & single-sided short shock (when a dangerous situation occurs & in the direction of a dangerous situation).

The car machine prompt sound includes four different modes: weather and lighting (humidity, temperature, weather conditions, etc.), static road information (lane information, road facility information), dynamic road information (road dynamic changes, road facility dynamic changes, communication environment dynamic changes), and traffic participant information (pedestrian information, non motorized vehicle information, motorized vehicle information).

Dependent Variable The following indicators and data are used to evaluate the impact of sensory channels on users’ mastery of driving processes and driving environments:

Situational awareness ability: Evaluating an individual’s ability to interpret and understand surrounding environmental information. In the autonomous driving environment, it includes identifying the surrounding traffic environment, road signs, pedestrians, and other potential factors Scene judgment accuracy: used to measure the accuracy of judging 10 specific scenarios in a simulated driving environment. Accuracy can indicate whether users can accurately and timely identify and respond to changes in the environment.

Scene judgment determines my level: measuring the user’s confidence level when making judgments Experimental Simulation of Autonomous Driving Scenarios

Our experiment relies on a virtual driving test bench, and we designed and simulated a real road environment through photographic restoration methods. Based on the literature review of driving scenario elements (Zhu et al., 2019), a total of 10 driving scenarios were designed in the experiment, including different traffic environment elements such as weather and lighting, static road information, dynamic traffic information, and traffic participants. The 10 scenarios are:

1. Driving on snow covered roads on dangerous snowy days (weather and lighting: weather conditions and light intensity) 2. Sudden rainstorm, reduced visibility (weather and light: visibility) 3. The traffic lights ahead turn red (static road information: traffic signal lights) 4. Forward U-turn (static road information: intersection shape) 5. Road maintenance ahead, with obstacle maintenance signs (dynamic road information: road dynamic changes - road maintenance) 6. Sudden situation on the road ahead, with trees breaking (dynamic road information: dynamic changes in road facilities - trees breaking) 7. Entering the tunnel ahead, signal obstruction or weakening (dynamic road information: dynamic changes in communication environment - signal obstruction) 8. Sudden situation on the road ahead, pedestrians crossing the road (traffic participant information - pedestrian information) 9. Non motorized vehicles driving on the right side (Traffic Participant Information - Non motorized Vehicle Information) 10. Motor vehicles overtaking from the right to grab lanes (Traffic Participant Information - Motor Vehicle Information)

Figure 2: 10 experimental scenarios with different traffic environment elements included.
Original PDF Figure 2:

Experimental Procedure After welcoming the participants and briefly introducing the research objectives, we asked them to sign a user informed consent form. Next, we explained in detail the experimental tasks and procedures, and informed the participants of the sensory channels needed for this experiment and what data to collect. Before the experiment begins, the experimenters will demonstrate how to correctly use the two sensory channels introduced, such as how to transmit information, and participants can ask questions at any time during this process.

Beyond Vision: Non-Visual Sensory Channels Assist Blind Passengers 157 After ensuring that participants have a clear understanding of the experimental tasks, the researchers will guide them to sit in the driver’s seat of the test bench and help them attach heart rate monitoring devices. Throughout the experiment, heart rate data will be used to observe the user’s emotional state as an auxiliary measure. This experiment is divided into five groups to evaluate the performance differences of users under different conditions:

Table 1: Experimental group setting.

Table available in the original PDF paper.

Group Group Setting Group1 Participants with normal vision do not wear eye masks to evaluate their performance under normal visual conditions

Group2 Participants wear eye masks to simulate the visual impairment of a blind person.

Group3 Participants wear eye masks and in the experiment, traffic environment elements are provided through the car’s infotainment system.

Group4 Participants wear eye masks and obtain traffic environment elements through vibrating gloves.

Group5 Participants wear eye masks and receive traffic environment elements provided by the car’s infotainment system and vibrating gloves.

To avoid inter group interference, the 10 driving scenarios in each group are randomly selected, and participants will experience all designed scenarios and make timely scene judgments after each experience. After experiencing 10 driving scenarios in each group, users will fill out a situational perception scale and an emotion recording scale. After the experiment, we will conduct a semi-structured interview for about 5 minutes. Participants need to review the experimental process and answer questions, evaluate different sensory channels, and provide some explanation for the previously filled out scale.

Participants This experiment recruited adults aged 18–45, located in Changsha, China. The recruitment criteria are as follows: no cognitive impairment, basic social cognition and judgment ability. There are no auditory or tactile impairments. Having a certain level of understanding of autonomous driving and no bad driving habits.

We selected 16 healthy adults with experience in autonomous driving from numerous applications to participate in this experiment. The participants in the experiment included industrial designers, experimenters with design backgrounds, and automation industry professionals who played the role of autonomous driving passengers in this experiment.

Analysis After careful inspection at the end of the experiment, the data of the 16 participants in the experiment were reliable and effective, and their data was analyzed. The accuracy of scene judgment is the main indicator for evaluating the participants’ mastery of driving process and environment

158 Zhi-Yang and Dan-Hua under different sensory channels. In the five control experiments, the accuracy ranged from 15.63% to 96.25%. The tactile assistance group achieved a scene judgment accuracy rate of 51.25%, significantly higher than the 15.63% of the simulated blind group, indicating that tactile feedback has a certain effect in compensating for visual loss. However, its standard deviation is 0.14, reflecting significant differences in adaptability to tactile stimuli among individuals, leading to low consistency in judgment. In terms of average confidence, the tactile assistance group had a score of 54.52%, indicating that participants had limited dependence on tactile information and experienced significant fluctuations in information, which affected their overall confidence level. The situational perception ability is 42.85 ± 17.44, which is better than the simulated blind group, but significantly lower than the visual group, indicating the limitations of tactile assistance in environmental perception.

The auditory assistance group showed excellent performance in scene judgment accuracy, reaching 94.38%, demonstrating the effectiveness of auditory information in conveying environmental details. Its standard deviation is 0.10, indicating that auditory assistance performs well in providing consistent judgment support. The average confidence level increased to 79.58%, indicating that auditory cues significantly enhanced participants’ confidence and helped to more accurately evaluate the driving environment. In terms of situational perception ability, the auditory assistance group achieved 64.19 ± 13.51, which is superior to the tactile assistance group, confirming the advantage of auditory in enhancing environmental perception ability.

In summary, auditory assistance is superior to tactile assistance in improving scene judgment accuracy, enhancing participant confidence, and improving situational perception ability, demonstrating the important role of auditory information in assisted driving. Although tactile assistance partially compensates for visual loss, its effectiveness is relatively limited and varies greatly among individuals.

Figure 3: The accuracy and confidence level of users’ judgment.
Original PDF Figure 3:
Figure 4: User’s perception ability of driving environment and vehicle behavior.
Original PDF Figure 4:

When exploring the role of tactile and auditory assistance in sensory channels, this study found that tactile assistance exhibits significant advantages in providing spatial positioning and operational feedback, especially in the perception of display position. Specifically, the tactile group outperformed the auditory group (42.09 ± 20.86) in perceiving the position of dynamic road information (44.41 ± 26.51), indicating that tactile assistance can more accurately guide drivers to perceive spatial information.

Table 2: Performance of different groups under different scene elements.

Table available in the original PDF paper.

Group 1 Group 2 Group 3 Group 4 Group 5 Weather and lighting 94.06± 11.62 35.75± 27.44 48.94± 25.66 77.81± 17.5 85.94± 13.46

Dynamic road information

75.79± 24.46 11.10± 18.19 41.90± 31.29 73.06± 24.03 85.15± 14.44

Traffic participant information

85.00± 12.50 14.50± 18.87 54.12± 27.82 75.25± 17.33 87.50± 10.21

Location of dynamic road information

85.69± 12.88 5.53± 14.49 44.41± 26.51 42.09± 20.86 81.81± 17.63

Location information of traffic participants

83.12± 15.80 6.38± 11.19 42.02± 25.06 38.38± 26.12 84.88± 17.02 Based on the content analysis of semi-structured interviews, clear and structured language prompts can not only compensate for the lack of visual information, but also directly alleviate the core anxiety caused by unknown situations by providing clear explanations and expectations. Its soothing effect surpasses the traditional model of relying on visual self observation but lacking systematic explanation in some aspects. In contrast, although pure tactile feedback (tactile assistance group) outperformed the simulated blind group in the negative emotional dimension, its regulatory effect was far inferior to auditory feedback. This reveals that simple directional or state based tactile cues have limitations in alleviating deep psychological stress due to their limited information carrying capacity and more ambiguous

160 Zhi-Yang and Dan-Hua interpretation, making it difficult to support users in forming a complete situational cognition. Multiple users have pointed out that multimodal composite feedback (tactile & auditory) typically exhibits optimal or suboptimal regulatory effects on various emotional indicators, especially in creating the lowest levels of tension and depression. Touch can provide immediate and directional spatial alerts, while hearing provides rich and semantically clear explanations. The combination of the two creates a more predictable and controllable cognitive environment. This proves that multimodal interaction design with auditory guidance and tactile supplementation is a necessary path to effectively enhance the psychological safety of blind passengers during autonomous driving.

DISCUSSION

This study explores how auditory and tactile feedback can be used alone or in combination to enhance the situational perception and sense of safety of blind people in autonomous driving environments. The results indicate that providing non visual information through structured sensory channels can significantly improve participants’ understanding, judgment accuracy, and emotional comfort of driving situations compared to the situation of blindfolding without assistance. These findings highlight the importance of providing convenient information in intelligent cockpit design and provide empirical evidence for the effectiveness of multimodal feedback in assisting blind passengers.

A major finding is that speech or sound based information is particularly effective in conveying complex and abstract environmental information, such as the behavior of traffic participants, dynamic road changes, and upcoming traffic operations. Auditory information can directly decode contextual content, reducing the process of participants interpreting symbols. Tactile feedback relies on learned vibration patterns and requires cognitive translation to understand its meaning, which may increase cognitive load and lead to greater individual performance differences. However, it demonstrates significant value in conveying spatial and directional cues. Participants are able to infer the direction of dangerous objects, turns, or nearby objects using vibration patterns, indicating that tactile channels are highly suitable for assisting spatial perception. However, the relatively large variability observed when using only touch suggests that tactile mapping must be carefully designed and may require personalized customization or extended training time. On most indicators, including scene judgment accuracy, situational perception, and perceived safety, the feedback method combining auditory and tactile senses achieved the highest overall performance. This result supports the view that multimodal feedback can fully utilize the complementary advantages of different sensory channels. These two modes work together to present driving situations more comprehensively and accurately, thereby reducing uncertainty and improving predictability.

From a design perspective, the research results provide several insights for user-friendly intelligent cockpit interfaces. Firstly, auditory feedback should be regarded as the primary channel for conveying complex dynamic

Beyond Vision: Non-Visual Sensory Channels Assist Blind Passengers 161 information, such as the behavior of traffic participants and upcoming maneuvers. Secondly, tactile feedback can serve as a supplementary channel for spatial positioning, directional alerts, and time critical prompts. Thirdly, combining auditory and tactile feedback can improve system stability and better adapt to individual preferences and sensory advantages. Designers should also consider adjustable information density and customized feedback mapping to address individual differences in perception and learning.

ACKNOWLEDGMENT

Thank you to my supervisor for their guidance and assistance during the writing process.

REFERENCES

Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969).

Vision substitution by tactile image projection. Nature, 221(5184), 963–964. Banks, V. A., Eriksson, A., O’Donoghue, J., & Stanton, N. A. (2018). Is partially automated driving a bad idea? Observations from an on-road study. Applied ergonomics, 68, 138–145. Endsley, M. R. (2017). Toward a theory of situation awareness in dynamic systems.

In Situational awareness (pp. 9–42). Routledge. Fagnant, D. J., & Kockelman, K. (2015). Preparing a nation for autonomous vehicles: opportunities, barriers and policy recommendations. Transportation Research Part A: Policy and Practice, 77, 167–181. Goodall, N. J. (2014). Ethical decision making during automated vehicle

crashes. Transportation Research Record, 2424(1), 58–65. Harper, C. D., Hendrickson, C. T., Mangones, S., & Samaras, C. (2016). Estimating potential increases in travel with autonomous vehicles for the non-driving, elderly and people with travel-restrictive medical conditions. Transportation research part C: emerging technologies, 72, 1–9. Hussain, R., & Zeadally, S. (2018). Autonomous cars: Research results, issues, and

future challenges. IEEE Communications Surveys & Tutorials, 21(2), 1275–1313. Jones, L. A., & Sarter, N. B. (2008). Tactile displays: Guidance for their design and application. Human factors, 50(1), 90–111. Kaczmarek, K. A., Webster, J. G., Bach-y-Rita, P., & Tompkins, W. J. (2002).

Electrotactile and vibrotactile displays for sensory substitution systems. IEEE transactions on biomedical engineering, 38(1), 1–16. Katz, B. F., Kammoun, S., Parseihian, G., Gutierrez, O., Brilhault, A., Auvray, M., ... & Jouffrais, C. (2012). NAVIG: Augmented reality guidance system for the visually impaired: Combining object localization, GNSS, and spatial audio. Virtual Reality, 16(4), 253–269. Loomis, J. M., Golledge, R. G., & Klatzky, R. L. (1998). Navigation system for the

blind: Auditory display modes and guidance. Presence, 7(2), 193–203. Maidenbaum, S., & Amedi, A. (2015, March). Non-visual virtual interaction: Can Sensory Substitution generically increase the accessibility of Graphical virtual reality to the blind?. In 2015 3rd IEEE VR international workshop on virtual and augmented assistive technology (VAAT) (pp. 15–17). IEEE. Spence, C., & Ho, C. (2008). Tactile and multisensory spatial warning signals for

drivers. IEEE Transactions on Haptics, 1(2), 121–129.

162 Zhi-Yang and Dan-Hua Yang, W., Huang, J., Wang, R., Zhang, W., Liu, H., & Xiao, J. (2021). A survey on

tactile displays for visually impaired people. IEEE Transactions on Haptics, 14(4), 712–721. Zhao, Y., Bennett, C. L., Benko, H., Cutrell, E., Holz, C., Morris, M. R., & Sinclair, M. (2018, April). Enabling people with visual impairments to navigate virtual reality with a haptic and auditory cane simulation. In Proceedings of the 2018 CHI conference on human factors in computing systems (pp. 1–14). Zhu, B., Zhang, P. X., Zhao, J., Chen, H., Xu, Z., Zhao, X., & Deng, W. (2019).

Review of scenario-based virtual validation methods for automated vehicles. China Journal of Highway and Transport, 32(6), 1–19.

Cover of the proceedings volume Usability and User Experience

Explore the complete digital volume

Usability and User Experience

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 979-8-950676-08-6

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →