Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

A Synergistic, Non-Invasive Sensing-Fusion Approach for Predictive Kinetosis Monitoring in Autonomous Vehicles

Institute of Vehicle System Technology, Karlsruhe Institute of Technology, Karlsruhe, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1007123

Abstract

The coming of autonomous vehicles promises a "passenger economy," a vision jeopardized by the challenge of kinetosis (motion sickness). Effective mitigation requires non-invasive, predictive monitoring, yet current methods are impractical. This paper presents a robust methodology that validates a synergistic fusion of thermal and visible-light imaging as a reliable respiratory biomarker. Our system employs a thermal camera to track temperature differentials at the nostrils and an RGB camera to monitor thoraco-abdominal movements. We introduce a real-time signal processing pipeline featuring: (1) dynamic, multi-modal region-of-interest tracking, (2) independent signal "activity gating" to reject noise, and (3) a temporal peak-fusion algorithm to compute a single, robust breathing rate. The primary contribution is the demonstration of this system's technical feasibility and resilience to real-world failure modes. In a pilot study, we demonstrate high accuracy against a ground-truth metric and, crucially, show the system maintains a stable output during facial and torso occlusions that would cause single-modality systems to fail. This robust, non-invasive system represents an important technical step toward truly human-centric autonomous vehicles such as the C2CBridge Vehicle.

INTRODUCTION

Autonomous vehicles are speculated to redefine future mobility, freeing occupants from the task of driving and unlocking a new “passenger economy” (Strategy Analytics, 2017). However, this transition from active driver to passive passenger introduces a series of human-factor impediments. One of the main ones being kinetosis (or motion sickness). The incidence of motion sickness is expected to surge as passengers engage in non-driving-related activities such as reading or screen-based work during their trips (Diels and Bos, 2016).

This discomfort (kinetosis) arises from a “nauseogenic triad” inherent to the autonomous nature of the vehicle:

1. Sensory Conflict: The mismatch between a visually static interior and the vestibular system’s sensation of motion. 2. Lack of Control: The passenger’s inability to physically control the vehicle’s path and trajectory. 3. Anticipation Failure: The inability to predict the vehicle’s upcoming movements (Reason and Brand, 1975).

Widespread passenger discomfort could threaten to undermine the value proposition of autonomous vehicles. To try and mitigate this, one approach could be for the vehicle to be able to detect the onset of distress before it reaches the threshold of conscious nausea.

Current assessment methods are unsuitable for this task. Subjective scales (e.g., M-SSQ) are retrospective and intrusive (Golding, 2016). Objective, contact-based physiological sensors (such as ECG or EGG) are too invasive for a consumer environment. This creates a critical need for a monitoring paradigm that is simultaneously objective, non-invasive, predictive, and robust.

This paper proposes a solution by validating respiration as a key predictive biomarker. The physiological link between nausea and respiration is well- established in research; the emetic (vomiting) reflex is a respiratory event coordinated by the autonomic nervous system (Chang et al., 2010). Previous studies indicate that changes in respiratory rate and stability often precede the conscious perception of nausea (Al-Naji and Chahl, 2017).

Our contribution is a synergistic sensing-fusion system that non- invasively measures respiration with high robustness. We fuse data from two complementary, low-cost modalities:

1. Thermal Imaging: Monitoring temperature oscillations at the nostrils caused by inhalation (cool air) and exhalation (warm air). 2. Visible-Light (RGB) Imaging: Monitoring subtle, periodic intensity changes on the upper torso caused by thoraco-abdominal respiratory movements.

By fusing these two signals (Figure 1), our system overcomes the individual weaknesses of each modality, creating a reliable wellness biomarker to enable pre-emptive HMI interventions.

Figure 1: Pipeline of the synergistic fusion system.
Original PDF Figure 1:

RELATED WORK

Contactless Respiration Monitoring Thermal Imaging: Thermal cameras can detect the subtle temperature differential between ambient air and inhaled/exhaled breath (See Figure 2). This method has been successfully used to monitor respiratory rates in various contexts (Al-Naji et al., 2019; 2023). Its primary advantage is its effectiveness in variable and complete darkness. However, it is sensitive to occlusions.

Figure 2: Thermal changes in the nostril area .
Original PDF Figure 2:

Visible-Light (RGB) Imaging: Standard RGB cameras can track respiration by measuring periodic movements, often by tracking optical flow or mean pixel intensity of a chest or thorax area (D’mello-Johnson et al., 2017). While low-cost, this method requires adequate ambient light and can be spoofed by a passenger’s gross body movements. The Thorax (RGB) signal processing pipeline is shown in the following figure:

Figure 3: (Top) The raw signal extracted from the chest ROI, showing significant noise and baseline drift. (Middle) The same signal after applying a 2nd-order Butterworth bandpass filter (0.1-0.5 Hz), which isolates the respiratory waveform. (Bottom) The final peak detection step, where each “x” marks a detected breath on the filtered signal, used to calculate the final BPM.
Original PDF Figure 3:

A Synergistic, Non-Invasive Sensing-Fusion Approach 653 Sensor Fusion for Robustness

Given the in-vehicle environment’s challenges, relying on a single modality is not enough. The “synergetic use” of multiple sensing techniques offers a path to greater robustness (Viola and Jones, 2001). Our work adapts this concept specifically for kinetosis monitoring, where robustness is of uppermost importance.

METHODOLOGY

Our system is implemented in MATLAB and built upon the described dual- camera setup and a real-time processing pipeline:

Figure 4: Algorithmic pipeline.
Original PDF Figure 4:

Our system is a multi-component algorithm that transforms raw pixel data into a respiratory rate, as shown in Figure 2. The Thermal camera captures a raw thermal array ( ) THM I and a co-registered visible-light image ( ) RGB T I − . The RGB Camera captures a high-resolution color feed of the passenger’s upper torso ( ). RGB EXT I − Component 1—Cross-Modal ROI Localization We use a Cascade Object Detector on the visible-light feeds

( ) , RGB T RGB EXT I I − − to locate the passenger’s face. The Nostril ROI ( ) nostril ROI and ( ) chest ROI are then defined geometrically. To prevent jitter, the nostril ROI position is smoothed using an EMA with 0.25 α = A “coasting” logic maintains the ROIs for 15 frames if detection is temporarily lost.

Component 2—Voxel-Based and Motion-Base Thermal/Visible Signal Extraction The respiratory signal is extracted from the thermal data by analyzing the voxels (volume pixels) within the nostril ROI . We employ a Minima Selection technique, as the inhalation of cool air provides the most robust signal.

( ) ( ) ( ) ( ) min , , , . THM th nostril S t I x y t x y ROI = ∀ ∈ At the same time, the chest respiratory signal is extracted by tracking Mean Intensity Modulation within the chest ROI . As the chest expands and contracts, the mean grayscale value of the region modulates periodically ( ) ( ) ( ) ( ) , , , . vis Gray chest S t mean I x y t x y ROI = ∀ ∈ Component 4—Robust Fusion & Rate Estimation The raw signals, th S and vis S are processed in a final fusion module:

1. Filtering: Both camera signals are passed through a 2nd-order, zero- phase Butterworth bandpass filter( ) 0,1 0,5 Hz − to isolate the respiratory waveform. 2. Activity Gating: The system calculates the standard deviation of the raw signal over a 5-second window. If the signal’s variance is below a threshold( ) 0.5, 0.3 vis th τ τ = = it is considered “inactive” and is ignored by the peak detector. 3. Temporal Peak-Matching: An algorithm is run on all active signals. A temporal fusion algorithm then combines the peak lists. This logic prioritizes thermal peaks but “claims” any matching visible-light peaks within a 0.75s window. Unclaimed visible peaks are also counted, creating a “best-of-both” signal. 4. Rate Calculation: The final fused respiratory rate is calculated from the total fused breath count over the time window.

METHODOLOGY

To validate the technical feasibility, accuracy, and robustness of our fusion system, we conducted a two-part pilot study. The goal was to first, quantify the system’s accuracy against a ground truth and to second, demonstrate the algorithm’s resilience to common signal-loss scenarios that might occur in shared mobility vehicles.

Pilot Validation of Accuracy: A ground-truth respiratory rate was established by manual breath counting through a custom application. The volunteer was instructed to breathe at three distinct paces for 60 seconds each: Normal (Eupnea), Slow (Bradypnea), and Fast (Tachypnea). The mean respiratory rate from our system was compared against the ground truth. The results (summarized in Table 1) show a good degree of accuracy across the physiological range, with a mean absolute error of less than 1 BPM.

Table 1: System accuracy validation. Comparison of the system’s fused respiratory rate against the ground truth at three distinct, controlled breathing paces.

Table available in the original PDF paper.

Condition Ground Truth Fused System Absolute Error Slow breathing 8.0 BPM 8.9 BPM 0.9 BPM

Normal breathing 14.0 BPM 14.7 BPM 0.7 BPM Fast breathing 22.0 BPM 21.2 BPM 0.2 BPM

Demonstration of Fusion Robustness: To simulate real-world challenges that might be encountered in the C2CBridge-Vehicle, we recorded a continuous session where the subject performed two specific occlusion events while breathing normally:

1. Facial Occlusion (5–10s): The subject placed a hand over their nose and mouth, occluding the nostril ROI . 2. Torso Occlusion (13–18s): The subject held a book over their upper chest, occluding the . chest ROI

A Synergistic, Non-Invasive Sensing-Fusion Approach 655 During the facial occlusion, the amplitude of the thermal signal dropped below our specified th τ . The system correctly ignored this input and derived the respiratory rate solely from the active chest signal. During the torso occlusion, the visible light signal flattened. The system identified this and relied exclusively on the clean thermal signal. The final fused output remained stable and accurate throughout both events, demonstrating robustness critical for a real-world deployment (Figure 5).

Figure 5: Demonstration of the robust fusion algorithm. This graph shows the system’s response to simulated occlusion events. The plot shows the filtered (solid) and raw (dotted) signals for the nostril (blue) and thorax (red).
Original PDF Figure 5:

DISCUSSION

The primary contribution of this work is the technical validation of a system for respiratory monitoring. Our pilot results confirm two points. First, the system is accurate (Table 1). Second, and more importantly, the system is robust (Figure 5).

In a dynamic in-vehicle environment, transient occlusions are inevitable. A system relying on a single modality would suffer from frequent data dropouts. Our activity-gated fusion logic overcomes this limitation, ensuring a stable, continuous output.

We acknowledge the limitations of this pilot study. The validation was performed in a static environment. However, as a proof-of-concept, these results successfully demonstrate the technical feasibility and core advantages of our fusion-based approach, establishing the necessary foundation for future, larger-scale studies on kinetosis prediction.

CONCLUSION AND FUTURE WORK This conference paper presents a non-invasive system for monitoring passenger respiration by synergistically fusing thermal and RGB camera data. Through a pilot study, we have demonstrated accuracy and more importantly, its robustness to common occlusion scenarios.

Future work will focus on:

1. Predictive Modeling: Using this system to extract features of Respiratory Rate Variability and train a machine learning classifier to detect physiological precursors to nausea/motion sickness.

2. Complex Validation: Conducting a larger-scale study in a vehicle in motion to correlate our physiological data with subjective reports of motion sickness. 3. HMI Integration: Using the predictive model to trigger real-time, therapeutic interventions, such as adjusting airflow or modifying the vehicle’s drive style.

ACKNOWLEDGMENT

“This paper was created in the “Country 2 City - Bridge” project of the “German Center for Future Mobility”, which is funded by the German Federal Ministry for Transport.”

REFERENCES

A. Al-Naji, M. J. Al-Khalidi, and J. Chahl, “Vision-based Respiration Rate Measurement using motion magnification,” Journal of Physics: Conference Series, vol. 2990, p. 012004, 2023. A. Al-Naji, S. L. Mohammed, and J. Chahl, “Non-Contact Respiratory Rate

Monitoring Using a Thermal Camera,” in 2019 International Conference on Control, Automation and Diagnosis (ICCAD), Grenoble, France, 2019, pp. 1–6. A. Al-Naji and J. Chahl, “Remote monitoring of breathing rate and statistics using a thermal camera,” Biomedical Optics Express, vol. 8, no. 10, pp. 4480–4494, 2017. C. Diels and J. E. Bos, “Motion Sickness in Automated Vehicles,” in Handbook

of Human Factors in Automated and Autonomous Vehicles, N. A. Stanton, Ed. Boca Raton, FL: CRC Press, 2016, pp. 439–462. J. F. Golding, “Motion sickness,” Handbook of Clinical Neurology, vol. 137, pp. 371–390, 2016. J. T. Reason and J. J. Brand, Motion Sickness. London, UK: Academic Press, 1975. N. B. D’mello-Johnson, et al., “Synergetic use of thermal and visible imaging techniques

for contactless and unobtrusive breathing measurement,” IEEE Transactions on Biomedical Engineering, vol. 64, no. 12, pp. 2809–2820, Dec. 2017. P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. I-511–I-518, 2001. Strategy Analytics, “The Passenger Economy: A $7 Trillion Market Opportunity,” A

study commissioned by Intel, 2017. [Online]. Available at: https://www.intel.com/ content/dam/www/public/us/en/documents/pdf/passenger-economy-report-au tonomous-driving.pdf Y. W. Chang, S. C. H. Yang, C. T. S. Huang, and T. C. Kuo, “Change in respiratory rate precedes the onset of nausea in susceptible women,” Neurogastroenterology & Motility, vol. 22, no. 10, pp. 1083–1088, 2010.

Cover of the proceedings volume Intelligent Human Systems Integration (IHSI 2026): Disruptive and Innovative Technologies

Explore the complete digital volume

Intelligent Human Systems Integration (IHSI 2026): Disruptive and Innovative Technologies

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-76-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →