Junya Tatsuno, Yushi Otsuka, Setsuo Maeda (2023). Detection of Unconscious Movements with RGB-D Camera for Objective Ride Comfort Evaluation. In: Human Factors in Transportation. AHFE Open Access, vol. 95, pp. 12–19. https://doi.org/10.54941/ahfe1003787
The previous studies reported a high correlation between the frequencies of the unconscious movements and participants' subjective ratings. Then, we have tried to develop other techniques that facilitate the digitization of participants' movements. For example, the method using body pressure distribution measurement system, and flex sensors have been reported. As these were instrument contact-constrained methods, this paper developed a measurement system that automatically extracts and classifies unconscious movements in a non-contact and non-constraint manner. In the accuracy evaluation experiment, the participants were asked to drive a driving simulator for 60 minutes and were captured with the developed system and a conventional video camera. As a result, the accuracy was insufficient, and there were two error types. The first error (false positive) was a case in which the program falsely detected the occurrence of a motion even though no unconscious movement occurred. The second one (false negative) was the opposite error, where the result incorrectly indicates the absence of movements when the movement was happening. By applying the proposed countermeasures to reduce these errors, the recognition accuracy of unconscious movements can be improved and applied to the objective evaluation of riding comfort.
Whole-body vibration exposure to occupants while riding in a vehicle affe- cts ride comfort. Therefore, automobile manufacturers and seat suppliers have conducted research and development that can reduce whole-body vibra- tion. In judging new products or technologies, they have utilized sensory evaluations, in which the subjective senses of skilled persons are used. However, as for sensory evaluations, since it takes a long time to train skilled workers, it takes a lot of work to hand down the skills to the
Detection of Unconscious Movements With RGB-D Camera 13
next generation. Although there is a questionnaire targeting general dri- vers as an alternative evaluation method, a certain number of samples is necessary considering the data variation due to individual differences. From this background, it is desired to develop objective evaluations of ride comfort.
On the other hand, it was reported that the frequency of unconscious movement during seated in the vehicle was effective as objective evaluation index of ride comfort (Sammonds et al. 2014). Then, unconscious movements were considered to be carried out to release blood flow obstruction caused by muscle compression. This study suggests a strong correlation between the frequency of unconscious movements and the subjective discomfort level of participants. This method has been applied to evaluating vehicle seats in a dri- ving simulator environment (Tatsuno et al. 2017). In this research, measurers visually checked the recorded movements of participants after the experi- ment. Then, it was found that this method had a high workload. Hence, the previous studies tried to develop other techniques to digitize participants’ movements. Specifically, the studies using motion capture systems (Tatsuno et al. 2018), flex sensors (Tatsuno et al. 2019), and body pressure distribu- tion measurement system (Tatsuno et al. 2020) were reported. Since these were instrument contact-constrained methods, there was an unquenchable concern. Thus, this study developed a measurement system that automati- cally extracts and classifies unconscious movements in a non-contact and non-constraint manner.
System Development Outline Figure 1 shows the configuration and processing flow of the system proposed in this research. In this research, an RGB-D camera (Intel, RealSense D435i) was used to construct a non-contact and non-constraint system. The RGB- D camera has a built-in RGB sensor and depth sensor. Applying the pose estimation technology Openpose (Cao et al. 2021) to images captured by the RGB sensor could calculate the two-dimensional coordinates (x, y) of joints. Moreover, adding the depth data z measured by the depth sensor could obtain three-dimensional coordinates (x, y, z). By processing the time series of angular change data of the body part, we could identify which part of the body moved. Subsequently, unconscious motions were classified into three types based on the combination of the body parts identified as causing the movement.
Motion Detection The 3D coordinates of the joints would change when the participant moved any body parts consciously or unconsciously. Therefore, in order to detect each body part motion, we programmed to sequentially calculate the changes in six directional vectors connecting the joints and two normal vectors to the plane composed of the two shoulders and the hip joints, as shown in Fig. 2.
In the subsequent process, we detected the occurrence of motion with the time-series data of the vector changes. This paper tried to utilize a change
Figure 1: Signal flow of our proposed system.Figure 2: Vectors of targeted body parts in the experiment.
points detection engine named ChangeFinder (Takeuchi and Yamanishi, 2006). ChangeFinder is an anomaly detection algorithm that can detect change points in time series data. So, we recognized the vector changes cau- sed by the motion as an anomaly applied ChangeFinder to a motion detection program.
Figure 3 shows the angle variation of the vector and the change point score calculated by ChangeFinder. A larger change point score means a significant change occurred in the time series data. Then, the time when the change point score output by ChangeFinder exceeded the threshold value were recorded by the program. Here, the values of the threshold were determined by trial and error. The sample graph in Fig. 3 shows the change point score exceeds
Detection of Unconscious Movements With RGB-D Camera 15
Figure 3: Sample graph to illustrate the change point score.
the threshold value seven times during the measurement period. Besides, the first point at which the threshold value is exceeded must be ignored due to the characteristics of the algorithm.
Classification of Movements Next, we created a program that can classify which of the three unconsci- ous movements, Type 1 (upper and lower limb movements), Type 2 (torso movements), and Type 3 (whole-body movements), from the recorded data. Figure 4 shows the classification algorithm based on the combination of body parts extracted by the motion detection program.
First, the detection window was slid, and body parts within the window area were listed. Second, after determining whether any body parts movement occurred in the window, the system returns to the window sliding step if there is no motion. Next, the window edge was shifted so that the motion that occurred earliest in the window became the first. After that, all body parts detected in the window were recognized as a single movement. That movement was classified according to the combinations of body parts, as shown in Table 1.
We incorporated a function to cancel the motion classification when we extracted any change in the direction vector connecting both wrists, in order to avoid false detection of conscious actions such as driving or instrument panel operation.
Evaluation of the Developed System A driving simulator experiment was conducted to evaluate the developed system (Figure 5).
A test course created in the VR environment is approximately 8 km long and has 18 vibration exposure sections with speed bumps. As connected vir- tually to the end and beginning terminals, participants could drive the vehicle as a circuit. In addition, there were two intersections where participants made right or left turns after a temporary stop. Participants in the experiment were
Figure 4: Flow chart of classification function.Table 1. Relationship between body part and movement type.
Table available in the original PDF paper.
Type # Torso Torso Thigh
Thigh Upper arm Upper arm
Forearm
(R) (L) (R)
(L) (R)
(L) (R)
Forearm (L) 1 1 √ √ 2 √ √
3 √ 4 √ 5 √ √ 2 1 √
2 √ 3 √ √ 3 1 √ √ 2 √ √
3 √ √ √ 4 √ √ 5 √ √ 6 √ √ √
7 √ √ √ 8 √ √ √ 9 √ √ √ √
asked to drive the DS for 60 minutes, and the experiment was filmed using the RGB-D camera installed on the top of the DS and a video camera set down on the side of the DS. After the experiment, we compared the freque- ncy of unconscious movements measured by the developed system with that using the visual observation of the video images.
In the experiment, healthy male university students with driver’s licenses were asked to participate. Before the experiment was designed, we obtained
Detection of Unconscious Movements With RGB-D Camera 17
Figure 5: Appearance of the experimental environment.
approval from the Bioethics Committee of the Faculty of Engineering, Kindai University (approval number: KUBE1404).
Table 2 shows the experimental results for three subjects. Precision and Recall were calculated as indicators of the detection accuracy of the program, and can be calculated using the following equations.
Equation reproduced from the original PDF.
As a result of calculating the total values, Precision and Recall became 28% and 41%, respectively. The reasons could be classified into two main categories. The first is a case in which the program falsely detected the occur- rence of a motion even though no unconscious movement occurred (false positive). The second case is that the program did not notice the event of a movement even though the movement was happening (false negative). It was considered that the cause of the false positives was a recognition error by Openpose. As shown in Fig. 6, the position of the left knee joint reco- gnized by Openpose changed even though the participant hardly moved his
Table 2. Evaluation results of our developed program.
Figure 6: Sample snapshots of participant when false positive occurred.
left thigh. Such a phenomenon would cause the program to detect unconsci- ous movement of the left thigh. A possible countermeasure was modifying the code to skip processing when the length between joints is short, like in Fig. 6(a). On the other hand, it was found that the cause of the false nega- tive was due to the threshold setting of the change point score. Therefore, we recognized the necessity of investigating the characteristics of the change point score in each body part.
CONCLUSION
This paper reported the development of a non-contact and non-restraint method to detect and classify unconscious movements that occur while riding in a vehicle. In this system, a commercial RGB-D camera was used as the optical system. A program installed skeleton estimation and change point detection as key software elements was developed. Specifically, change point scores were calculated from time series angular changes of body parts, and threshold processing was used to identify which body parts moved uncon- sciously. In addition, detected movements were classified into three types based on the combination of body parts. The results of a driving simulator experiment suggested the possibility of detecting and classifying unconscious movements with our developed system, although the accuracy needed to be improved. Thus, we proposed some improvements to decrease false positives and false negatives.
REFERENCES
Cao, Z., Hidalgo, G., Simon, T., Wei, S., Sheikh, Y. (2021). OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 43 No. 1. Sammonds, G. M., Fray, M., Mansfield, N. J. (2017). Effect of long term driving on
driver dis-comfort and its relationship with seat fidgets and movements (SFMs), Applied Ergonomics, Volume 58. Takeuchi, J., Yamanishi, K. (2006). A Unifying Framework for Detecting Outliers and Change Points from Time Series, IEEE Transactions on Knowledge and Data Engineering, Volume 18 No. 4. Tatsuno, J., Maeda, S., Sammonds, G., Mansfield, N. (2017). Possibility of auto-
mobile seat evaluation with seat fidgets and movements, Proceedings of the AHFE 2017 International Conference on Human Factors in Transportation, Los Angeles, CA. Tatsuno, J., Suyama, K., H., Nakamura, H., Maeda, S. (2020). Detection and Classification of Unconscious Movements with Body Pressure Distribution Measurement
Detection of Unconscious Movements With RGB-D Camera 19
for Ride Comfort Evaluation in Vehicle Seat, Proceedings of the AHFE 2020 International Conference on Human Factors in Transportation, San Diego, CA. Tatsuno, J., Suyama, K., Mitani, H., Nakamura, H., Maeda, S. (2019). Experimental Investigation of the Relationship between Human Discomfort and Involun- tary Movements in Vehicle Seat, Proceedings of the AHFE 2019 International Conference on Human Factors in Transportation, Washington D. C. Tatsuno, J., Suyama, K., Nakamura, H., and Maeda, S. (2018). Comparison of Dete-
ction Methods of Seat Fidgets and Movements in Vehicle Seat, Proceedings of 26th Japan Conference on Human Response to Vibration (JCHRV2018), Osaka, Japan.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Transportation
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-71-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional