Jichen Han, Xiaozhou Zhou, Chenglong Zong, Fei Teng (2023). The feasibility of fatigue state monitoring based on eye parameters in virtual reality environment. In: Intelligent Human Systems Integration (IHSI 2023): Integrating People and Intelligent Systems. AHFE Open Access, vol. 69, pp. 246–254. https://doi.org/10.54941/ahfe1002843
In human‒machine environments, real-time fatigue monitoring of operators is particularly important. Excessive fatigue will threaten the operation efficiency and even the safety of operators. At present, experimental research methods include simulation experiments, half-physical simulations and real environment tests. The simulation environment experiment has the advantages of low cost, high security, high repeatability and anti-interference from external factors. The virtual reality (VR) environment is a virtual scene based on a 3D virtual model, which is an important development trend of simulation experiment. However, the reliability of various measurement methods in VR simulation environment needs to be verified. In this study, the Varjo XR3 VR device with the best display effect currently on the market is used together with headphones to construct a closed simulation experiment scene and at the same time collect subjective evaluation of monotonous fatigue and eye parameters. Then, the mapping relationship between the two is explored through confirmatory experiments. A total of 10 subjects were recruited in this experiment to induce fatigue by cruising tasks with low load in flight missions. Subjects needed to independently complete the route cruise task for approximately 50 minutes and report their own fatigue subjective evaluation parameters using the Borg scale at the flight turning points every 3 minutes. The data of percentage of eyelid closure over the pupil over time (PERCLOS) and blink rate were obtained by using head display internal camera image and projection vectors algorithm, and the validity of eye data was evaluated by correlation validation with monotonic fatigue subjective evaluation parameters. The results show that the quality of eye parameters collected by VR meets the monitoring requirements, and parameters such as PERCLOS and blink rate have a high correlation with subjective monotonic fatigue evaluation. Eye monitoring has high availability to evaluate operator monotonic fatigue in virtual simulation environment.
In man-machine systems, operator fatigue monitoring is particularly impor- tant. According to the official definition of the International Civil Aviation Organisation (2018), fatigue is a physiological state of reduced mental and physical performance. Several studies in the construction industry (Wagstaff and Lie, 2011; Wong, 1994), pilots (Cabon et al., 2012; Sauvet et al., 2014), and transportation (Schleifer et al., 1989; Okogbaa et al., 1994) have shown
The Feasibility of Fatigue State Monitoring Based on Eye Parameters 247
that excessive operator fatigue severely affects the efficiency and safety of completing tasks.
Among the existing techniques for operator fatigue assessment, eye para- meters obtained by behavioral monitoring have the advantages of low inva- siveness and high correlation. For physiological monitoring methods such as respiration, ECG, and EEG, the current monitoring methods are mostly invasive studies conducted under laboratory conditions, and the research methods themselves are more intrusive to the flight mission, which is still far from being translated into monitoring applications in the actual flight environment. Compared to physiological measurements, human behavior measurements are easier to achieve in a non-invasive manner. With advances in technologies such as graphic recognition and machine learning, monito- ring eye states and expressions can be easily achieved with a single camera. Studies have verified that blink, PERCLOS, pupil analysis, and other related data obtained from eye behavior states are highly usable for operator fatigue assessment (Faure et al., 2016; Diaz-Piedra et al., 2019; Diaz-Piedra et al., 2021). Comparing various fatigue assessment experimental methods, simulation experiments have the advantages of low loss, high safety and high repe- atability compared with other methods. The existing studies on fatigue assessment mainly include simulation experiments, semi-physical simulation experiments and real environment experiments. In real environment experi- ments, the data obtained from subjects are generally the most realistic, but fatigue assessment experiments require subjects to enter different fatigue sta- tes for testing, and when subjects are in a medium to high level of fatigue, the work errors caused by too low a state will put the safety of subjects at risk. Semi-physical simulation experiments, that is, the use of screens or projecti- ons instead of the experimental external environment, are used to ensure that the subjects are in a relatively safe environment for different fatigue states of the test, such as Zhang (2020) and other research cases in the experimental method, but the method still needs to be built close to the real experimental equipment, and scene construction costs and program changes are unfavora- ble. In simulation experiments that only retain a small number of interactive physical devices, the remaining scenes are built using virtual scenarios. The experimental method can quickly build different experimental scenarios to reduce experimental damage and speed up the experimental process while ensuring a sense of experimental immediacy.
With the development of VR technology, the VR environment has gra- dually become an important component of the simulation experimental environment, but whether updated equipment can be used for fatigue asses- sment experiments still needs to be verified. The high immersion experience brought by VR headsets can provide a sense of presence and realism while greatly avoiding the interference of external factors during the test. With continuous breakthroughs in VR headset performance, the images presen- ted by the new generation of VR headsets are clearer and closer to reality, but whether such devices can be used in fatigue assessment still needs to be verified.
In this paper, we attempted to use the Varjo XR3 device as a validation target to collect eye state information and subjective evaluation information of mental fatigue in a flight simulation environment built by this VR device to test whether monotonic tasks in VR environment can induce fatigue quickly. The consistency between the eye state information and the subjective fatigue evaluation information under monotonic tasks was also explored to further verify whether the fatigue level of subjects under monotonic tasks could be captured in real time by eye state monitoring under this device.
METHODS
In this paper, we used a Varjo XR3 device with a headset to construct a closed simulation experiment scenario and extracted two ocular parameters, PERCLOS and blink rate, from the eye behaviour images captured by the internal camera device by a projection vector algorithm. The subject’s fatigue was induced by a simulated flight cruise experiment with low load, and the subject’s subjective fatigue during the experiment was recorded using the Borg scale. The purpose of the experiment was to verify the quality of the ocular parameters collected by the Varjo XR3 device and the correlation between this parameter and the subjective evaluation of the monotonic fatigue level and to verify the feasibility of using this ocular parameter for the assessment of the subject’s fatigue level in the simulation experiment.
Subjects The subjects were 10 college students aged 22–27 years, including 6 males and 4 females. All subjects were in good health, had no history of drug use, and had normal naked or corrected vision. All subjects were not allowed to consume alcohol or caffeine within 24 hours prior to the formal experiment and had adequate sleep.
To ensure that the experimental results would not be affected by chan- ges in the subjects’ task proficiency, a subject training session was arranged before the formal experiment. The training tasks included flight instrument recognition and cruise manoeuvres. The subjects were guided to learn crui- sing skills and practice them independently. When the subjects were able to maintain stable flight and complete the cruising turns in various directions within 5 seconds, they were considered to have passed the proficiency test. All subjects passed the training and assessment, and the total training time ranged from 1–2 hours.
Experimental Platform As shown in Figure 1, the experimental platform consists of the Varjo XR3 headset, a wired headset and the Thrustmaster A10C flight joystick. The three devices are connected to a workstation with two HTC vive base stations to assist in positioning the headset. Among the VR headsets currently availa- ble on the market, Varjo XR3 has the advantage in display performance, and the high-definition picture provided by the device can build a visual expe- rience closer to the real one for the experiment. The experiment simulation
The Feasibility of Fatigue State Monitoring Based on Eye Parameters 249
Figure 1: Experimental platform and experimental scene. The subjects were required to follow the designated route to complete the cruise task with the assistance of the central HUD and the radar panel in the lower right corner.
scene is built by Digital Combat Simulator World (DCS World) simulation fli- ght software, Varjo XR3 headset through Varjo base and steam VR software drive to present three-dimensional simulation experiment picture, and with wired headset transmission simulation environment sound, constitute a clo- sed simulation experiment environment. The headset has a built-in camera device that captures images of the subject’s eyes and presents them in the Varjo base software, and the experiment uses Open Broadcaster Software (OBS) software to record images of the subject’s eye behavior at 60 Hz. The subject was able to control the simulated fighter by using the flight joystick to complete the cruise task required by the experiment. To reduce the impact of the weight of the head-up display on the subject, a soft cushion was added to the contact part of the head-up display to reduce the pressure of the head-up display on the subject’s head.
Experimental Tasks The main purpose of this experiment was to induce subject fatigue quickly by using a monotonous task with low difficulty and low variability to induce a high level of subject fatigue in a relatively short period of time. However, the fatigue induced by monotonous tasks can be easily relieved by changes in the task, communication with others, etc., so that the subject’s fatigue level can be quickly reduced. Therefore, it is necessary to minimise the variability in the experimental task.
As shown in Figure 2, the experimental task was a 50-minute path cruise task with a constant airspeed of the simulated fighter, and the subject only needed to follow the path guided by the HUD and radar panel to control the fighter’s direction to complete a single cruise task. To maintain the low variability of the task complexity and to minimise the relief of monotonous fatigue by the subject’s subjective fatigue reporting, the experimental task set the steering point every 3 minutes, and the steering angle was set between 30–50 degrees, while the subject was asked to make a subjective report on his fatigue level according to the Borg scale at the same time. The subject received the report and recorded it simultaneously, thus avoiding interfering with the subject’s state by communicating with him/her. The first turning point was set 2 minutes after the start of the experiment, and data recording was started to ensure that the subjects entered the experiment. The total path
Figure 2: Experimental task process.
time of the experimental task was longer than 60 minutes, and the subjects were not informed of the exact total task time. The subject called off the experiment at 50 minutes after the subject passed the 16th turning point to prevent the subject’s sprint state near the end of the experiment from affecting the experimental results (Morales et al., 2017).
EXPERIMENTAL OBSERVED VARIABLES Projection Vector Algorithm and Eye Parameter Extraction
The Varjo XR3 headset has a built-in camera that can capture eye images that tend to be orthogonal during the subject’s wearing process, and this experi- ment uses a projection vector algorithm and an auxiliary graphic correction algorithm to extract the subject’s eye parameters. The process is shown in Figure 3. First, the black occluded area in the original image is replaced with white and divided into left and right eye parts for binarization. The second step is the angle calibration, and the acquired eye image is rotated to hori- zontal with the closed-eye frame image as the reference, and the vacant area generated by the rotation is replaced with white. The third step is the integra- tion projection of the calibrated image along the horizontal direction, where the number of black pixels in each row is integrated and presented. The last step is to remove the shadow interference around the eye to extract the eyelid height, calculate the average value of black dots in each row and zero out the data below the average value, while finding the boundary from the center of the black dot peak row to both sides to further eliminate the data in the non-eye- area, so that the number of non-zero rows is the value of eyelid height.
Based on the eyelid height values obtained by the algorithm, the PERCLOS and blink rate per unit time can be further calculated. The two parameters are calculated using P80 as the standard (Dinges et al., 1998); that is, frames with eyelid heights less than 80% of the maximum height are considered closed eyes, and the relevant calculation formula is as follows.
∗80% (1)
dmax −dmin
Equation reproduced from the original PDF.
The Feasibility of Fatigue State Monitoring Based on Eye Parameters 251
Figure 3: Process of eye parameter extraction.
!
#
Equation reproduced from the original PDF.
where dmax is the maximum distance of eyelid closure per unit time, and dmin is the minimum distance of eyelid closure per unit time. di is the dista- nce between the upper and lower eyelids at time i, and Pi is the eye-closed state at time i. When di > dclose, Pi=1, indicating that the current subject is recognised as the closed-eye state. Ti is the total number of sampled time points in a round of experiments. The blink rate is the number of consecutive eye closure segments per unit time.
Borg Scale The Borg scale is a subjective rating scale commonly used in experimental studies in medicine and psychology and is mainly used to assist subjects in self-perception ratings of the severity of certain conditions. As shown in Figure 4, the subjects in this experiment used the Borg CR10 scale for real-time subjective fatigue scoring.
RESULTS
This experiment used the Python program to process the data. Using the 30th-31st minute data as a benchmark, this experiment compares the blink
Figure 4: Subjective fatigue assessment scale set by the Borg CR10 scale.Figure 5: The trend of subjects’ PERCLOS, blink rate and subjective fatigue level.
rate statistics from the original eye video with the data derived from the pro- gram processing to confirm the accuracy of the eye parameters calculated by the program. Each subject’s statistical video clip was 48 minutes long, and a set of PERCLOS and blink rate data was generated for each right and left eye at 3-minute intervals, with one subjective evaluation data corresponding to it. To avoid physical differences between individuals, PERCLOS and blink rate data were Z score normalised. The original data distribution of the 10 subjects is shown in Figure 5.
For each group of data using Pearson correlation and Spearman correlation analysis, the correlation results for the overall data are shown in Table 1.
DISCUSSION
The results showed that after a 50-minute low-load cruising task, the subje- ctive fatigue level of the subjects showed a significant upwards trend, and the degree of change in the subjective fatigue level of most of the subjects could
The Feasibility of Fatigue State Monitoring Based on Eye Parameters 253
Table 1. Correlation results between subjective fatigue scores and eye parameters.
Table available in the original PDF paper.
Parameter 1 Parameter 2 Pearson Spearman r Sig. ρ Sig.
reach six levels (from level 2 to level 8), indicating that the experimental task could effectively stimulate a rapid increase in the fatigue level of the subjects.
The PERCLOS and blink rate parameters acquired by Varjo XR3 and pro- cessed by the algorithm were generally consistent with the visual statistics after the sampling test. The correlation analysis showed that the correlation between PERCLOS and subjective fatigue was greater than 0.6, showing a strong correlation. The correlation between blink rate and subjective fatigue was greater than 0.4, which was a moderate correlation. PERCLOS was more relevant than blink rate in assessing subjects’ fatigue, which is consistent with previous studies (Laouz et al., 2020).
CONCLUSION
In this paper, a flight driving simulation environment was built using a VR device with excellent display performance, the subject’s eye parameters were successfully captured using the built-in camera and projection vector and image processing algorithms, and the subjective fatigue assessment level during the test was obtained using the Borg CR10 scale. By designing a low load and low difficulty simulated cruising task, a large change in the sub- ject’s fatigue level was successfully induced in a relatively short period of time. Through Pearson correlation and Spearman correlation analysis, the moderate to high correlations between the two collected ocular parameters, PERCLOS and blink rate, and the subjective fatigue level were verified, fur- ther validating the feasibility of using the ocular parameters to assess the changes in the subject’s fatigue level in this VR simulation environment.
ACKNOWLEDGMENT
This work is supported in part by the National Natural Science Foundation of China (No. 71901061 and 52275238).
REFERENCES
Cabon, P., Deharvengt, S., Grau, J. Y., Maille, N., Berechet, I., & Mollard, R. (2012).
Research and guidelines for implementing Fatigue Risk Management Systems for the French regional airlines. Accident Analysis & Prevention, 45, 41–44. Diaz-Piedra, C., Rieiro, H., & Di Stasi, L. L. (2021). Monitoring army drivers’ workload during off-road missions: An experimental controlled field study. Safety science, 134, 105092.
Diaz-Piedra, C., Rieiro, H., Cherino, A., Fuentes, L. J., Catena, A., & Di Stasi, L. L.
(2019). The effects of flight complexity on gaze entropy: An experimental study with fighter pilots. Applied ergonomics, 77, 92–99. Dinges, D. F., Mallis, M. M., Maislin, G., & Powell, J. W. (1998). Evaluation of techniques for ocular measurement as an index of fatigue and as the basis for alertness management (No. DOT-HS-808-762). United States. National Highway Traffic Safety Administration. Faure, V., Lobjois, R., & Benguigui, N. (2016). The effects of driving environment
complexity and dual tasking on drivers’ mental workload and eye blink behavior. Transportation research part F: traffic psychology and behaviour, 40, 78–90. International Civil Aviation Organisation. (2018). ICAO safety management manual (Doc. 9859-AN/474). Laouz, H., Ayad, S., & Terrissa, L. S. (2020, June). Literature Review on Driver’s Dro-
wsiness and Fatigue Detection. In 2020 International Conference on Intelligent Systems and Computer Vision (ISCV) (pp. 1–7). IEEE. Morales, J. M., Díaz-Piedra, C., Rieiro, H., Roca-González, J., Romero, S., Catena, A., ... & Di Stasi, L. L. (2017). Monitoring driver fatigue using a single- channel electroencephalographic device: A validation study by gaze-based, driving performance, and subjective data. Accident Analysis & Prevention, 109, 62–69. Okogbaa, O. G., Shell, R. L., & Filipusic, D. (1994). On the investigation of the
neurophysiological correlates of knowledge worker mental fatigue using the EEG signal. Applied Ergonomics, 25(6), 355–365. Sauvet, F., Bougard, C., Coroenne, M., Lely, L., Van Beers, P., Elbaz, M., ... & Chennaoui, M. (2014). In-flight automatic detection of vigilance states using a single EEG channel. IEEE Transactions on Biomedical Engineering, 61(12), 2840– 2847. Schleifer, L. M., & Amick III, B. C. (1989). System response time and method of pay:
Stress effects in computer-based tasks. International Journal of Human-Computer Interaction, 1(1), 23–39. Wagstaff, A. S., & Lie, J. A. S. (2011). Shift and night work and long working hours-a systematic review of safety implications. Scandinavian journal of work, environment & health, 173–185. Wong, T. W. (1994). Occupational injuries among construction workers in Hong
Kong. Occupational medicine, 44(5), 247–252. Zhang, X., Wang, X., Yang, X., Xu, C., Zhu, X., & Wei, J. (2020). Driver drowsiness detection using mixed-effect ordered logit model considering time cumulative effect. Analytic methods in accident research, 26, 100114.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2023): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-45-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional