Simão Ferreira, Matilde Rodrigues, Nuno Rocha (2022). Validation of a Video-based System to Determine Heart Rate for Stress Monitoring. In: Safety Management and Human Factors. AHFE Open Access, vol. 64, pp. 1–5. https://doi.org/10.54941/ahfe1002621
Studies estimate that about 50% of all lost workdays are related to occupational stress. Academic researchers have been using heart rate variability (HRV) as an indicator of stress. As a way of providing the needed heart rate data, an unobtrusive approach points to video plethysmography, being a recent method that needs further investigation and validation. Specific barriers such as room lighting conditions and face movement have been identified as the main risks for software progression. The present chapter presents a validation protocol of a video-based system to determine heart rate for stress monitoring, under different illuminance levels and position conditions. We present an in-depth protocol on how to assess the reliability of a video facial recognition software on collecting physiological data (heart rate), and our software results when compared to the gold standard, Electrocardiogram (ECG).
Studies estimate that about 50% of all lost workdays are related to occu- pational stress (HSE, 2019). At a national and pan-European level, the total estimated cost of work-related stress in 2014 was observed to be considerable and ranged substantially from 195.29€ million to 165€ billion. Productivity- related losses were observed to proportionally contribute most of the total cost of work-related stress (between 70 to 90%), and that healthcare and medical costs constitute the remaining 10% to 30% (Hassard et al., 2017). Therefore, solutions to mitigate risk factors related to the working settings present an enormous potential and a clear substantial contribution. There is a tremendous need for the development of applications/software that combines multiple sources of data to gather the information that can improve emplo- yees’ well-being, commitment, and performance. For more than ten years, academic researchers have been using heart rate variability (HRV) as an indi- cator of stress. Some systematic reviews even state that HRV provides an important window into understanding stress (Järvelin-Pasanen et al., 2018; Kim et al., 2018).
Video plethysmography comes up as a way of providing the needed heart rate data through an unobtrusive approach, being a recent method that needs further investigation and validation. Some previous laboratorial protocols have been developed to assess video-based heart rate monitoring systems; however, resting positioning and optimal solutions have been the primary focus (Kwon et al., 2012; Koenig et al., 2016; Sanyal and Nundy, 2018; Martins et al., 2021). Specific barriers such as room lighting conditions, face movement, and skin pigmentation have been identified as the main risk factors for software progression (Addison et al., 2018; Rodriguez and Castro, 2018; Chen et al., 2018). The present chapter presents a validation protocol of a video-based system to determine heart rate for stress monitoring, under different illuminance levels and face position. We present an in-depth pro- tocol on how to assess the reliability of a video facial recognition software on collecting physiological data (heart rate), and our software results when compared to the gold standard, Electrocardiogram (ECG).
LABORATORY PILOT Protocol Design
A laboratory pilot where we could compare our software outputs to an ECG was designed and conducted. Our pilot contained three separate experimental conditions of data collection. The participants had two separate cameras: one front-facing, and the other at 45 degrees, at all times. The main difference in the phases were scenarios with different illuminance levels, face positioning (through task vs resting), and low/high-resolution camera, and different levels of workload through Nback tasks and email writing (Figure 1).
Sample The participants included in the protocol were above 18 years old, with enough cognitive skills to read, interpret and understand the details of the study, as well as signing the participants ‘consent. We did not exclude par- ticipants with cardiac pathologies to address the software’s precision and sensibility. Every participant performed all the tasks in the protocol.
Environmental Conditions and Workstation The data collection room was prepared with the ability to change between different illuminance scenarios, monitored with a luxmeter (Delta-OHM, HD 2302.0), and monitored for air temperature and relative humidity (VelociCalc Multi-Function 9565-X, TSI), as well as for noise levels (Soundmeter, Bruel & Kjaer 2250-D00). The laboratory was arranged according to the current recommendations for illuminance levels (ISO 8995:2002), air temperature and relative humidity (Portuguese legislation). Additionally, it was ensured an appropriate workplace according to each subject’s anthropometric chara- cteristics (seat height, table height, backrest, harms rest, keyboard and mouse position, and visor height and inclination). These comfortable settings were kept during the different scenarios, except for the illuminance levels, as we assessed a video monitoring system under different lightning scenarios.
Validation of a Video-Based System to Determine Heart Rate for Stress Monitoring 3
Figure 1: Lightning conditions.
Measurements We started the protocol with three different scales for measuring stress, worry and coping strategies, being DASS-21 (Depression Anxiety and Stress Scale), PSWQ (Penn State Worry Questionnaire) and Brief Cope (Coping Orienta- tion to Problems Experienced) (Ribeiro et al., 2004; Jiménez-Ros et al. 2019; Carver, 1997). These scales are important to address baseline measurements, specially stress and coping strategies.
The ECG was collected using the BIOPAC’s MP36 Data Acquisition Unit, certified as a medical device, with a sampling rate of 500 Hz. The ECG signal was recorded using the electrode lead set SS2LB and disposable electrodes.
Having in mind different tasks for collecting heart rate, we decided to engage in two different tasks. Two N-back tasks of n-2 and n-3, and a pra- ctical email writing task. This workload was specifically selected due to its heart rate effects and cognitive workload (Vera et al., 2017; Zhou et al., 2022; Wang et al., 2022). The workload was assessed with NASA-TLX for each individual task (Hart, 2006).
Preliminary Results The laboratory pilot report the limitations of video plethysmography sof- tware and their reliability. In Table 1, we can see an example for an ECG recording, and bellow our systems average results.
It is important to mention some degree of difficulty regarding skin pigmen- tation and difference face shapes regarding heart rate detection. A review in
Table 1. Preliminary results for ideal lighting settings.
Table available in the original PDF paper.
Ideal lighting settings (example) 1st Minute 2nd Minute 3rd Minute 4th Minute 5th Minute First 3 minutes All 5 minutes
2016, mentioned that future work regarding stress detection, should be car- ried out combining information of different modalities and creating methods for data obtention in a totally unobtrusive way, necessary for practical real- life monitoring (Alberdi et al., 2016). This software is going to integrate a recommendation system after being validated as a reliable heart rate monitor.
In 2016, a development team pointed in the right direction, mentioning that creating a real-time, multiparameter physiological measurement plat- form with high video resolution video was the next step (Rahman et al., 2016). With the advance in video cameras availability, almost every laptop and mobile device is provided with high resolution cameras capable of producing high quality imagery for heart rate detection.
CONCLUSION
The laboratory pilots report the limitations of video plethysmography sof- tware. This validation and its methodology contribute to the development of future stress detection applications/software. It is important to mention the importance of a well-lighted work environment, the use of a recent webcam and the integration of a multiparameter physiological measurement in order to create a prediction algorithm for stress detection.
ACKNOWLEDGMENT
The authors would like to acknowledge the important contribution of the researcher Carlos Campos in developing this protocol, and also the important financial backup from the Mad@Work project: Mental Wellbeing Manage- ment and Productivity Boosting in the Workplace, funded by ITEA3 and Compete 2020 - POCI-01-0247-FEDER-046168|Lisboa-01-0247-FEDER- 046168.
REFERENCES
Addison, P. S., Jacquel, D., Foo, D. M. & Borg, U. R. 2018. Video-based heart rate monitoring across a range of skin pigmentations during an acute hypoxic challenge. Journal of clinical monitoring and computing, 32, 871–880. Alberdi, A., Aztiria, A. & Basarab, A. 2016. Towards an automatic early stress reco-
gnition system for office environments based on multimodal measurements: A review. Journal of biomedical informatics, 59, 49–75. Carver, C. S. 1997. You want to measure coping but your protocol’too long: Consider the brief cope. International journal of behavioral medicine, 4, 92–100.
Validation of a Video-Based System to Determine Heart Rate for Stress Monitoring 5
Chen, X., Cheng, J., Song, R., Liu, Y., Ward, R. & Wang, Z. J. 2018. Videobased heart rate measurement: Recent advances and future prospects. IEEE Transactions on Instrumentation and Measurement, 68, 3600–3615. Hart, S. G. NASA-task load index (NASA-TLX); 20 years later. Proceedings of the
human factors and ergonomics society annual meeting, 2006. Sage publications Sage CA: Los Angeles, CA, 904–908. Hassard, J., Teoh, K., Visockaite, G., Dewe, P. & Cox, T. 2017. The financial cost of work-related stress: a systematic review. Health and Safety Executive, 2019. Work-related stress, anxiety or depression
statistics in Great Britain. Health and Safety Executive London. Järvelin-Pasanen, S., Sinikallio, S. & Tarvainen, M. P. 2018. Heart rate variability and occupational stress-systematic review. Ind Health, 56, 500–511. Jiménez-Ros, A. M., Carmona-Márquez, J. & Pascual, L. M. 2019. Pathological
worry in portugal: The portuguese version of the Penn State Worry Questionnaire (PSWQ). The Spanish journal of psychology, 22. Kim, H.-G., Cheon, E.-J., Bai, D.-S., Lee, Y. H. & Koo, B.-H. 2018. Stress and heart rate variability: a meta-analysis and review of the literature. Psychiatry investigation, 15, 235. Koenig, N., Seeck, A., Eckstein, J., Mainka, A., Huebner, T., Voss, A. & Weber,
S. 2016. Validation of a new heart rate measurement algorithm for fingertip recording of video signals with smartphones. Telemedicine and e-Health, 22, 631–636. Kwon, S., Kim, H. & Park, K. S. Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone. 2012 annual international conference of the IEEE engineering in medicine and biology society, 2012. IEEE, 2174–2177. Martins, P., Barbosa, T., lima, l., Souza, I., Ramos, C., Souza, B., da Rocha, F.,
Barroso, S. & Vitorino, O. 2021. Validation study of the use of the HRVCAM software for the evaluation of heart rate and heart rate variability at rest. European Heart Journal - Digital Health, 3, 98–104. Rahman, H., Ahmed, M. U., Begum, S. & Funk, P. Real time heart rate monitoring from facial RGB color video using webcam. The 29th Annual Workshop of the Swedish Artificial Intelligence Society (SAIS), 2–3 June 2016, Malmö, Sweden, 2016. Linköping University Electronic Press. Ribeiro, J. L. P., Honrado, A. & Leal, I. 2004. Contribuição para o estudo da ada-
ptação portuguesa das escalas de ansiedade, depressão e stress (EADS) de 21 itens de Lovibond e Lovibond. Rodriguez, A. & Castro, J. 2018. Video pulse rate variability analysis in stationary and motion conditions. Biomedical engineering online, 17, 1–26. Sanyal, S. & Nundy, K. K. 2018. Algorithms for monitoring heart rate and respiratory
rate from the video of a user’s face. IEEE Journal of translational engineering in health and medicine, 6, 1–11. Vera, J., Jiménez, R., García, J. A. & Cárdenas, D. 2017. Intraocular pressure is sensitive to cumulative and instantaneous mental workload. Applied Ergonomics, 60, 313–319. Wang, X., Lavender, S. A., Sommerich, C. M. & Rayo, M. F. 2022. The effects of using
a footrest during computer tasks varying in complexity and temporal demands: A postural and electromyographic analysis. Applied Ergonomics, 98, 103550. Zhou, H., Molesworth, B. R., Burgess, M. & Hatfield, J. 2022. The effect of broadband noise on learning and dynamic decision-making and how cognitive workload and sex moderate its effect. Applied Ergonomics, 98, 103604.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Safety Management and Human Factors
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-40-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional