Edoardo Spairani, Rita Paradiso, Giovanni Magenes (2024). Deep learning based Human Activity Recognition in first responders wearing a sensorized garment. In: Cognitive Computing and Internet of Things. AHFE Open Access, vol. 124, pp. 33–42. https://doi.org/10.54941/ahfe1004701
Safety and well-being of first responders operating in hazardous environments are paramount considerations. These individuals routinely find themselves immersed in dangerous situations, leading to heightened levels of both physical and mental stress. In this context, a system for the automatic and real-time monitoring of first responders' (FRs) activities could play an important role in timely identifying potentially dangerous situations. The present paper addresses this issue and introduces a Deep Learning (DL) based Human Activity Recognition (HAR) approach for the automatic identification of tasks carried out by first responders. In our proposed framework, we leverage the use of a garment equipped with various integrated sensors to capture both physiological and inertial measurements during the course of first responders' duties. For this aim we harness the power of DL techniques, specifically recurrent neural networks (RNNs), aiming at achieving an accurate classification of a limited set of diverse tasks. To validate the efficacy of our proposed system, we conducted the evaluations on a comprehensive hold-out set compiled from real-world scenarios, involving FRs. The results of our evaluation showcase not only high accuracy (0.9813) but also robust reliability in classifying the activities undertaken by the operators. The implications of our deep learning-based activity recognition framework extend beyond mere classification, since gaining insights into the risk associated to a particular task performed could enable the development of more effective, timely and safer emergency response strategies.
In recent years, wearable devices have become an established reality, finding application in an expanding array of fields (Smith et al., 2023). Wearables that capture physiological signals, in particular, have proliferated, transcend- ing their traditional role in medical applications. Presently, these wearables play a pivotal role as primary data sources in sophisticated systems that seamlessly integrate with human activities across various domains, including work, home, and sports. This evolution underscores the growing centrality of physiological wearables as indispensable components in complex sys- tems designed to interact with individuals throughout their daily routines
(Smuck et al., 2021). Over the last few years, a great attention has been posed on wearable systems aiming at supporting operators working in high risk con- ditions (Curone et al., 2012; Bonfiglio et al., 2011; Magenes et al., 2011). The present work is situated within the context of the European project SIXTHSENSE (https://sixthsenseproject.eu), which goal is to develop intel- ligent solutions for supporting at-risk operators such as firefighters, forest workers, mountain rescuers etc., whose daily work duties expose themselves to potentially hazardous situations. Within this overarching framework, our presented study aligns with the broader mission of SIXTHSENSE by contributing insights into the development of intelligent systems that can enhance the safety and efficiency of first responders (FRs) in the face of their challenging work environments.
More in detail, the overarching goal of the present study regards the devel- opment and implementation of a Deep Learning (DL) based Human Activity Recognition (HAR) approach, based on physiological and inertial signals col- lected through wearable garments. In a working context that continuously exposes first responders to heightened physical and mental stress, a system for the real-time monitoring of FRs activities could play an important role for their safeguarding.
This task has yet been attempted by our research group in the frame of the European Project ProeTEX (PROtective Electronic TEXtiles for emergency operators) (www.proetex.org), which aimed at demonstrating the suitability of wearable technologies to improve the safety, efficiency, and coordination of emergency operators, such as fire fighters or Civil Protection rescuers et al., 2010; Curone et al., 2010). In particular, Curone et al. proposed a simple system to classify the performed activity based on the fusion of a set of inertial-derived features as well as heart rate (HR) (Curone et al., 2010).
In the present work we want to move further, by exploiting the inherent flexibility of DL, proposing a system, relying on recurrent neural networks (RNNs), as implemented for example in (Paydarfar et al., 2020) (Murad et al., 2017), hopefully capable of accurately classifying a limited set of 8 typical human actions (standing, sitting, laying, walking, running, jumping, walking uphill, walking downhill) which are common both to everyday life scenarios and to first responders’ duties. Particularly, the developed system consists of a Gated Recurrent Unit (GRU) net (Cho et al., 2014) which is fed with a set of physiological and inertial recordings collected through a wear- able garment. The adoption of RNNs in our approach serves to enhance the efficiency of feature extraction by enabling the automatic identifica- tion of intrinsic features embedded within the signals. Unlike traditional machine learning methods that rely on manual or pre-defined feature extrac- tion techniques (Gorjani et al., 2021), RNNs possess the inherent capability to capture temporal dependencies and patterns present in sequential data. This adaptability allows our model to autonomously discern and extract nuanced features, providing a more comprehensive understanding of the dynamic information encoded within the signals. By leveraging the power of RNNs, our approach not only streamlines the feature extraction pro- cess but also harnesses the rich contextual information encapsulated in the raw signals, contributing to effectiveness of the classification process of our approach.
Deep Learning Based Human Activity Recognition in First Responders Wearing 35
The rationale behind considering the integration of accelerometric and physiological signals is rooted in the fact that each source provides only par- tial information concerning the actual level of physical activity of a subject (Brage et al., 2005). While accelerometers can identify a wide array of actions, they fall short in assessing the physical effort exerted during a movement (e.g., they cannot distinguish between activities such as “walking on a level surface” and “walking uphill”) (Fehling et al., 1999). Conversely, physiolog- ical parameters like heart rate (HR) capture the physical effort involved in an activity (using the previous example, HR is higher when a subject “walks uphill” compared to “walks on a flat surface”), but they may be subject to external influences such as psychological stress or environmental conditions.
The primary aim of this work is not solely to create a model capable of identifying a highly specific set of activities. Rather, the goal is to explore how the combined use of a particular type of wearable devices, coupled with advanced deep learning methods, has the potential to lead the development of intelligent systems capable of providing efficient support to operators dur- ing their work activities. Our work serves not only as a proof of concept, but also as an indication of how, in subsequent phases, the model could be further trained to recognize specific events intrinsic to the performed work activities. By demonstrating this broader potential, we highlight the capacity for intelligent systems to adapt and enhance their functionality based on the unique demands of professional tasks.
MATERIALS AND METHODS
The Employed Sensorized Garment and the Collected Signals A scheme depicting the garment employed in this study is illustrated in Figure 1 and consists of a wearable textile system produced by Smartex srl. (https://www.smartex.it/). This system, built upon the Wearable Wellness System (WWS), underwent modifications and tailoring to fulfill the specifi- cations of the SIXTHSENSE project, wherein various sensors are integrated into a unified wearable prototype. More specifically, for what concerns phys- iological signals, the garment is equipped with textile electrodes for the ECG acquisition and a textile piezoresistive sensor for breath measurement, based on the thorax expansion caused by breathing. These sensors are made of yarns and are fully integrated into the textile structure that composes the garment.Kinematic measurements, on the other hand, are detected through a InvenSense MPU-9250 IMU. The latter MPU-9250 is a multi-chip module (MCM) consisting of two dies integrated into a single QFN package. One die houses the 3-Axis gyroscope and the 3-Axis accelerometer; the other die houses the AK8963 3-Axis magnetometer. Hence, by using this system, 11 signals are registered at each acquisition, i.e., 2 physiological tracings (ECG and breathing signal) and 9 inertial signals from the IMU (3 readings from the accelerometer, 3 from the magnetometer and 3 from the gyroscope). ECG data are collected with a sample frequency (fs) of 250 Hz, while respira- tory signal and IMU measurements are collected with fs = 25 Hz. Starting from ECG and breathing signals, heart rate and breathing rate series are then obtained with a classical peak detection algorithm based on the one presented
by Pan and Tompkins (Pan et al., 1985) and then resampled at the same sample frequency of the inertial signals (fs = 25 Hz). An illustration exempli- fying the total set of kinematic signals, together with the two series derived from physiological acquisitions (HR and BR) collected within a 50 seconds acquisition is shown in Figure 2.
Figure 1: Illustration of the employed WWS. The vest is equipped with two textile electrodes to collect the ECG signal, one piezoresistive textile sensor to measure respi- ratory signals and a data logger (RUSA device). A 9 d.o.f IMU integrated in the R.U.S.A device leads to collect 9 kinematic tracings at each acquisition (3 readings from the accelerometer, 3 from the magnetometer and 3 from the gyroscope).Figure 2: Illustration of a 50 seconds excerpt for the 3 magnetometer, accelerometer and gyroscope readings and the two derived measures (heart rate and breathing rate signals) acquired through the Smartex WWS.
Deep Learning Based Human Activity Recognition in First Responders Wearing 37
Data Collection: Training and Testing Set To facilitate the training of the proposed neural architecture, a controlled data collection process was undertaken. Three participants were enrolled (2 males and 1 female aged 27±5). Participants were equipped with the Smartex wear- able garment and were asked to perform sequences of predefined actions, encompassing typical activities in controlled environmental settings. Partic- ularly 8 actions were considered: standing, sitting, laying, walking, running, jumping, walking uphill and walking downhill. These activities are common both to everyday life scenarios and to first responders’ duties. A corpus of 15 sequences with varying length was considered for the training phase. The lat- ter were normalized to be in range 0–1 in order to facilitate the model training and were split in 2 seconds chunks with a 0.5 stride. With this choice, each input to the net consists of a tensor with 50 points in time and 11 dimensions, one per each signal. To avoid training biases a process of class balance was applied to ensure each activity was equally represented within the training set. After the training phase, the performances of the proposed HAR model were tested on a hold out test set including 30 signals acquired in real life scenar- ios, involving FRs. Acquisitions were performed during scheduled sessions in Bormio (Italy, 2022), Postojna (Slovenia, 2022), Kopaonik (Serbia, 2023), Rijeka (Croatia, 2023) over the three-year lifespan of the European project SIXTHSENSE.
The Developed Neural Architecture The proposed neural architecture is illustrated in Figure 3. Our model archi- tecture, implemented using the Keras deep learning framework, comprises a sequential stack of Gated Recurrent Unit (GRU) layers, designed to cap- ture and learn temporal dependencies within sequential data. The model begins with a GRU layer consisting of 128 units and is configured to return sequences, providing a deeper understanding of the input data’s tempo- ral dynamics. A dropout layer with a probability of 0.4 is incorporated after the first GRU layer to prevent overfitting. The subsequent layers follow a similar pattern, incorporating additional GRU layers with decreas- ing units (64 units and 32 units, respectively) and maintaining the return sequences configuration. Dropout layers are interspersed after each GRU layer to further enhance the model’s generalization capabilities. Following the GRU layers, densely connected layers contribute to the model’s abil- ity to capture non-linear dynamics. Two dense layers with 32 units each and ReLU activation functions are introduced, providing the network with the capacity to understand complex patterns in the learned features. The final dense layer consists of 8 neurons with a softmax activation function which returns the probability of the input chunk to belong to one of the 8 possible activities. The model training was stopped by the adopted anti overfitting criteria after 200 epochs. Adam optimizer was employed with a learning rate of 1e-4 and categorical cross entropy was considered as loss function.
Figure 3: Scheme of the proposed neural network. The model consists of a series of 3 GRU layers interspersed by as many dropout layers. These blocks are followed by two dense layers with ReLU activation functions, always interspersed by dropout layers. The final layer consists of a dense one, with one neuron for each activity to classify and with Softmax activation.Table 1. Confusion matrix per activity.
Table available in the original PDF paper.
RESULTS
The obtained results are summarized in the confusion matrix in Table 1, the analysis of which reveals the trained model’s accuracy in classifying each of the 8 activities considered. Static positions (‘stand’, ‘sit’, ‘lay’) were all cor- rectly identified more than 98% of times and this underscores the model’s adeptness in capturing subtle temporal dynamics inherent in stationary pos- tures. Little misclassifications occurred, especially for what concerns the “sitting” activity, which was confused 1% of times with both standing and sitting postures. This may be attributed to various factors, including the inher- ent variability in subjects’ heights and the diverse heights of objects on which individuals may sit. Even for what concerns, instead, non-standing activities it could be observed that the proposed DNN is able to achieve good classi- fication performances. In particular, ‘running’ and ‘walking uphill’ activities were correctly spotted 98% of times, while ‘jumping’ and ‘walking down- hill’ were correctly identified 99% of times. The activity of “walking” seems
Deep Learning Based Human Activity Recognition in First Responders Wearing 39
to be the most misclassified one. More in detail the latter was classified as “walking uphill” (2% of times) and “walking downhill” (2% of times) and 1% of times as “running”. These nuanced misclassifications merit further investigation to refine the model’s discriminative prowess, especially in dis- ambiguating closely related activities. There’s however to take into account that the confusion matrix shown in Table 1 summarizes the obtained DNN’s performances on real experimental data; despite rigorous efforts, the label- ing process may introduce limited errors. For example, instances where the expected activity is ‘walking’ may contain few portions where the subject is, for some reasons, ‘walking’. To quantify the overall performances of the pro- posed DNN we performed the computation of diverse score metrics, which are defined as follows (equations from 1 to 9): Accuracy = TP + TN TP + FP + FN + TN (1)
Equation reproduced from the original PDF.Equation reproduced from the original PDF.
FN, FP, TN and TP in equations respectively indicate the number of False Negative, False Positive, True Negative and True Positive.
Performance scores obtained with the presented model are reported in Table 2. The accuracy of 98.13% (error = 0.0187) reflects the overall cor- rectness of the DNN’s predictions, demonstrating a high level of agreement between predicted and actual classes and highlighting its robustness in min- imizing prediction errors. This is consistent for what concerns the recall (0.9813), which indicates the DNN’s ability to effectively capture positive instances, showcasing its sensitivity to the presence of the target class. High levels of specificity and precision are also achieved (0.9973 and 0.9813) and the low false positive rate (FPR) of 0.27% signifies the DNN’s minimal ten- dency to generate false alarms, ensuring a high level of precision in its positive predictions. The F1 score, which provides a balanced measure of the DNN’s precision and recall, has a value of 98.12% indicating a harmonious trade- off between precision and sensitivity. The Matthews Correlation Coefficient
(MCC) of 97.86% reflects the strength of the relationship between predicted and observed classifications, considering both true and false positives and negatives. Finally, a Kappa value of 0.9143 reflects the DNN’s high level of agreement between predicted and actual classes.
DISCUSSION AND CONCLUSION In the present work we tackle the challenge of classifying a set of activities performed by First Responders (FRs) using Deep Learning (DL), and par- ticularly RNNs. Leveraging inertial and physiological data collected from wearable devices, our focus extends beyond mere classification, delving into the nuances of real-world scenarios faced by FRs. The efficacy of our DL- based HAR approach is evaluated, shedding light on its potential to enhance safety of operators working in hazardous environments. The presented DNN based on GRU successfully classifies first responders’ activities, achieving an overall accuracy of 98.13% and showcasing robust performance across var- ious metrics. Beyond mere specific activity classification, the work serves as a proof of concept, indicating the broader potential of the analyzed system integrating the use of a wearable device and DL. In fact, the core concept goes beyond the pursuit of the most efficient architecture or the classifica- tion of specific actions. Respect to other existing works (e.g. Curone et al., 2010) which focus on the detection of specific activities, our primary goal is to demonstrate the potentials of combining DL techniques, specifically RNNs, with signals acquired from high-quality wearable garments, such as the WWS from Smartex s.r.l. This synergy is not only about accurate classifi- cation but, more importantly, about playing a pivotal role in safeguarding operators working in high-risk conditions. Additionally, this study marks an initial stride toward the development of a real-time system which could eventually be adopted in real life scenarios. A system of this kind could play an important role in increasing safety of FRs, providing a means to con- stantly monitor the physical and mental activity of operators in high-risk environments, enabling the timely identification of potentially dangerous sit- uations and allowing for more prompt intervention in emergency situations. While the current study focuses on the classification of general activities, future developments could delve into the analysis of specific high-risk actions,
Deep Learning Based Human Activity Recognition in First Responders Wearing 41
such as falling, to enhance the precision and applicability of the developed system in real scenarios. Additionally, exploring how changes in physiolog- ical parameters during the execution of these actions may indicate stress or fatigue levels represents a promising avenue for further investigation. These nuanced insights could contribute to a more comprehensive understanding of the dynamics involved in high-risk scenarios, providing valuable. informa- tion for the refinement and expansion of the proposed intelligent monitoring system.
ACKNOWLEDGMENT
The authors express their gratitude to Smartex s.r.l for their collaboration and provision of materials. The authors are also grateful to Gianluca de Toma for his noticeable technical support throughout the course of this work.
REFERENCES
A. Bonfiglio, D. Curone et al., (2011). “Emergency and work”. Wearable Monitoring Systems, pp. 205–219. DOI: 10.1007/978-1-4419-7384-9_10. A. J. Paydarfar, A. Prado et al. (2020) “Human Activity Recognition Using
Recurrent Neural Network Classifiers on Raw Signals from Insole Piezore- sistors”, th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), New York, NY, USA, pp. 916-921, doi: 10.1109/BioRob49111.2020.9224311. A. Murad, J. Y. Pyun. (2017) “Deep Recurrent Neural Networks for Human Activity Recognition” Sensors (Basel). Nov 6;17(11):2556. doi: 10.3390/s17112556. PMID: 29113103; PMCID: PMC5712979. A. A. Smith, R. Li et al. (2023) “Reshaping healthcare with wearable biosensors” ci
Rep 13, 4998. https://doi.org/10.1038/s41598-022-26951-z. D. Curone, A. Tognetti et al., (2010), “Heart Rate and Accelerometer Data Fusion for Activity Assessment of Rescuers During Emergency Interventions”, IEEE Trans Inf Technol Biomed. 2010 May;14(3): 702–10. doi: 10.1109/TITB.2010.2047727. Epub 2010 Apr 8. PMID: 20378475. D. Curone, E. Secco et al., (2010), “Smart garments for emergency operators: The
ProeTEX project”, IEEE Transactions on Information Technology in Biomedicine, 14 (3), art. no. 5443746, pp. 694–701. doi: 10.1109/TITB.2010.2045003. D. Curone, E. Secco et al. (2012) “Assessment of sensing firefighters uniforms for physiological parameter measurement in harsh environment”. IEEE Trans- actions on Information Technology in Biomedicine, 16 (3), art. no. 6123206, pp. 501–511. doi: 10.1109/TITB.2011.2182615. D. Curone, G. Dudnik et al. (2007) “Smart garments for safety improvement of
emergency/disaster operators,” in Proc. IEEE Eng. Med. Biol. Soc. Conf., 2007, pp. 3962–3965. G. Magenes, D. Curone et al., (2010) “Fire Fighters and rescuers monitoring through wearable sensors: The ProeTEX project”. Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC’10, art. no. 5627452, pp. 3594–3597. doi: 10.1109/IEMBS.2010.5627452. G. Magenes, D. Curone et al., (2011) “Biosensing and environmental sensing for
emergency and protection e-Textiles” Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, art. no. 6092063, pp. 8365–8366. doi: 10.1109/IEMBS.2011.6092063.
J. Pan, W. J. Tompkins (1985) “A Real-Time QRS Detection Algorithm” IEEE Transactions on Biomedical Engineering. BME-32 (3): 230–236. doi: 10.1109/TBME.1985.325532. PMID 3997178. K. Cho, B. van Merrienboer et al. (2014) “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation”Association for Com- putational Linguistics) arXiv:1406.1078. M. Smuck, C. A. Odonkor et al. (2021) “The emerging clinical role of wearables: fac-
tors for successful implementation in healthcare”NPJ Digit Med. Mar 10;4(1):45. doi: 10.1038/s41746-021-00418-3. PMID: 33692479; PMCID: PMC7946921. O.Majidzadeh Gorjani, J.Radek Byrtus et al., (2021) “Human Activity Classification Using Multilayer Perceptron” Sensors 21, no. 18: 6207. https://doi.org/10.3390/ s21186207. P. C. Fehling, D. L. Smith et al. (1999), “Comparison of accelerometers with oxygen
consumption in older adults during exercise,” Med. Sci. Sports Exerc., vol. 31, no. 1, pp. 171–175, 1999. PROETEX project (2009). [Online]. Available: www.proetex.org. S. Brage, N. Brage et al. (2005), “Reliability and validity of the combined heart rate and movement sensor Actiheart,” Eur. J. Clin. Nutr., vol. 59, pp. 561–570, 2005. SIXTHSENSE project (2019). [Online]. Available: https://sixthsenseproject.eu SMARTEX s.r.l [Online]. Available: https://www.smartex.it/.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Cognitive Computing and Internet of Things
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-00-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional