Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Automation of the Collection and Processing of Physiological Functions from Wearable Sensors

Department of Informatics, Faculty of Natural Sciences and Informatics, Constantine the Philosopher University in Nitra Trieda Andreja Hlinku 1, 949 01 Nitra, Slovakia ↗
Publication year
Digital Object Identifier10.54941/ahfe1001473

Abstract

The current period is evidence of the massive replacement of human strength by the results of the mental cognitive work of researchers. To cope with this situation, it is necessary to examine the impact of these technologies on the quality of life in the work and home environment when introducing new technologies. The main purpose of this work will be the collection of physiological functions such as heart rate in real-time from wearable sensors by creating an application that automatically collects sensor data. The work also proposes a mobile application that will be able to read the sensor data from the database and display it. The data from the sensors will be later used in a complex system and its proposal will also be described in this paper. Such a system could be used in many industries, like the automotive industry, industrial factories, smart homes and even education.

INTRODUCTION

Wearable physiological monitoring systems collect and send physiological data to a remote monitoring station using a variety of sensors built into the wearer’s clothing. The data collected at the remote monitoring station is compared to the wearer’s general health status. Wearable monitoring systems allow users to remotely check their vital signs and receive feedback in order to maintain good health (Pandian et al., 2008). Fitness trackers can now detect and quantify biological signs such as heart rate, blood flow, and blood oxygen levels thanks to biosensing technologies. This advancement allows for applications in fitness, healthcare, and a variety of other fields. Physici- ans may now utilize consumer-grade sensors to track their patients’ health in real time, which could help with diagnosis and therapy (Veber, Pesek and Aberšek, 2021). These devices may also improve one’s chances of survi- ving unanticipated illnesses. Because they sought care after their smart watch constantly displayed abnormally high heart rate readings, individuals survi- ved rhabdomyolysis, a disorder characterized by intense physical exercise in extreme heat (Block, 2015). Health-tracking features have the potential to expand the health-care ecosystem. Human factors design area, which includes

© 2022. Published by AHFE Open Access. All rights reserved. 45

46 Fodor and Balogh

interaction between human and small interfaces, cognitive model, contex- tual awareness, learnability, and adaptation, is still untapped (Siewiorek, Smailagic and Starner, 2008; Namestovski et al., 2020).

RELATED WORK

The challenge of assessing human emotions is an intriguing one that has got- ten a lot of attention in recent years. It’s important to note the specific link between numerous seemingly unrelated fields and areas, such as computer science and psychology. As a result, it offers a lot of potential for enhancing the connection between humans and computers in a variety of areas, inclu- ding education, medicine, and business (Magdin, Turcani and Hudec, 2016). Researchers are interested in physiological and emotional monitoring, par- ticularly in the fields of stress and anxiety monitoring (Goodfellow et al., 2015). Various strategies were utilized in some of the studies in order to elicit emotional responses. For example, utilizing movies in studies based on the assumption that certain movie genres elicit various feelings in the individuals (Li and Chen, 2006). For instance, we can suppose that horror films elicit fee- lings of fear or disgust in the viewer, but comedies elicit feelings of happiness. This experiment can also be carried out in a virtual reality setting, which can elicit more emotional responses (Magdin et al., 2021). All of these methods can be utilized to learn more about emotion recognition (Minaˇrík and Sta- stny, 2008) based on physiological functions, and the findings can be applied to create emotionally intelligent machines in the future (Picard, Vyzas and Healey, 2001) and can be combined with facial recognition (Katona et al., 2020). Authors Otsuka & Ohya (Otsuka and Ohya, 1997) used image and video processing to analyze facial emotions. They attempted to classify nume- rous facial emotions into individual groups by analyzing facial functions and quantifying the volume of movement on the face. In this paper, we’ll look at devices that allow us to collect data on physiological functions, because we can also classify emotional states using physiological functions (Kim and André, 2008). There are devices that can not only measure and display the needed functions, but can also transfer data to other devices or communicate with them. These are fitness trackers or other “things” that may be worn for an extended period of time without interfering with the user’s work. Intero- perability is a feature of these gadgets, which means they can work together and communicate with devices of different types and are all connected by a single computer. One device, for example, can detect heart rate, while ano- ther can measure body temperature, and a central computer can analyze and correlate the data. Fitness trackers (wristbands) were used to assess heart rate in sports by authors Cvetkovi´c et al. (Cvetkovi´c et al., 2018), de Zambotti et al. (de Zambotti et al., 2016), Kwak et al. (Kwak et al., 2017), and Yong et al. (Yong et al., 2018). Athletes were given fitness trackers to wear and were given tasks to complete, such as jogging or other activities. The first sensor was detecting heart rate and transferring the information further. The data was transferred for processing after the wristband was connected to the central computer. The data from the wristband was received by the system

Automation of the Collection and Processing of Physiological Functions 47

server, which processed it into a predetermined format. It also gives the termi- nal access to the data that has been processed. Authors Rosenblum, Yacoob and Davis (Rosenblum, Yacoob and Davis, 1996); Yacoob & Davis (Yacoob and Davis, 1996) analyzed facial expressions using image and video proces- sing. By observing facial functions and measuring the volume of movement on the face, they tried to classify various facial expressions into individual categories.

MATERIALS AND METHODS

The main idea is to use common, non-invasive IoT (Internet of Things) devices for automatic emotion recognition in different situations. For this purpose, wearable technologies can be used, such as smart watches and fitness trackers that are most commonly equipped with different sensors and are able to measure heart rate among other physiological functions, such as galvanic skin response and body temperature. The problem with these devi- ces is however, the availability of the sensor data. One of the most popular fitness trackers are the Mi Bands by Xiaomi. After linking the band to the Mi Fit app developed by Huami technologies, all measured values are log- ged. Bluetooth technology is used to secure pairing with the application. The bluetooth 5.0 low-energy chipset allows for a fast and reliable connection while also reducing battery use. This technology also ensures that the bat- tery lasts a long time and that data is transmitted reliably. A high-density lithium-polymer battery provides twenty days of resilience and longevity. It is possible to export heart rate data from a specific interval in the Mi Fit application, however it is a lengthy process. The users must first log in and select the data that they want to export. The data is then emailed as a CSV file to the user’s email address. The usage of a third-party application called Tools & Mi Band, which also allows the user to export the heart rate data in a CSV file without having to log in and wait for the file to arrive in the email, is a speedier option for getting heart rate data. The user can also send the CSV file to others via messaging apps or save it to their personal cloud storage (Landa, Procházka and Stastny, 2013), such as Google Drive.

We designed a simple application (Fodor and Balogh, 2021) that can open a CSV file (e.g., from Google Drive), cycle over each row, and construct JSON objects to store the observed heart rate more effectively. These objects are then combined to form a JSON array, which is then delivered to a server. The server’s software then decodes the JSON array and inserts new records into a database table using SQL.

We can partially automate the data storage process in a database by using this approach, and we can also store data from many sensors from different sources in the same database. This is particularly useful for synchronizing data from multiple sources for the same user.

Not only can the application submit data, but it can also read the contents of a database table and display it in a ListView container. If we have more data from different sources (e.g., Arduino or Raspberry Pi), we can use this application to read the values in the database from all of the sources with their respective sensors in the future.

48 Fodor and Balogh

This method wasn’t enough for our needs as we want to create a fully automatic application that can transfer the sensor data from a smart watch or fitness tracker to a specific server-side script, where the data is processed and stored for later use in an automatic emotion recognition software. For this reason, we have examined the wearable products made by Fitbit. Fitbit is a fitness and consumer electronics firm based in the United States. It makes wireless fitness monitors and activity trackers, such as smartwatches, pedo- meters, and monitors for heart rate, sleep quality, as well as related software. Google purchased the company in 2021. The company offers an open sof- tware development kit (SDK), where developers can use JavaScript, CSS and SVG to develop applications and clock faces for Fitbit OS.

RESEARCH METHODOLOGY We have used the Fitbit Studio to create the application that can automati- cally send obtained sensor data to a server to process the data and store it in database tables. The application was tested on the Fitbit Sense product that contains a heart rate sensor. The main fragment of the JavaScript code running on the wearable can be seen below.

if (HeartRateSensor) { const hrm = new HeartRateSensor({ frequency: 1 }); hrm.addEventListener("reading", () => { hrtext.text = ‘${hrm.heartRate}‘; if (messaging.peerSocket.readyState === messaging.peer

Socket.OPEN) { messaging.peerSocket.send(‘${hrm.heartRate}‘); } }); hrm.start(); } If a heart rate sensor is available, a new constant is created for the heart rate sensor. The frequency parameter defines how frequently we want to read the value from the sensor. In the example above, the frequency is set to 1, which means the value from the sensor will be read every second. An event listener is then added to the sensor, so a code can be executed every time there is a new reading from the sensor. Unfortunately, the wearable can’t communicate with a server so the sensor data has to be sent to the compa- nion first, which is the mobile application communicating with the wearable. In order to do so, Messaging API was used. The application and compa- nion are commonly referred to as “peers”. Before any messages can be sent, each peer must successfully open a MessageSocket connection. Connections are made automatically, and the open event is fired when a connection is made. Peers can send and receive simple messages once they’ve established a connection. The device peer, for example, may ask the companion peer to seek data from a web service, and the companion could then relay the data back to the device. In this case, the sensor value is sent through the open socket.

Automation of the Collection and Processing of Physiological Functions 49

The companion code can be seen below: messaging.peerSocket.addEventListener("message",

(evt) => { console.error(JSON.stringify(evt.data)); var currentDate = new Date().toLocaleString(); var data = {rate: evt.data, date_time: currentDate}; fetch(’https://mywebsite/myphpscript.php’, { method: ’POST’, body: JSON.stringify(data), }).then(res => res.json()) .then(response => console.log(’Success:’,

JSON.stringify(response))) .catch(error => console.error(’Error:’, error)); }); An event listener is added on the companion side to listen to messages sent by the wearable. A JSON object is then created with key – value pairs, where the sensor value and the current timestamp is added. The FETCH API was used to send the data to a PHP script on a server. The response from the server is then logged in the console, however text containers can also be used to display the information to the user. The main part of the code on the server can be seen below:

$content = file_get_contents(’php://input’); $data_array = json_decode($content, false); echo json_encode($data_array); $result = ""; $dt = $data_array->date_time; $rate = $data_array->rate;

$sql = "INSERT INTO heart_rate (date_time, rate) VALUES (’".$dt."’, ’".$rate."’)"; if(mysqli_query($conn,$sql)) { $result = "Data sucessfully uploaded"; } else { $result = "ERROR ". $sql; } echo json_encode($result); The raw data from the request is taken and converted to a PHP array. The values are extracted from the array by using the keys that were defined in the client. Using SQL, the data is inserted to the database table and a response is sent back to the client. The data in the database can be seen below (Figure 1).

As it can be seen from the timestamps, we have a reading from the heart rate sensor from every second, so in the context of emotion recognition we can consider this real-time monitoring. Another advantage of the created application is that it also works if the screen of the wearable is off and by default it detects if the wearable is off wrist, therefore the event calling the

50 Fodor and Balogh

Figure 1: The sensor data in the database (own data).
Original PDF Figure 1:

code that adds new records to the database table is not executed if there are no new heart rate readings.

CONCLUSION AND FUTURE WORK Automatic gathering of physiological functions by using smart watches, fitness trackers and other microcomputers with sensors can be used in com- bination with facial recognition software to recognize the emotional states of the users. The emotion classified by the facial recognition software will be used as a reference value, which means that if the facial recognition softw- are outputs “happy”, then the user’s measured physiological functions (heart rate, body temperature, galvanic skin response) can be analyzed and com- pared to other users whose facial recognition software outputted “happy” to find connections and correlations. A supervised learning process can be implemented using machine learning algorithms to map the input features (physiological functions) to the outputs (7 different emotional states based on Ekman’s classification) based on example input-output pairs after having a complete dataset with measured physiological functions of different users where the target emotional states occur several times. The training data (the dataset of physiological functions and the emotional state classified by the facial recognition program) will then be analyzed by the supervised lear- ning algorithm, which will provide an inferred function that can be used to identify the users’ emotional states. In the best-case scenario, the algorithm will be able to accurately assess the emotional state of individuals simply by measuring their physiological data, without the usage of facial recognition software. A user interface can then be created by combining this algorithm with a mobile or web application. When studying the created dataset, this strategy can also assist us in gaining interdisciplinary knowledge:

  • We could determine how much the measured physiological data reduced or increased for each user when they were experiencing specific emotions.
  • We could determine how long the participants were in a neutral state and how long each emotion lasted.

ACKNOWLEDGEMENT

This research has been supported by the projects KEGA 036UKF-4/2019, Adaptation of the learning process using sensor networks and the Internet of Things, APVV-18-0473 Machine translation error classification model: a

Automation of the Collection and Processing of Physiological Functions 51

step towards a more objective assessment of translation quality and the civic association OZ DIVAI.

REFERENCES

Block, J. (2015) How An Apple Watch Helped Save A High School Football Player’s Life. Available at: https://www.huffpost.com/entry/paul-houle-apple-watch_ n_5601878de4b08820d91a4688 (Accessed: 20 December 2021). Cvetkovi´c, B. et al. (2018) ‘Real-time activity monitoring with a wristband and a

smartphone’, Information Fusion, 43, pp. 77–93. doi: https://doi.org/10.1016/j. inffus.2017.05.004. de Zambotti, M. et al. (2016) ‘Measures of sleep and cardiac functioning during sleep using a multi-sensory commercially-available wristband in adolescents.’, Physiology & behavior, 158, pp. 143–149. doi: 10.1016/j.physbeh.2016.03.006. Fodor, K. and Balogh, Z. (2021) ‘Sensory Monitoring of Physiological Functions

Using IoT Based on a Model in Petri Nets BT - Web Information Systems Engi- neering – WISE 2021’, in Zhang, W. et al. (eds). Cham: Springer International Publishing, pp. 435–443. Goodfellow, I. J. et al. (2015) ‘Challenges in representation learning: a report on three machine learning contests.’, Neural networks : the official journal of the Internati- onal Neural Network Society, 64, pp. 59–63. doi: 10.1016/j.neunet.2014.09.005. Katona, J. et al. (2020) ‘Using Eye-Tracking to Examine Query Syntax and Method

Syntax Comprehension in LINQ’, in 11th IEEE International Conference on Cognitive Infocommunications (CogInfoCom). Electr Network, pp. 437–443. Kim, J. and André, E. (2008) ‘Emotion recognition based on physiological changes in music listening’, IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(12), pp. 2067–2083. doi: 10.1109/TPAMI.2008.26. Kwak, Y. H. et al. (2017) ‘Flexible heartbeat sensor for wearable device’, Biosensors

& bioelectronics, 94, pp. 250–255. doi: 10.1016/j.bios.2017.03.016. Landa, J., Procházka, D. and Stastny, J. (2013) ‘Point cloud processing for smart systems’, Acta Universitatis Agriculturae et Silviculturae Mendelianae Brunensis, 61, pp. 2415–2421. doi: 10.11118/actaun201361072415. Li, L. and Chen, J. (2006) ‘Emotion Recognition Using Physiological Signals’, in

Lecture Notes in Computer Science, pp. 437–446. doi: Li, L., & Chen, J. (2006). Emotion Recognition Using Physiological Signals. Lecture Notes in Computer Science, 437–446. doi:10.1007/11941354_44. Magdin, M. et al. (2021) ‘Automatic detection and classification of emotional states in virtual reality and standard environments (LCD): comparing valence and arousal of induced emotions’, Virtual Reality. doi: 10.1007/s10055-021-00506-5. Magdin, M., Turcani, M. and Hudec, L. (2016) ‘Evaluating the Emotional State

of a User Using a Webcam’, International Journal of Interactive Multimedia and Artificial Inteligence, 4, pp. 61–68. doi: 10.9781/ijimai.2016.4112. MINA ˇRÍK and Stastny, J. (2008) ‘Recognition of Randomly Deformed Objects’. Namestovski, Ž. et al. (2020) ‘The Correlations between Health Behaviors and the Use of Health-Preserving Mobile Applications in Young Adults’, Acta Polytech- nica Hungarica, 17, pp. 115–128. doi: 10.12700/APH.17.2.2020.2.7. Otsuka, T. and Ohya, J. (1997) ‘Recognizing multiple persons’ facial expressions

using HMM based on automatic extraction of significant frames from image sequences’, Proceedings of International Conference on Image Processing, 2, pp. 546–549 vol.2. Pandian, P. et al. (2008) ‘Wireless Sensor Network for Wearable Physiological Monitoring’, JNW, 3, pp. 21–29. doi: 10.4304/jnw.3.5. 21–29.

52 Fodor and Balogh

Picard, R. W., Vyzas, E. and Healey, J. (2001) ‘Toward machine emotional intelligence: analysis of affective physiological state’, IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 23(10), pp. 1175–1191. doi: 10.1109/34.954607. Rosenblum, M., Yacoob, Y. and Davis, L. S. (1996) ‘Human expression recogni-

tion from motion using a radial basis function network architecture.’, IEEE transactions on neural networks, 7(5), pp. 1121–1138. doi: 10.1109/72.536309. Siewiorek, D., Smailagic, A. and Starner, T. (2008) Application Design for Wearable Computing, Synthesis Lectures on Mobile and Pervasive Computing. doi: 10.2200/S00106ED1V01Y200801MPC003. Veber, M., Pesek, I. and Aberšek, B. (2021) ‘Using EEG Device in the Pilot Develo-

pment of Cyber-Physical Learning Model’, in 2021 44th International Conven- tion on Information, Communication and Electronic Technology (MIPRO), pp. 1571–1575. doi: 10.23919/MIPRO52101.2021.9596859. Yacoob, Y. and Davis, L. (1996) ‘Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow.’, IEEE Trans. Pattern Anal. Mach. Intell., 18, pp. 636–642. doi: 10.1109/34.506414. Yong, B. et al. (2018) ‘IoT-Based Intelligent Fitness System’, J. Parallel Distrib.

Comput., 118(P1), pp. 14–21. doi: 10.1016/j.jpdc.2017.05.006.

Cover of the proceedings volume Human Factors and Wearable Technologies

Explore the complete digital volume

Human Factors and Wearable Technologies

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-05-6

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →