Iraklis Varlamis, Michail Maniadakis, Georgios Athanassiou (2022). A Micro-moment recommendation framework in industrial environments. In: Human Factors and Systems Interaction. AHFE Open Access, vol. 52, pp. 96–104. https://doi.org/10.54941/ahfe1002144
Today, a large part of the labor policies in the EU aim at extending the active participation of older (i.e. 50+) employees in the workforce in order to avoid the respective pressure on the national economies and health systems as well as potential shortcomings in qualified personnel due to demographical changes in the entire population. Preventing involuntary early retirement goes hand in hand with supporting self-sufficient and healthy living. The present work considers the use and exploitation of modern technological advancements to support the achievement of the above goal. Specifically, we propose a new approach to developing complex recommendation systems, which are capable of monitoring and supporting the daily activities of employees in a personalized manner, both at work and during their broader daily activities. The proposed approach is based on the new Micro-Moments (MiMos) concept for critical event recognition, incorporating multiple streams of complementary information from a distributed sensor network that is flowing into the system based on IoT technologies. The recommendation system follows a user-centered approach for providing (personalized) suggestions that support the occupational safety of users, improve their health and enhance their productivity, in a personalized way. This paper summarizes the concept of Micro-Moments (MiMos) and how it contributes to issuing recommendations based on specific user needs. We also present the current version and implementation of the system in the field of port logistics, where it is observed that recommendations delivered at the right time to the right person can help improve the efficiency of the workforce and extend its working capacity.
The aging process is associated with individual changes in physical, cognitive and psychosocial properties which can in turn affect job performance, safety, and health and overall well-being (Hartshorne and Germine, 2015; Hedge and Borman, 2018; Salthouse, 2021). The degree of gravity of negative effects of aging on older workers depends on one hand on individual characteristics and life conditions; and on the other hand, on the balance between the specific
A Micro-Moment Recommendation Framework in Industrial Environments 97
job-related demands and the available resources (Crawford et al, 2016). This is especially important in jobs with unfavorable characteristics, such as high physical demands and prolonged muscular strain, low complexity, increased attentional requirements in combination with repetitive tasks and reduced autonomy (McGonagle et al, 2015). In such environments, job requirements cannot be fully compensated by experience and extensive know-how of the older workforce but rather necessitate structured interventions both on the individual as well as on the (work-) organization level.
Technological, non-intrusive solutions for a holistic and personalized monitoring of safety, health and well-being in work environments and beyond can be of great assistance for the purposes of supporting OSH management and well-being, promoting sustainable work, healthy aging and good perfor- mance for the aging workforce. Data-based, context-aware and personalized decision-support systems that make use of commercially available wearable and portable devices are examples of such systems.
RECOMMENDER SYSTEMS FOR THE SUPPORT OF OSH AND WELL-BEING OF EMPLOYEES Recommender systems (RSs) are a subclass of information filtering systems that seek to predict the user interest (rating, preference) for considered items in order to prioritize these items for each individual (Ricci et al, 2011). Up to now, a small number of works have considered the use of recom- mender systems (RSs) to support aspects of occupational performance. For example, an RS has been proposed to handle occurring work events at each moment and recommend relevant documents to employees working online (Damiani et al, 2015). Another RS that considers user preferences and perso- nality types before recommending the appropriate shared computer resources through a smart mobile device has been proposed (Park, 2019).
The group of recommender systems that promote healthy lifestyle and well-being, are also relevant. Such RSs recommend: personalized diets and drug treatments to diabetic patients (Ali et al, 2018), nutrition plans that take into account user food preferences, food ingredients and dietary restrictions (Espín et al, 2016), and treatment guidelines to patient carers (Afolabi and Toivanen, 2018). In the lifestyle domain, fitness assistants are used that com- bine recommendations for training activities and friends with similar training profiles (Dharia et al, 2018).
The current work presents a novel application of recommender systems that specifically targets the prevention of OSH risks and the mitigation of negative effects stemming from job demands on older workers as well as the strengthening of personal resources in order to enhance the health and well-being of the employees. Among the various contextual dimensions (e.g. spatial, temporal, state or mood, weather, etc.) of recommendations, our work focuses on the notion of time that can critically affect the effectivity (in terms of relevance, validity and credibility) of recommendations since it is directly linked to the evolution of user preferences, habits and interests (Campos et al, 2014) and can assist in improving the recommendation acce- ptance (Dali Betzalel et al, 2015). In particular, the proposed system exploits
the concept of micro-moments (Enache and Morozan, 2017) for identifying the right (i.e. appropriate and situation-relevant) moments for sending an actionable recommendation and maximizing its validity and probability to be accepted and introduces it in the domain of well-being and health in the work environment.
The implemented system uses data from smart devices and sensors to moni- tor users’ behavior, psycho-physiological states and cognitive performance on a daily basis.
By considering historical data, the system utilizes a personalized point of reference for each employee linked with aspects of overall well-being and OSH performance. Critical health-related indicators such as cardiovascular activity, workload, body postures during work, sleep duration and cognitive and physical activity are regularly monitored and evaluated against standards and user-specifc benchmarks in order to support maintaining the balance between work demands and physiological/cognitive worker resources. Based on this logic, personalized OSH recommendations are issued in order to support to the mitigation of potential risks, ensure prolonged work ability, promote adaptive capacities and create the necessary conditions for healthy and sustainable work for older employees in the working context.
RECOMMENDATIONS & MICRO-MOMENTS The integration of the Micro-Moments (MiMos) concept in recommender systems is among the novelties of the proposed system. Information-rich MiMos guide the triggering of recommendations for actions that correspond tightly with the specific needs and preferences of users in a given time and situ- ation. Interestingly, most existing systems do not consider the role of timing as a contextual factor when generating recommendations. Along this line, we have identified critical time-related challenges for RSs, which correspond with the following aspects:
Type of recommendations: Recommendations are prioritized depending on their criticality and are grouped, based on the degree of urgency for subsequent user action and response to the system, into alerts, suggesti- ons and notifications at a single-user or group level. Alerts have the highest priority, being delivered as soon as an eminent safety or health risk is iden- tified (e.g. entering the hazardous zone of a moving container, or having a heart rate that exceeds expected limits) and require immediate action and its confirmation by the user. Moreover, alerts addressing the state of wor- kers as a collective (e.g. elevated fatigue indicators for 80% of the workers in a shift) are considered, addressing the responsible supervising manager for countermeasures. Suggestions comprise activities that can help users in improving their physical fitness or mental skills and require user feedback, either explicit or implicit, to validate (non-)acceptance of the proposed action. Finally, notifications consider how the individual activities of the users comply with general rules of wellness or good practice (e.g. sufficient sleep time) and contain informative statements and/or reminders.
A Micro-Moment Recommendation Framework in Industrial Environments 99
Figure 1: Block diagram illustrating the composite system and its components.
User work schedule: Match recommendations to the work schedule, taking into account the shift position, working hours and shift-breaks, and by visually monitoring the work-related activities of workers in real time(e.g. recommend muscle stretching when workers have free time).
Aspect-related reminders for action later in the day: Users are notified about future actions they can perform to improve their health and well- ness. Timely reminders are sent when the action has to be implemented (e.g. at wake up, if the user is detected having a sleep debt (i.e. a deficit in hours from the “normal”duration) from the previous night, he/she is noti- fied for scheduling an early sleep for the following night and a reminder is set for the night).
Inference for appropriate timing for the delivery of recommendations: This refers mostly to off-work recommendations. It is implemented by monitoring the users’ state, to identify the context that better matches a given recommendation (e.g. users are recommended to perform an afternoon cognitive training session when their heart rate is low and smar- twatch accelerometer exhibits decreased variance, which may indicate they are not involved in other activities). Determining the right context renders the delivery of recommendations timely appropriate and improves the rate of acceptance by users.
SYSTEM OVERVIEW The architecture of the implemented system (see Figure 1), broadly comprises i) a multitude of sensors for collecting user and environment data, ii) a set of software components that perceive user behavior, iii) a streaming layer for collecting behavioral traits and conditions, iv) a personalization layer that examines the condition and context of the worker, the history of worker’s
activities, as well as the recent physical and cognitive performance, to trigger the relevant MiMos, v) a recommendation layer which generates and issues recommendations to a single worker, a group of workers and/or the foreman worker depending on the situation, vi) the interaction layer which includes a set of cognitive games via a dedicated app, the visualization dashboard and the dialog manager for providing the recommendation to the user via diffe- rent modalities (speech, text or vibration). All data is streamed in real-time through a Universal Messaging bus, which provides the brokering service for modules communication and interaction.
THE SYSTEM PIPELINE: FROM MIMOS TO RECOMMENDATIONS The system collects information from multiple and complementary low-cost sensors to support recommendations. The sensors unobtrusively monitor users, capture and process real-time information to robustly detect poten- tial user risks. The sensory modalities that are currently integrated in our system are summarized below:
Wellness Sensor. The Garmin Vivoactive-3 smartwatch collects heart- rate, beat-to-beat intervals, the number of steps, and 3D accelerometer information.
Environmental Sensors. A sensor box that hosts multiple third-party sen- sors has been developed, in order to capture environmental data, including ambient temperature, relative humidity, barometric pressure, wind speed, air quality, illuminance and noise level.
Positioning Sensors. Positioning information is transmitted through the smartphone multi-GNSS chipset by simultaneously tracking the signals bro- adcasted from all operational satellites (i.e. GPS, GLONASS, Galileo and BeiDou), providing increased robustness, reliability and spatial coverage.
Vision. Visual information is captured through a monocular camera instal- led on a mobile harbor crane. This camera shares a common reference frame with the crane, determined using a set of reference ground control points surveyed with topographic equipment (see for more information: Lourakis et al, 2020). Visual input is used to monitor moving containers in correlation with workers location and trigger proximity alerts.
Speech. The human voice encapsulates linguistic and paralinguistic infor- mation, which relates to a speaker’s current states, traits and wellbeing. Workers utterances are captured through the microphone embedded in the Xiaomi Mi 9 smartphone of each worker. The smartphone is the main interaction point with the users.
All sensors are deployed in the workplace while the wristwatch and the smartphone are also used outside the workplace, after work, providing in real-time a loose state of users physiological and mental state over the day.
OSH INTERVENTIONS The system has been piloted in two industrial domains with different chara- cteristics and OSH challenges: manufacturing and logistics port operations. A thorough research of the specific risks and challenges for older workers
A Micro-Moment Recommendation Framework in Industrial Environments 101
in the two domains was conducted within the sustAGE project prior of the design of interventions and the respective recommendations1,2. This paper focuses on the case of port/container handling operations in a medium-sized European commercial port. The work process in the sustAGE use case of normal cargo operations entails three occupational roles with distinct tasks and responsibilities: a crane operator (CO) and a team of dock workers (DW) with a foreman (FW). COs work in the crane’s cabin to move containers from and to moored merchant vessels. DWs work on the quay and on the vessel’s deck lashing/unlashing containers in accord with the crane. The FW is addi- tionally responsible for assigning DW to task, supervising the work process and coordinating all activities of DWs during operations. Can also interrupt cargo operations for safety reasons, and grant short breaks “on-demand”. In the case of off-work recommendations, the main challenges of the system are the limited monitoring of users with smartwatch data only and, at the same time, the significantly increased alternatives for potential user activities that may impede the matching of system recommendations with the behavior of the users at a given time.
INTERVENTIONS AT WORK In the aforementioned operations scenario, the system is capable of dete- cting related MiMos that are linked to OSH-related recommendations as summarized below.
Worker fatigue: This is associated with high fatigue events of a single wor- ker or a group of workers. The smartwatch is used to continuously collect heart rate (HR) and heart rate variability (HRV) data from each worker as indicators of fatigue events. The user’s cardiovascular data during the last hour is forwarded to a Long-Short Term Memory neural network classifier that is trained to infer the user’s current state of fatigue and initiate recom- mendations for recovery actions (breaks and adaptation of sleep hours). In case of high fatigue, the respective MiMo is issued and the system i) examines the worker’s context (at work or not), ii) examines the average fatigue state of the team, iii) gets the worker’s sleep hours the previous day. Depending on the case, the system either associates the fatigue with a sleep dept in the previous night and notifies the worker to adjust his/her sleep schedule, or if the sleep dept has been accumulated it recommends the worker to additionally take a micro break. In the latter case, the foreman will be notified, as he/she will grant the micro-break and reorganize work in the team. When fatigue events occur for multiple workers, a recommendation is sent to the foreman to slow down operations (after notifying the CO) or grant an unplanned short break.
Worker safety/Accident prevention: Taking advantage of the real-time users’ location monitoring and containers’ tracking, the distance between the two is continuously estimated. If this distance is below a safety threshold, the system immediately alerts users to move away from the hazardous zone eliminate the risk of an accident.
1sustAGE Deliverable-2.2: Risks for OSH and well-being, and recommendation framework 2sustAGE Deliverable-2.4: The sustAGE user-centered framework
Adverse environmental conditions: Dock-workers operate in open space so the system monitors the environment to identify conditions that may cause discomfort, or increase accident risks. By blending information from the tem- perature, humidity and wind speed sensors, and computing the heat and chill index (i.e. the “Feels-like” condition) the system can detect situations that require intervention. In such cases, the list of workers and their supervisors are notified and short breaks for resting, or hydration, are recommended. In the case of extremely high wind at the port with frequent high heart-rate inci- dents observed in the team of workers, crane operators are notified to seize operations in order to minimize accident risk.
Interventions Outside Work In addition to the recommendations within the workplace, the system supports the physical and cognitive skills of workers outside the work environment that target:
Sleep debt: Sleep duration drastically affects workers’ performance. The system estimates the hours of sleeps every night and when they are below a threshold, informs the user and suggests a compensating adaptation of the sleep duration for the subsequent night(s). In case of a recurring pat- tern of short sleep duration and subsequent increase of sleep debt, the user is informed about the effects of accumulated fatigue due to poor sleep and is recommended to adjust his/her long term go-to-bed strategy.
Physical fitness: Following WHO guidelines (WHO, 2007), the system encourages users to engage in weekly physical activities to improve their aero- bic fitness, always considering the context of action. Given that port workers in the given port do not have a constant work schedule, the system consi- ders their location and only when they are at home, they are recommended to have a 1h session of physical activity (waking/light jogging/cycling). The acceptance of the recommendation is evaluated with a relevant question to the users three hours later and the response is used the following day by the system to decide whether a new recommendation for physical activity should be provided or not.
Cognitive skills: Gamified cognitive training in the form of carefully desi- gned Cognitive Games (CGs) supports the maintaining cognitive skills that are related to work performance and general health and well-being and have been found to decline with age (e.g. memory, executive functions, proces- sing speed) (Ballesteros et al., 2014; Strobach & Karbach, 2021). Serious Six different CGs are used to provide port workers with the option to exercise their cognitive skills. The system recommends the users to participate in a cognitive game challenge three times a week, but the users are able to play at any time. CG recommendations arrive in the afternoon when workers are at home. The recommendation acceptance is verified by the CG backend that logs the hours played and the scores achieved by each user.
CONCLUSION
This paper introduces a new approach to creating a personalized recommen- dation system that supports employees’ safety, health and well-being for and
A Micro-Moment Recommendation Framework in Industrial Environments 103
outside the work environment. A rich IoT ecosystem allows to unobtrusively receive and process psychophysiological characteristics of employees that are related to their work obligations and broader activities during the day. Utilizing the information collected by the low-cost sensors, several dimen- sions of occupational safety, health and well-being can be monitored in real-time, potential risks are detected and personalized recommendations are provided that mitigate negative job-related effects and user states and help senior employees to maintain or even strengthen their physical and cogni- tive resources and maintain adequate job performance until their retirement. The results from the implementation of the first functional system in the transportation work environment with real users indicated that the persona- lized recommendations helped the port workers to gradually improve their cognitive skills and effectively regulate aspects of their overall physical con- dition related to the interplay performance and well-being3 (sleep schedule adjustment, better rest after fatigue, improved body posture in work tasks (Papoutsakis et al., 2021)). Future work will entail the full-scale implementa- tion of the system in outdoor and indoor work environments and will focus on OSH recommendations on the collective (i.e. work group) level, on asses- sing aspects of mental workload, on the preemptive support of best-practice activities in order to strengthen the framework with more recommendations and personalization. It will also thoroughly examine the factors that influ- ence employees’ decisions to follow recommendations and adopt healthier habits as indicators for system effectivity.
ACKNOWLEDGMENT
This work has been supported by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 826506 (sustAGE).
REFERENCES
Afolabi, A.O. and Toivanen, P., 2018. Recommender systems in healthcare: Towards practical implementation of real-time recommendations to meet the needs of modern caregiving. In Handbook of Research on Emerging Perspectives on Healthcare Information Systems and Informatics (pp. 323-346). IGI Global. Ali, F., Islam, S.R., Kwak, D., Khan, P., Ullah, N., Yoo, S.J. and Kwak, K.S., 2018.
Type-2 fuzzy ontology–aided recommendation systems for IoT–based healthcare. Computer Communications, 119, pp. 138–155. Ballesteros, S., Prieto, A., Mayas, J., Toril, P., Pita, C., Ponce de LeÃ3n, L., Reales, J. M., & Waterworth, J. (2014). Brain training with non-action video games enhances aspects of cognition in older adults: A randomized control- led trial. Frontiers in Aging Neuroscience, 6. https://doi.org/10.3389/fnagi.2014. 00277 Campos, P.G., Díez, F. and Cantador, I., 2014. Time-aware recommender systems:
a comprehensive survey and analysis of existing evaluation protocols. User Modeling and User-Adapted Interaction, 24(1), pp. 67–119.
3sustAGE Deliverable-6.3: First integrated prototype user evaluation and impact analysis
Crawford, J., Davis, A., Cowie, H., Dixon, K., Graveling, R., Belin, A., Dupont, C,.
2016. The ageing workforce: implications for occupational safety and health: a research review. 10.2802/646283. Dali Betzalel, N., Shapira, B. and Rokach, L., 2015, September. “ Please, Not Now!” A Model for Timing Recommendations. In Proceedings of the 9th ACM Conference on Recommender Systems (pp. 297–300). Damiani, E., Ceravolo, P., Frati, F., Bellandi, V., Maier, R., Seeber, I. and Waldhart, G.,
2015. Applying recommender systems in collaboration environments. Computers in Human Behavior, 51, pp. 1124–1133. Dharia, S., Eirinaki, M., Jain, V., Patel, J., Varlamis, I., Vora, J. and Yamauchi, R.,
2018. Social recommendations for personalized fitness assistance. Personal and Ubiquitous Computing, 22(2), pp. 245–257. Enache, E. and Morozan C., 2017. Micro moments marketing-tendencies in the brands marketing. Management Strategies Journal, 35(1), pp. 304–310. Espín, V., Hurtado, M.V. and Noguera, M., 2016. Nutrition for Elder Care: a
nutritional semantic recommender system for the elderly. Expert Systems, 33(2), pp. 201–210. Hartshorne, J.K. and Germine, L.T., 2015. When does cognitive functioning peak?
The asynchronous rise and fall of different cognitive abilities across the life span. Psychological science, 26(4), pp. 433–443. Hedge, J.W. and Borman, W.C., 2018. Employee age and performance in organizations. In Aging and Work in the 21st Century (pp. 123–145). Routledge. Lourakis, M., Pateraki, M., Karolos, I.A., Pikridas, C. and Patias, P., 2020.
Pose estimation of a moving camera with low-cost, multi-GNSS devices. ISPRS- International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 43, pp. 55–62. McGonagle, A.K., Fisher, G.G., Barnes-Farrell, J.L. and Grosch, J.W., 2015. Individual and work factors related to perceived work ability and labor force outcomes. Journal of Applied Psychology, 100(2), p.376. Papoutsakis, K., Papadopoulos, T., Maniadakis, M., Lourakis, M.,Pateraki, M., &
Varlamis, I. (2021). Detection of physical strain and fatigue in industrial environ- ments using visual and non-visual sensors. In proc. 14th PErvasive Technologies Related to Assistive Environments Conference (PETRA 2021) Park, J.H., 2019. Resource recommender system based on psychological user type indicator. Journal of Ambient Intelligence and Humanized Computing, 10(1), pp. 27–39. Ricci, F., Rokach, L., & Shapira, B. (2011). Introduction to recommender systems
handbook. In Recommender systems handbook (pp. 1-35). Springer, Boston, MA. Salthouse, T.A., 2021. Individual differences in working memory and aging. Current Issues in Memory, pp. 299–318. Strobach, T., & Karbach, J. (Hrsg.). (2021). Cognitive Training: An Overview of Features and Applications. Springer International Publishing. https://doi.org/10.1007/978-3-030-39292-5 World Health Organization. (2007). A guide for population-based approaches to
increasing levels of physical activity: implementation of the WHO global strategy on diet, physical activity and health.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors and Systems Interaction
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-28-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional