Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Building Trust and safety Correlates for Autonomous Systems using Physiological, Behavioral, and Subjective Measures

1 Department of Engineering, School of Science & Technology, Nottingham Trent, University, Clifton, Nottingham, NG11 8NS, UK
2 Engineering Innovation Group, PEPSICO Europe, 4 Leycroft Road, Leicester LE4 1ET, UK ↗
Publication year
Digital Object Identifier10.54941/ahfe1001595

Abstract

The use of collaborative robots (cobots) in the industrial setting has grown and continues to expand globally, especially in the context of the smart factory. Mistrust and stress results, as cobots don’t provide facial, auditory, and visual cues that workers normally use to predict behavior. For quantification of mental stress, physiological, behavioral and subjective measures are integrated, processed and analyzed in a smart factory lab setting. The impact on the human workers as mental stress and fatigue conditions are correlated with the task complexity, speed of work, length of collaborative task and cobot payload etc. Multimodal functional neuroimaging was used to record participants’ neural and cardiac activity, in addition to the standard subjective and behavioral measures as they collaborated with robots in multitasking contexts. Preliminary results show that task complexity is positively correlated with beta and gamma band power, left prefrontal cortex activation, and heart rate, while it is negatively correlated with alpha band power during task performance.

INTRODUCTION

The use of collaborative robots (cobots) in the industrial setting has grown and continues to expand globally (Kaivo-Oja, 2020), especially in the context of the smart factory. As a result, the aspects of occupational safety and health prevention are expected to gain attention as the cobots do not work in isola- tion (Wang et al., 2019) but rather share the space with humans. Mistrust and stress results (Arai et al., 2010), as cobots don’t provide facial, auditory, and visual cues that workers normally use to predict behavior. These factors may generate anxiety and fear and can adversely influence human workers’ atti- tudes towards and acceptance of cobots and technologies (Kopp et al., 2020). The trust model for automation (French B, Duenser A, 2018) categorizes

© 2022. Published by AHFE Open Access. All rights reserved. 1

it in two types, i.e., initial trust before the actual scenario implementation and the dynamic trust that is experienced during the interaction. This resea- rch is focused on how improved mental health and safe working conditions can be developed for the optimal use of collaborative robots in the dynamic trust regime. This includes psychological safety, mental health and well-being. It is found that the quantification of stress data can lead to the identifi- cation of an under-stress worker (Aghajani et al., 2017) and other mental states, thus allowing improvements in worker well-being, efficiency and trust on technology. This is supportive to further developing the mental health risk prevention, relaxation and mitigation strategies, thereby building trust and psychological safety, in different ways of interactions of worker and robot.

For quantification of mental stress, physiological (Schaal et al., 2019), beh- avioral (Brünken et al., 2004) and subjective (Demetriou Constantina, 2015) measures are integrated, processed and analyzed in a smart factory lab setting. The work is aimed to lead towards mentally and emotionally safe working conditions in cobotic environment. The strategy to quantify stress can lead to designing suitable collaborative spaces and jobs, under safe stress limits, thereby, enhancing the motivation, performance and trust on the autono- mous systems. The impact on the human workers as mental stress and fatigue conditions are correlated with the task complexity, speed of work, length of collaborative task and cobot payload etc. Brain physiological measures are acquired to record participants’ neural activity and heart rate, in addition to the standard subjective and behavioral measures. Experiments are designed to perform an object pick-and-place task and co-working with robots. The primary job/activity in the experiment is designed as a sorting activity based on the stroop colour and word test that is a well-known neurophysiological test. The secondary auditory task is added to the primary cobot stroop task (Zakeri et al., 2021) that resembles the scenario for workers to deal with multitasking and handle simultaneous demands like in the actual industrial environment.

Non-invasive neuroimaging data acquisition and processing is used to find the correlations of mental stress with respect to variations in work environ- ment conditions. Mental states often correlate with the brain’s alpha and beta rhythms and changes in haemoglobin concentrations which are collecti- vely observable by a multimodal technique such as EEG+fNIRS (concurrent electroencephalography and functional near-infrared spectroscopy). These patterns are responsible for increasing the information content of the measu- red signals and increase the accuracy of the decoding of mental states. This research defines a strategy for experimental design and the initial acquired patterns against the designed process tasks.

METHODOLOGY

The experiment is designed using collaborative robots and the data acqui- sition system to measure the cerebral activity of the participant, while the participant is working with cobot just like industrial worker remain busy with their tasks in the automated environment.

Building Trust and Safety Correlates for Autonomous Systems 3

Figure 1: Human-robot collaboration: (a) Cobot Stroop task (b) Combination of 3 dif- ferent factors: motion speed, payload capacity and task complexity at 2 levels of high (H) and low (L).
Original PDF Figure 1:

Mental Stress Measurement The main physiological measure in this experiment is EEG in which brain electrical potential detects the neural activity from the voltage differences between electrodes placed on the scalp. The other technique in combination is fNIRS that detects changes in cerebral blood flow and related haemoglobin concentration. Increase in Heart rate (HR) can also be affected by stress and can be calculated from fNIRS signal. We analyse HR for each episode of task and the rest, separately. These provide ways to quantify stress, cognitive workload, sustained attention, vigilance, stress, drowsiness and effects on verbal and spatial memory (Ayaz & Dehais, 2021).

Experimental Procedure Based on the rationale, the experiments are designed using healthy adult volunteers in one main trial to perform an object pick and place task and co-working with robots (See Figure 1-a). Each experimental session is appro- ximately designed to last for an hour, that includes the total time spent by participants to perform the tasks.

The stroop task is a standard psychometric task that has been designed as the main task for the participant in which the colour of the ink has to be considered for decision rather than the printed name of the colour. Forty woo- den cubes with the name of the colour printed with a non-matching colour on each cube are provided for Stroop task. E.g., the label ‘red’ may be prin- ted using colour green. It typically takes more time and cognitive effort to identify the colour of the label. The cubes are placed in a marked corner of the workspace. The Cobot Stroop task is designed in a collaborative manner where the cobot picks up a cube and pass it to the participant. The participant takes the cube from the cobot and place it in the marked area on the working table based on the printed colour on the cubes. If the speed of participant in performing the task is slower than the cobot, they may not be able to take the next coming cube from the cobot. Therefore, the cube moved by the cobot may be dropped on the table and counted as an error for participants’ per- formance. The Cobot Stroop task has to be performed under different speeds of cobot motion, cobot payload capacities, and task complexity from low to high, where the combination of these factors leads to 8 different conditions called episodes as shown in Fig. 1-b.

Figure 2: Participants take the rest episode and fill NASA-TLX questionnaire after completing each task episode.

Figure available in the original PDF paper.

In order to add complexity to the Cobot Stroop task (primary task), a secondary task is added, simultaneously. In this case, the task complexity is high. Participants perform the secondary task by pressing down on the foot pedal, and in response, a series of beeps generated with duration ranged from 500 to 1000 ms. The secondary task is an auditory task employed in this study as a loading task to assess the effect of stress on the primary Cobot Stroop task. It involves the operator’s ability in response to the auditory stimulus while reaction time and detection rate will be recorded and considered as behavioral measurements. This design is close to the industrial environment scenario where sometimes human workers must deal with multitasking and handle simultaneous demands. EEG+fNIRS data collection starts by perfor- ming the secondary task alone for two minutes followed by performing the Cobot Stroop task (episode 1, Figure1-b) for four minutes. Episode 1 is consi- dered as a baseline where the cobot motion speed, payload capacity and task complexity are set as low. A rest episode is then taken for two minutes (See Figure 2). Participants in rest episode calmly sit on a chair for two minutes. Following the first rest episode, participants perform the Cobot Stroop task based on the identified episodes, one after another starting from episode 2.

Data Collection & Pre-processing EEG data recorded from 19 scalp electrodes, based on the international 10-20 system, by TMSi Mobita wireless data acquisition, at a sampling rate of 2000 Hz. The fNIRS data were collected at 10 Hz by Artinis Octamon. The eight fNIRS channels with separation of 20-30 mm between transmitters-receiver pairs cover a region between FP1-F3-F7 on the left and its symmetric counter- part on the right. Initially datasets from 9 healthy volunteers were used in this study. The NASA-TLX questionnaire is a standard form that estimates sub- jective mental workload and stress assessment by considering six established factors, i.e., mental demand, physical demand, temporal demand, performa- nce, effort, and frustration. Each factor has a score ranged from 0 to 100 in increments of 5. In this study, participants are asked to fill the NASATLX questionnaire after completing each task episode, and the average of the six factors are considered.

The raw EEG were pre-processed to remove artifacts and minimize non- brain components of the signals using an ICA-based method (Zakeri et al., 2020b). The data was also band-passed filtered at 0.16-40 Hz using a zero- phase Hamming windowed-sinc FIR filter to reduce the slow drifts and high frequency artefacts, and down-sampled to 200 Hz to reduce computational and storage cost. Then, the relative frequency band-power (FBP) was calcu- lated in distinct frequency bands such as delta (0.5–4 Hz), theta (4–8 Hz),

Building Trust and Safety Correlates for Autonomous Systems 5

alpha (8–12 Hz), four intervals ranging from low beta (12–16 Hz) to high beta (24–28 Hz), and low gamma (28-32 Hz). fNIRS signals may contain artifacts arising from: subject head motion that transiently affects optode coupling, generating sharp deflections; muscle oxygenation, especially near the temporalis muscle, in the form of high-amplitude excursions lasting several seconds; average amplitude differences between channels due to dif- ferent optical coupling and calibration which may remain stable during the experiment; transient large deviations from blood perfusion changes due to subject upper body motion; systemic heart beat artifact at about 1Hz and Mayer waves at about 0.1 Hz and blood flow in superficial (non-cerebral) tissue.

In order to mitigate these artifacts we implemented the following steps. For each fNIRS channel the detector reading was band-pass filtered (0.15-0.5 Hz), then converted into changes of oxy- and deoxy-hemoglobin concentrations via the modified Beer-Lambert law. Filtering eliminated the heartbeat as well as the slowest components. However, the time scale of Mayer waves partly overlap with the expected time scale of cerebral activity, hence were not eliminated. In order to mitigate large amplitude artifacts (systemic such as Mayer waves and motion or muscle induced), outliers in each channel were detected and excluded by using the criterion of being more than three scaled median absolute deviations (MAD) away from the median. In order to minimize subject-specific and optode cou- pling related inter-channel differences, the hemoglobin time series within each task episode was normalized by dividing it by the standard devi- ation of the subsequent rest episode. In the case of the Baseline epi- sode, normalization was with respect to the initial Rest episode. The systematic components were expected to be globally present in the data. Therefore, to further suppress the systematic components at every time point, each channel’s signal was divided by the channel average. These measures collectively addressed all the artifact types listed above, inclu- ding skin blood flow. Further details can be found in our previous work (Zakeri et al., 2020a, 2020b).

RESULTS & DISCUSSION

Figure 2 shows the relative frequency band power (FBP) that were averaged over a number of participants in rest and task episodes. The FBP value for each subject was averaged over all electrodes. The results indicate that alpha activity is higher during rest episode in comparison with task episodes where participants were performing a task and their mental workload was higher. On the other hand, the high beta and gamma band powers are higher during task performance than rest episode. This indicates the negative correlation between alpha and mental workload, and positive correlation between beta and mental workload. The highest beta and gamma band powers obtained during the experimental episode 2 where payload and task complexity were high, following by episodes 1, 3 and 4. Considering the most difficult episode where payload, task complexity and robot motion speed were set to high, the beta and gamma band powers increased compared to episodes 5, 6 and 7, but

Figure 3: Averaged relative frequency band power (FBP) for rest and experimental episodes.
Original PDF Figure 3:
Figure 4: Prefrontal activation during task performance.
Original PDF Figure 4:

decreased in comparison with the first 4 episodes. The reason could be the effect of learning during the experiment. The lowest beta and gamma powers obtained during episode 7 where task complexity and payload were set to low. Based on the FBP results there is a positive correlation between beta and gamma activities and the stress induced by increasing the payload and task complexity.

Activations averaged separately over the right and left prefrontal cortex (PFC) are shown in Figure 3 for Baseline and each Task. The boxes indicate the distribution of the activations over the subjects. The central mark in a box indicates the subject median, and the bottom and top edges of the box are the 25th and 75th percentiles of the distribution over subjects. Results indicate that Baseline activation was the lowest and the Task activation was generally higher in the left PFC than on the right, this being particularly pronounced in episodes 2, 3 and 8. We also calculated the activations corresponding to the type of task loading. The average activation across all low-payload tasks were calculated for the right PFC and this is shown as the first box on the left in Figure 4.

As in the previous figure the central mark in the box shows the median of the distribution over participants. This was repeated for the high-payload

Building Trust and Safety Correlates for Autonomous Systems 7

Figure 5: Prefrontal activation grouped according to the type of task load.
Original PDF Figure 5:
Figure 6: Heart rate in each experimental episode. Boxes indicate the distribution over participants.
Original PDF Figure 6:

tasks and the result is shown as the second box from left in Figure 4. Com- parison of these results suggest that the payload did not significantly affect the activation. Similar calculations were performed for speed and comple- xity, and separately for left PFC. Figure 4 suggests that there was an increase in left PFC activation due to increased speed. The rightmost pair of values in Figure 4 show that high complexity resulted in higher activation in the left PFC.

The heart rate for each experimental episode is shown in Figure 5. The figure indicates that the Rest and Task episodes had the lowest and highest heart rates, with the Baseline occupying an intermediate range. In addition, the median heart rate peaked at Task 2 and then decreased. Although the difference between Rest and other episodes were significant there was no statistically significant differences in HR between different types of task loa- ding. Our results show that activation was on the whole greater in the left PFC than on the right. This is consistent with the known dominance of the left hemisphere in attention and overall movement organization and sele- ction regardless of subject handedness (Serrien & Sovijärvi-Spapé, 2016). Figure 3, 4 and 5 indicate that higher complexity increased FBP and left PFC’s activation, while higher payload or speed may have slightly decrea- sed it. In our experimental design, higher task complexity was induced by having a secondary task (pedal response to auditory cue) performed in paral- lel with the primary cobot-Stroop task. Thus, our results show that higher PFC engagement and FBP were caused by multi-tasking. The drop in activa- tion with increasing payload and speed suggested by Figure 3 and Figure 4 may be due to the fact that a participant may reach capacity and begin to be overloaded, with a resulting decrement in concentration and PFC activation (Aghajani et al., 2017).

CONCLUSION

The aim of the present study is to assess mental stress imposed to human worker during working with robots in smart factory industrial scenario. The experiment was designed by integrating physiological, behavioral and sub- jective measures to analyse mental stress in a smart factory lab setting. We studied the effect of the imposed stress to the human based on task comple- xity, robot’s payload capacity and motion speed at two levels of high and low. Our initial findings show the general agreement between FBP and deox- yhaemoglobin concentration changes metrics, and the positive correlation between task complexity and mental workload and stress level, as well as the positive correlation between HR and FBP changes during task and rest epi- sodes. Our results suggest that noninvasive mobile functional neuroimaging can provide discriminative metrics for quantifying mental stress caused by cobots.

ACKNOWLEDGMENT

This research has received funding from Connected Everything (CE II) Fea- sibility Studies, Engineering & Physical Science Research Council (EPSRC Award reference No. 4811539) and PepsiCo Europe.

REFERENCES

Aghajani, H., Garbey, M., & Omurtag, A. (2017). Measuring mental workload with EEG+fNIRS. Frontiers in Human Neuroscience, 11, 359. https://doi.org/10.3389/FNHUM.2017.00359/BIBTEX Arai, T., Kato, R., & Fujita, M. (2010). Assessment of operator stress induced by robot collaboration in assembly. CIRP Annals - Manufacturing Technology, 59(1), 5–8. https://doi.org/10.1016/j.cirp.2010.03.043 Ayaz, H., & Dehais, F. (2021). NEUROERGONOMICS. Handbook of Human

Factors and Ergonomics, 816–841. https://doi.org/10.1002/9781119636113. CH31 Brünken, R., Plass, J. L., & Leutner, D. (2004). Assessment of cognitive load in multimedia learning with dual-task methodology: Auditory load and moda- lity effects. Instructional Science, 32(1–2), 115–132. https://doi.org/10.1023/b: truc.0000021812.96911.c5 Demetriou Constantina, B. U. O. and C. A. Essau. (2015). Self-Report Questionnai-

res. In In The Encyclopedia of Clinical Psychology (pp. 1–6). John Wiley & Sons, Inc. https://doi.org/https://doi.org/10.1002/9781118625392.wbecp507. French B, Duenser A, H. A. (2018). Trust in Automation - A Literature Review report.

CSIRO, Clayton Victoria. Hoff, K. A., & Bashir, M. (2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177/0018720814547570 Kaivo-Oja, K. M. and J. (2020). Collaborative Robots: Frontiers of Current Lite-

rature. Journal of Intelligent Systems: Theory and Applications, 3(2), 13–20. https://doi.org/https://doi.org/10.38016/jista.682479 Kopp, T., Baumgartner, M., & Kinkel, S. (2020). Success factors for introducing industrial human-robot interaction in practice: an empirically driven framework. International Journal of Advanced Manufacturing Technology, 685–704. https://doi.org/10.1007/s00170-020-06398-0

Building Trust and Safety Correlates for Autonomous Systems 9

Schaal, N. K., Hepp, P., Schweda, A., Wolf, O. T., & Krampe, C. (2019). A Functional Near-Infrared Spectroscopy Study on the Cortical Haemodynamic Responses During the Maastricht Acute Stress Test. Scientific Reports, 9(1), 1–11. https://doi.org/10.1038/s41598-019-49826-2 Serrien, D. J., & Sovijärvi-Spapé, M. M. (2016). Manual dexterity: Functional

lateralisation patterns and motor efficiency. Brain and Cognition, 108, 42–46. https://doi.org/10.1016/J.BANDC.2016.07.005 Wang, L., Gao, R., Váncza, J., Krüger, J., Wang, X. V., Makris, S., & Chryssolouris, G. (2019). Symbiotic human-robot collaborative assembly. CIRP Annals, 68(2), 701–726. https://doi.org/10.1016/j.cirp.2019.05.002 Zakeri, Z., Mansfield, N., Sunderland, C., & Omurtag, A. (2020a). Cross-validating

models of continuous data from simulation and experiment by using linear regression and artificial neural networks. Informatics in Medicine Unlocked, 21, 100457. https://doi.org/10.1016/J.IMU.2020.100457 Zakeri, Z., Mansfield, N., Sunderland, C., & Omurtag, A. (2020b). Physiological correlates of cognitive load in laparoscopic surgery. Scientific Reports 2020 10:1, 10(1), 1–13. https://doi.org/10.1038/s41598-020-69553-3 Zakeri, Z., Omurtag, A., Breedon, P., Hilliard, G., & Khalid, A. (2021). Studying

Mental Stress Factor in Occupational Safety in the Context of the Smart Factory. Proceedings of the 31st European Safety and Reliability Conference, ESREL 2021, September, 92–99. https://doi.org/10.3850/978-981-18-2016-8_024-cd

Cover of the proceedings volume Industrial Cognitive Ergonomics and Engineering Psychology

Explore the complete digital volume

Industrial Cognitive Ergonomics and Engineering Psychology

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-11-7

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →