Shashank Uttrani, Bhavik Kanekar, Aadhar Gupta, Harsh Katakwar, Varun Dutt (2022). Evaluating Human Performance in a Complex Search-and-Retrieve Task. In: Human Error, Reliability, Resilience, and Performance. AHFE Open Access, vol. 33, pp. 141–147. https://doi.org/10.54941/ahfe1001578
Prior research has investigated human performance in simple psychological tasks with a smaller cognitive workload. However, little is known about how humans learn in complex search-and-retrieve simulated environments. The primary objective of our research was to evaluate human performance in a complex search-and-retrieve environment. We developed a complex simulated environment, mimicking a military on-ground operation, using Unity 3D with targets and distractors. Fifty human participants were recruited to play the simulated game for 25 minutes. Participants were tasked to maximize their score by collecting targets items and avoiding distractor items available within the environment. The game's duration was divided into training and testing phases, which differed in terms of availability of feedback and the time duration (15 minutes for the training phase and 10 minutes for the test phase). In the training phase, the participants were allowed to navigate the environment to collect the items (14 targets and 7 distractors) with scores as feedback. Participants had to navigate the environment while collecting the items (28 targets and 14 distractors) to maximize their score without feedback. Results revealed a significant difference in the performance of human participants from the training phase to the test phase. The participants scored significantly more in the test phase without feedback than the training phase with feedback. Also, there was a significant increase in the proportion of targets collected over the time in both the train and test phases. We highlight the implications of developing simulation tools for training personnel in different tasks.
One of the fundamental reasons to study human performance under sim- ple and cognitive workload is to understand the intrinsic limitations of the human information processing system (Schneider and Shiffrin, 1977). This research is essential for the further development of human-like mach- ine intelligence and understanding the multitasking abilities of humans (Robertson, 1985).
A number of studies in psychology can be found illustrating the effect of simple and complex cognitive on tasks involving human per- formance (Schneider and Shiffrin, 1977, Norman and Bobrow, 1975,
Harrison et al., 2013). Norman and Bobrow (1974) analyzed the impact on human performance when numerous active cognitive processes compete for limited processing resources of the brain. Their results demonstrate a decre- ase in performance when two or more cognitive processes compete and may interfere with one another. However, the study did not involve any trai- ning condition for the participants to learn before being tested for their performance.
Several neurological studies have also been conducted to assess the cogni- tive workload and its impact on human performance. Harrison et al. (2010) used optical brain imaging sensors to assess the cognitive workload of air traf- fic controllers using an air traffic monitoring simulator in training and test conditions. Results revealed a significant increase in the performance in the test condition after training. However, no such investigation was performed using human participants for search-and-retrieve tasks (Vohra et al., 2022).
Although prior research has investigated human performance in simple psychological and neurological tasks with a smaller cognitive workload, how- ever, little is known about how humans learn in complex search-and-retrieve simulated environments. The primary objective of this research is to evalu- ate human performance in a complex search-and-retrieve environment under the influence of the availability of feedback. Also, our objective is to evaluate human performance over time in such complex and high cognitive demand tasks.
In what follows, we present a lab-based experiment involving human participants, who were tasked to explore the environment present in the simulation and maximize their score by collecting as many target items while avoiding the distractor items. Finally, we close the paper by discussing the implications of our results for the human factors community.
EXPERIMENT This section details an experiment to evaluate human performance in the presence or absence of feedback in a search-and-retrieve task (Vohra et al., 2022).
Participants A total of 50 participants were recruited from the Indian Institute of Tech- nology Mandi to participate in the study. Participation in the study was voluntary, and about 71% of participants were males, and the rest were females. Ages ranged from 22 to 39 years (Mean = 25.5 years and stan- dard deviation = 3.4 years). The education level of the participants differed as 5.5% were pursuing undergraduate degrees, 69% were graduate degrees, and 25.5% were pursuing doctoral degrees. The demographics were: 93% possessed degrees in science, technology, engineering, and mathematics, and 7% had degrees in humanities and social sciences. Participants were com- pensated INR 50 (~0.67 USD) for their participation in the study. This study was carried out following the Ethics Committee’s recommendations at the Indian Institute of Technology Mandi with written consent from all participants.
Evaluating Human Performance in a Complex Search-and-Retrieve Task 143
Experiment Design A simulated environment depicting a search-and-retrieve on-ground military operation was developed using Unity 3D (Xie, 2012), a professional gaming engine. The environment consisted of 4 buildings with several items (targets and distractors) spread randomly throughout the environment. The recrui- ted participants were tasked to maximize their score by collecting as many target items as possible while avoiding distractor items. The simulation game- play was divided into two phases: the training phase and the test phase. In the training phase, participants were given feedback in the form of a score upon collecting an item. For collecting each target item, a positive reward of 5 points was awarded to the participant, whereas, for collecting each distra- ctor item, a negative reward of −5 points was awarded to the participant. However, no feedback was provided to the participants in the test phase as they collected the items in the environment. The training phase continued for a total of 15 minutes, whereas the test phase continued for 10 minutes. Also, the total number of targets and distractors in the training phase were 14 and 7, respectively, and the total number of targets and distractors in the test phase were 28 and 14, respectively. Moreover, we increased the types of targets and distractors in the test phase compared to the training phase to generalize the learning in the environment.
Procedure The search-and-retrieve simulation game was developed using Unity 3D (Xie, 2012; Field, 2013). Participants were recruited for the experiment to play the game. Their objective was to collect as many target items as pos- sible in the environment while avoiding the distractor items. The gameplay was 25 minutes long, divided into two phases: training and test. The training phase was 15 minutes long and the test phase was 10 minutes long. At the beginning of the gameplay, participants were presented with the instructions related to the game, and their consent and demographic details were recor- ded. Upon completion of the game, participants were thanked remunerated for their participation.
Data Analyses We checked different assumptions and performed a repeated measures analy- sis of variance (ANOVA) (Field, 2013) for analyzing data to investigate the influence of the availability of feedback on human performance. The alpha level was set at .05, and the power was set at .80. The dependent measure, i.e., the score of each participant, was calculated as a difference of total positive reward collected and total negative reward collected as shown in Eq. 1.
Score =
Total targets collected −Total distractors collected
∗5 Q-Q plots (between expected quantiles and normal quantiles) (Marden, 2004) were used to determine the normality of the dependent vari- able. Also, the homogeneity in the variance of the dependent variable was determined using the scatter plots.
Figure 1: The average score attained by human participants across the training and test phases. The error bar shows 95% confidence interval around the average estimate.
RESULTS
We performed a repeated measure analysis of variance to investigate the influ- ence of the availability of feedback on human performance and the increase in human performance over time.
Influence of Availability of Feedback on Human Performance The availability of feedback significantly influenced human performance, i.e., the total score achieved by a participant in the training and test phase (F (1, 49) = 58.127, p < .001, η2 = 0.543). Figure 1 shows that participants sco- red an average of 24.6 points in the training phase (feedback-present) and participants scored an average of 46.6 points in the test phase (feedback- absent). Therefore, our expectations of humans performing better in the feedback-absent condition after training in the feedback-present condition have been met.
Influence of Time on Human Performance Over the Time There was a significant increase in the performance of human participants in the training phase (F (14, 36) = 7.949, p < .001, η2 = 0.590) as well as test phase (F (9, 41) = 12.611, p < .001, η2 = 0.639). Figure 2 shows that the performance of human participants increased gradually in both training and test phases. However, participants scored more in the test phase as compared to the training phase. In the training phase, participants scored an average cumulative score of 24.6 in 15 minutes, while in the test phase, participants scored an average cumulative score of 46.6 in 10 minutes. Therefore, our expectations have been met.
DISCUSSION & CONCLUSION Prior research has investigated psychological and neurological parameters that influence human performance in different tasks involving little cognitive
Evaluating Human Performance in a Complex Search-and-Retrieve Task 145
Figure 2: The average cumulative score of human participants over the time across training and test phases. The error shows 95% confidence interval around the average estimate.
workload (Schneider and Shiffrin, 1977; Norman and Bobrow, 1975; Harrison et al., 2013). However, little was known about how humans would perform in a complex search-and-retrieve task over time. Also, little was known about how the availability of feedback would influence the perfor- mance of humans in such cognitive-demanding tasks. The main objective of this paper was to address this literature gap.
Our results revealed that human performance was better in the feedback- absent condition after training in the feedback-present condition. A signifi- cant improvement in human performance was observed between the training and test phase. A likely reason for this result could be the ability of humans to learn the distinction between targets and distractors while exploring the environment in the feedback-present condition and then retrieve the learned knowledge in the test phase, i.e., the feedback-absent condition. Moreover, in the test phase, the number and types of targets and distractors were increa- sed. However, the human participants were able to clearly able to differentiate items as targets and distractors and score more than the training phase. This indicates the generalization ability of human learning across the training and test phase.
Results also revealed an increase in human performance over time in the training and the test phase. This increase in human performance is likely because participants gained familiarity with the environment over time. This increased familiarity facilitated the collection of targets items, thereby increasing the score of participants over time.
This research has some implications for the human-factors community. First, an implication from our results is that human performance can be improved by providing planned training in a known environment with the availability of feedback. Second, human data from this simulation can be used to develop cognitive models using theories like instance-based learning theory to account for human actions in similar search-and-retrieve tasks.
A limitation of this research is that the results have been obtained from a laboratory experiment involving a simulation tool. Although the real-world conditions could be different from the conditions stipulated in the lab, some of the conclusions mentioned above are likely to hold in the real world. One reason for this expectation is because the search-and-retrieve task was deve- loped to simulate real-world experience using a state-of-the-art professional gaming engine, i.e., Unity 3D (Xie, 2012). Second, this research tried to reproduce the dynamics of an actual search-and-rescue military operation: a training phase (feedback-present) followed by a test phase (feedback-absent).
Various ideas can be taken forward from this research for future experi- mentation. First, similar simulations can be developed for different cognitive demand tasks (Kool et al., 2010) based on their heterogeneity (van Maanen et al., 1989) and level of required workload (Putze et al., 2010). Second, various machine learning models (such as Soft-Actor Critic (Haarnoja et al., 2018) and Proximal Policy Optimization (Wang et al., 2020)) and compu- tational cognitive models (such as instance-based learning models (Gonzalez et al., 2003, Gonzalez and Dutt, 2011, Sharma et al., 2020)) can be developed to account for human choices in such tasks. Third, multi-agent simulations of such games can be developed to investigate how human performs in a team with other humans or robots in similar search-and-retrieve tasks (Sheridan, 2016). We plan to continue experimenting with some of these ideas in our ongoing research.
ACKNOWLEDGMENT
A grant supported this research from Center for Artificial Intelligence and Robotics, Defence Research and Development Organization titled “Repli- cating human cognitive behavior on robots’ models using ACT-R cogni- tive architecture for search-and-retrieve missions in virtual environments” (Project number: IITM/DRDO/VD/324) to Prof. Varun Dutt.
REFERENCES
Field, A. 2013. Discovering statistics using IBM SPSS statistics, sage. Gonzalez, C. & Dutt, V. 2011. Instance-based learning: Integrating sampling and repeated decisions from experience. Psychological Review, 118, 523–551. Gonzalez, C., Lerch, J. F. & Lebiere, C. 2003. Instance-based learning in dynamic
decision making. Cognitive Science, 27, 591–635. Haarnoja, T., Zhou, A., Abbeel, P. & Levine, S. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In: JENNIFER, D. & ANDREAS, K. (eds.) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research: PMLR. Harrison, J., Izzetoglu, K., Ayaz, H., Willems, B., Hah, S., Woo, H., Shewokis, P. A.,
Bunce, S. C. & Onaral, B. Human Performance Assessment Study in Aviation Using Functional Near Infrared Spectroscopy. 2013 Berlin, Heidelberg. Springer Berlin Heidelberg, 433–442. Kool, W., McGuire, J. T., Rosen, Z. B. & Botvinick, M. M. 2010. Decision making and the avoidance of cognitive demand. Journal of experimental psychology: general, 139, 665.
Evaluating Human Performance in a Complex Search-and-Retrieve Task 147
Marden, J. I. 2004. Positions and QQ Plots. Statistical Science, 19, 606–614. Norman, D. A. & Bobrow, D. G. 1975. On data-limited and resource-limited processes. Cognitive Psychology, 7, 44–64. Putze, F., Jarvis, J. & Schultz, T. Multimodal Recognition of Cognitive Workload
for Multitasking in the Car. 2010 20th International Conference on Pattern Recognition, 23–26 Aug. 2010 2010. 3748-3751. Robertson, I. T. 1985. Human information-processing strategies and style. Behaviour & Information Technology, 4, 19–29. Schneider, W. & Shiffrin, R. M. 1977. Controlled and automatic human information
processing: I. Detection, search, and attention. Psychological Review, 84, 1–66. Sharma, N., Uttrani, S. & Dutt, V. Modeling the Absence of Framing Effect in an Experience-Based COVID-19 Disease Problem. International Conference on Cognitive Modelling, 2020. Sheridan, T. B. 2016. Human–Robot Interaction:Status and Challenges. Human
Factors, 58, 525–532. Van Maanen, L., Been, P. & Sijtsma, K. 1989. The Linear Logistic Test Model and heterogeneity of cognitive strategies. In: Roskam, E. E. (ed.) Mathematical Psychology in Progress. Berlin, Heidelberg: Springer Berlin Heidelberg. Vohra, I., Uttrani, S., Rao, A. K. & Dutt, V. Evaluating the Efficacy of Different Neu-
ral Network Deep Reinforcement Algorithms in Complex Search-and-Retrieve Virtual Simulations. 2022 Cham. Springer International Publishing, 348–361. Wang, Y., He, H. & Tan, X. 2020. Truly Proximal Policy Optimization. In: RYAN, P. A. & VIBHAV, G. (eds.) Proceedings of The 35th Uncertainty in Artificial Intelligence Conference. Proceedings of Machine Learning Research: PMLR. Xie, J. Research on key technologies base Unity3D game engine. 2012 7th Interna-
tional Conference on Computer Science & Education (ICCSE), 14–17 July 2012 2012. 695-699.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Error, Reliability, Resilience, and Performance
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-09-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional