Yingting Chen, Taro Kanno, Kazuo Furuta (2022). Complex Scenario Design for Investigating Cognitive Process for Problem-Solving Collaboration. In: Human Factors, Business Management and Society. AHFE Open Access, vol. 56, pp. 25–32. https://doi.org/10.54941/ahfe1002248
Complex problem solving (CPS) has been a field that uses computer-simulated scenarios and has been applied in problem-solving-related studies. However, the problem scenario has not been thoroughly discussed as an essential factor in determining the reliability of the studies. Consequently, there are no systematic principles for scenario design in CPS studies. This study was performed to establish fundamental standards for complexity analysis in the CPS scenario design. We created a high-fidelity problem scenario to investigate the cognitive processes in CPS discussions. The reliability of the system and scenario was validated by five industrial experts. The findings of this study can be applied to future experiment designs, meta-analysis methods, and study replications.
In complex problem-solving (CPS) studies, the problem scenario design may directly influence the quality of data collection. Computer-simulated scena- rios have been used in problem-solving-related studies; however, the problem scenario has not been thoroughly discussed as an essential factor in determi- ning the reliability of these studies. Consequently, there are no systematic principles for analyzing problem scenarios or the scenario design in CPS stu- dies. In addition, when studies need to be cross-compared, there are often no adequate metrics to refer to. Hence, only general comparisons can be made using the number of variables (Funke, 1991) or structure of the task (Stadler et al., 2015).
This study falls under a project that aims to propose a platform-versatile CPS discussion guideline. This paper documents the steps taken to design a complex experimental scenario. We proposed a set of standards that enable an in-depth analysis of the problem complexity. We further designed a com- plex, high-fidelity, computer-simulated scenario using the proposed standards and verified it with expert knowledge. The results presented in this paper
Table 1. Taxonomy for fundamental scenario complexity analysis.
Table available in the original PDF paper.
Taxonomy Dimensions Basic factor Complex factor Time-related Time-dependency Event-driven Clock-driven Decision making Discrete Continuous Feedback interval Immediate Delayed System behavior Information availability Transparent Opaque Randomness Deterministic Stochastic Variable’s values Discrete Continuous Individual-related Interaction type Reactive Predictive Representation forming Learning Non-learning Problem-solving behavior Search-based Understanding-based Scenario features Knowledge requirements Lean Intensive Inter-links Linear Nonlinear Uniqueness Well-defined Ill-defined
contribute to future experimental designs, meta-analysis methods, and study replications.
METHOD
Taxonomy for Fundamental Complexity Analysis “Complex problem” as a general term covers various topics. Jonassen (2000) proposed a typology comprising 11 types of problems and, correspondingly, a set of high-level strategies to support problem-space construction. Alth- ough the researcher suggested criteria for problem categorization, it was still difficult to design a complex scenario with descriptive content. For insta- nce, abstractness, which is also interpreted as domain specificity, has several values including problem-, context-, case-, issue-situated, etc. However, these values are not necessarily mutually exclusive, and do not have standard inter- pretations without further definitions. To define complex problems, previous studies have proposed lists for complexity analysis (Quesada, Kintsch and Gomez, 2005; Grösser, 2017). We revised the captioned lists into a taxonomy: time-related, system behavior, individual-related, and scenario features. Each characteristic can be identified through either complex or basic factors that can be used as Boolean parameters in a simulated scenario design (refer to Table 1. The definition of each dimension can be found in the two literatu- res). Thereafter, we created a quantitative interpretation of the fundamental scenario complexity, which is the number of complex factors to be conside- red. We emphasize the term “fundamental” because Boolean parameters are regarded as extremes, whereas in the real world, representations could be intermediate between these parameters (Miller, 1988).
Scenario Design Based on a widely used scenario, the Tailor shop (Danner et al., 2011; Süß and Kretzschmar, 2018), we designed a complex scenario based on the afore- mentioned taxonomy. The scenario simulates coffee booth operations, where
Complex Scenario Design for Investigating Cognitive Process 27
Figure 1: Simulation interface.
problem-solvers make operational strategies each month, in a course of one simulated year with the goal of maximizing the monthly balance.
The scenario adopted in this study has a fundamental complexity of six factors, including opaqueness, predictive interaction, search-based problem- solving behavior, continuous variables, non-linear inter-links, and knowledge intensity. Its system has 27 intervention variables, 25 non-modifiable varia- bles, 19 hidden variables, and 122 interlinks. To reduce the variability due to knowledge intensity, an video was created to inform general participants about basic café operational knowledge, before they operate the system.
The reliability of the scenario was validated by expert participants with more than five years of experience in café operational management. The evaluation was conducted in two steps: first, the participants were asked to complete the entire simulation in a 60-minute one-on-one online interview; thereafter, they were required to answer a questionnaire regarding their problem-solving process. The questionnaire evaluates three aspects using a five-point Likert scale: participants’ problem-solving ability, scenario fidelity, and system usability (refer to Table 2). “Problem-solving ability” measures the ability to understand a problem, devise a plan, execute the plan and review (Polya, 1945). “Scenario fidelity” evaluates whether the simulated situation is intuitive enough for the experts to apply their theoretical know- ledge and practical experience rationally (Chen, Kanno and Furuta, 2021). Finally, “System usability”evaluates the ease of use of the simulation UI (Bar- num, 2021). The scenario is considered reliable if the average score of each category exceeds three.
RESULTS
Five industrial experts evaluated the simulated scenarios. The problem- solving performance of the simulated system is illustrated in Figure 1 The
Table 2. Question for problem-solving process evaluation.
Table available in the original PDF paper.
1. Problem-solving ability 2. Scenario fidelity 3. System usability a. I was able interact with the system without difficulties. b. I was able to understand the numbers in the system without difficulties. c. I was able to understand the graphs in the system without difficulties. d. I was able to interact with the system without stress.
a. I could provide specific rationales for my actions during the simulation scenario. b. The simulation scenario allowed me to apply my knowledge reasonably. c. The simulation scenario allowed me to apply my practical experience reasonably. d. My actions were based on the knowledge and skills I acquired from my professional training. e. My actions were based on the experience I acquired from daily life experiences. f. The introduction video and texts reflected real-world scenarios reasonably g. The results of the simulation reflected real-world scenarios accurately.
a. I understood the meaning of all given elements in the simulation scenario. b. I became aware of some not-given elements in the simulation scenario. c. I could explain the reasons regarding the changes in performance. d. I established my goal in the simulation scenario. e. I determined the appropriate actions to take to achieve my goal in the simulation scenario. f. I executed these actions that I determined to achieve my goal in the simulation scenario. g. I kept checking the effectiveness of my actions. h. I revised my planned actions by reviewing the overall effectiveness of my approach.
monthly balance ranged from −1.12E + 06 to 7.29E + 05. Two out the five participants had negative average balance (Participant A and E), and the other three managed to obtain a positive balance (Participant B, C, D). Parti- cipants were able to achieve their first break even by the eighth month; only Participant C consistently recorded profit.
The results of the scenario evaluation are listed in Table 3. The problem- solving-ability scores exhibited a moderate positive correlation with the monthly balance (0.52). Moreover, among all sub-questions, 1c, 1h, and 2f had strong positive correlations with the monthly balance (0.71, 0.78, and 0.71, respectively); as a reliability indicator, the means of all three values exceeded three (4.40, 3.88, and 4.50, respectively). Similarly, the mean of all the sub-questions in each category also exceeded three. Questions 1c, 2c,
Complex Scenario Design for Investigating Cognitive Process 29
Figure 2: Problem-solving performance of expert participants.Table 3. Problem-solving process evaluation.
Table available in the original PDF paper.
Participants A B C D E Corr. with monthly balance Problem-solving ability 3.88 4.25 4.50 5.00 4.38 0.52 Scenario fidelity 3.63 3.75 3.50 5.00 3.50 0.14 System usability 5.00 4.75 3.75 5.00 4.00 −0.40
2e, and 2f all yielded an average score of 3.6, the lowest among all sub- questions. In contrast, Questions 1d, 1f, and 1g yielded 4.8, the highest among all sub-questions. System usability had a significantly higher interqu- artile range (1.00) compared to those of problem-solving ability and scenario fidelity (0.25, 0.25).
DISCUSSION
Experts determined that the system was reliable overall. The positive cor- relation between the monthly balance and problem-solving ability indicated that the simulation result was a valid measure of the participants’ reasoning ability. Meanwhile, the overall scenario fidelity exerted less influence on the monthly balance than the introductory video solely. These results demon- strate the practicability of designing a less biased knowledge-intense scenario for problem-solving investigations.
The weaknesses of the designed scenario can be obtained from the questi- ons with the lowest scores (1c, 2c, 2e, and 2f). These questions indicated the impossibility of reflecting expert experience. This problem is also reflected in the problem-solving performance. When expert participants defined a sce- nario based on the situation of their own shops, rather than a logical analysis of the scenario, they were less likely to obtain a positive monthly balance (see A, D, and E at Month 1 in Figure 2). However, we expected such bias to be less likely when the task was assigned to general participants whose problem representation was based on the introductory video and experience as consu- mers. Moreover, solely relying on expert experience in the simulated scenario could negatively impact problem representation. For example, Participant C established a strategy from Month 1 based on personal experience. Howe- ver, he was not motivated to improve his strategy because his business was
already profitable. Consequently, he failed to discover some hidden elements, which led him to rate the scenario fidelity lower.
In future research, a troubleshooting task may be used instead of a performance-neutral beginning to motivate users to extend the problem representation. In addition, the influence of interdependency between months on performance should be investigated. Finally, we were unable to quantify the complex and basic factors using a standard scale; this should be attempted in future research to achieve more comprehensive scenario comparisons.
CONCLUSION
We proposed standards for CPS scenario design and documented the steps taken toward designing high-fidelity computer-simulated complex scenarios. Furthermore, we designed a taxonomy for fundamental complexity analysis and questionnaires for expert validation and result analysis. The scenario was a simulated coffee booth for which problem solvers crafted operational strategies each month over the course of one simulated year with the goal of maximizing the monthly balance. Experts determined that the system was reliable and accurately reflected their reasoning ability. The findings of this study can be used in future experiment designs, meta-analysis methods, and study replications.
ACKNOWLEDGEMENT
This work was partly supported by JSPS KAKENHI Grant Number JP19H02384.
REFERENCES
Ackerman, P. L. and Kanfer, R. (1993) “Integrating Laboratory and Field Study for Improving Selection: Development of a Battery for Predicting Air Traf- fic Controller Success,” Journal of Applied Psychology, 78(3), pp. 413–432. doi: 10.1037/0021-9010.78.3.413. Barnum, C. M. (2021) “6 - Preparing for usability testing,” in Barnum, C. M. B. T.-U.
T. E. (Second E. (ed.). Morgan Kaufmann, pp. 197–248. doi: https://doi.org/10. 1016/B978-0-12-816942-1.00006-X. Berry, D. C. and Broadbent, D. E. (1987) “The combination of explicit and implicit learning processes in task control,” Psychological Research, 49(1), pp. 7–15. doi: 10.1007/BF00309197. Broadbent, D. E., FitzGerald, P. and Broadbent, M. H. P. (1986) “Implicit and expli-
cit knowledge in the control of complex systems,” British Journal of Psychology, 77(1), pp. 33–50. doi: 10.1111/j. 2044–8295.1986.tb01979.x. Chen, Y., Kanno, T. and Furuta, K. (2021) “Complex problem solving discussion and its performance indicators,” LifeTech 2021 - 2021 IEEE 3rd Global Conference on Life Sciences and Technologies, (LifeTech), pp. 365–368. doi: 10.1109/Life- Tech52111.2021.9391892. Danner, D. et al. (2011) “Measuring Performance in Dynamic Decision Making:
Reliability and Validity of the Tailorshop Simulation,” Journal of Individual Differences, 32(4), pp. 225–233. doi: 10.1027/1614-0001/a000055.
Complex Scenario Design for Investigating Cognitive Process 31
Funke, J. (1985) “Steuerung dynamischer Systeme durch Aufbau und Anwendung subjektiver Kausalmodelle. [Control of dynamic systems through development and application of subjective models of causality.],” Zeitschrift für Psychologie mit Zeitschrift für angewandte Psychologie. Germany: J.A.Barth Verlag, pp. 443–465. Funke, J. (1991) “Solving complex problems: Exploration and control of complex
systems,” in. Greiff, S., Wüstenberg, S. and Funke, J. (2012) “Dynamic Problem Solving: A New Assessment Perspective,” Applied Psychological Measurement, 36(3), pp. 189–213. doi: 10.1177/0146621612439620. Grösser, S. N. (2017) “Complexity Management and System Dynamics Thinking
BT - Dynamics of Long-Life Assets: From Technology Adaptation to Upgrading the Business Model,” in Grösser, S. N., Reyes-Lecuona, A., and Granholm, G. (eds.). Cham: Springer International Publishing, pp. 69–92. doi: 10.1007/978-3- 319-45438-2_5. Halford, G. S., Wilson, W. H. and Phillips, S. (1999) “Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology,” Behavioral and Brain Sciences, 21(6), pp. 803–864. doi: 10.1017/S0140525X98001769. Hübner, R. (1987) “Eine naheliegende Fehleinschätzung des Zielabstandes bei der
zeitoptimalen Regelung dynamischer Systeme,” Zeitschrift für experimentelle und an gewandte Psychologie, (1), pp. 38–53. Jonassen, D. H. (2000) “Toward a design theory of problem solving,” Educational Technology Research and Development, 48(4), pp. 63–85. doi: 10.1007/BF02300500. Kleiter, G. D. (1970) “Trend-control in a dynamic decision-making task,” pp. 387–397. Mané, A. and Donchin, E. (1989) “The space fortress game,” Acta Psychologica, 71(1), pp. 17–22. doi: https://doi.org/10.1016/0001-6918(89)90003-6. Miller, J. (1988) “Discrete and continuous models of human information proces-
sing: Theoretical distinctions and empirical results,” Acta Psychologica, 67(3), pp. 191–257. doi: https://doi.org/10.1016/0001-6918(88)90013-3. Moray, N., Lootsteen, P. and Pajak, J. (1986) “Acquisition of Process Control Skills,” IEEE Transactions on Systems, Man, and Cybernetics, 16(4), pp. 497–504. doi: 10.1109/TSMC.1986.289252. Opwis, K. and Spada, H. (1985) “Erwerb und Anwendung von Wissen über
ökologische Systeme (Acquisition and application of knowledge about ecologi- cal systems),” in Albert, D. (ed.) Grundlagenforschung. Hogrefe, pp. 258–260. Available at: http://edoc.unibas.ch/dok/A5252670. Polya, G. (1945) How to solve it; a new aspect of mathematical method., How to solve it; a new aspect of mathematical method. Princeton, NJ, US: Princeton University Press. Preußler, W. (1985) “Über die Bedingungen der Prognose eines bivariaten ökolo-
gischen Systems [On the determinants of the forecast of a bivariate ecologic system],” in Memo No. 31 at the chair Psychology II. Bamberg: University of Bamberg. Quesada, J., Kintsch, W. and Gomez, E. (2005) “Complex problem-solving: A field in search of a definition?,” Theoretical Issues in Ergonomics Science, 6(1), pp. 5–33. doi: 10.1080/14639220512331311553. Reichert, U. and Dörner, D. (1988) “Heurismen beim Umgang mit einem ‘einfachen’
dynamischen System. [Heuristics of coping with a ‘simple’ dynamic system.],” Sprache & Kognition. Switzerland: Verlag Hans Huber, pp. 12–24.
Stadler, M. et al. (2015) “Complex problem solving and intelligence: A metaanalysis,” Intelligence, 53, pp. 92–101. doi: https://doi.org/10.1016/j.intell.2015. 09.005. Süß, H.-M. and Kretzschmar, A. (2018) “Impact of Cognitive Abilities and Prior Kno-
wledge on Complex Problem Solving Performance – Empirical Results and a Plea for Ecologically Valid Microworlds,” Frontiers in Psychology, p. 626. Available at: https://www.frontiersin.org/article/10.3389/fpsyg.2018.00626. Thalmaier, A. (1979) “Cognitive solutions for the optimal control of a dynamic system.,” Zeitschrift für Experimentelle und Angewandte Psychologie. Germany: Hogrefe & Huber Publishers, pp. 388–421. Wagener, D. and Wittmann, W. W. (2002) “Personalarbeit mit dem komplexen Szena-
rio FSYS:,” Zeitschrift für Personalpsychologie. Hogrefe Verlag, 1(2), pp. 80–93. doi: 10.1026//1617-6391.1.2.80. Wiltshire, T. J., Butner, J. E. and Fiore, S. M. (2018) “Problem-Solving Phase Transitions During Team Collaboration,” Cognitive Science, 42(1), pp. 129–167. doi: https://doi.org/10.1111/cogs.12482.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors, Business Management and Society
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-32-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional