Lisa Elliott, Morgan Janney (2022). Measuring Mental Models. In: Training, Education, and Learning Sciences. AHFE Open Access, vol. 59, pp. 367–371. https://doi.org/10.54941/ahfe1002423
The construct of mental models has been a useful tool for training and learning. Many instructors ask students to draw how something works or their understanding of the subject. While this gives the instructor useful feedback, comparing their own understanding to the students understanding. There are many variations in understanding that are distinct yet correct. The Structure, Behavior and Function framework (SBF, Hmelo-Silver and Pfeffer, 2004) successfully described how mental models change at different levels of learning. However, measuring mental models can present an analysis challenge. This study outlines another way to apply the SBF framework to quantify a person’s mental model of a common task: posting a message to a social network. Was a person's mental model of this task universal? Did participants understand what happened to their data? Participants in this study were university students from three different regions of the United States who were familiar with social media. Participants described in words, pictures, or a diagram of what happens to a comment after it is entered on a social media site. Results demonstrate the lack of a universal mental model amongst participants which suggests a poverty in how data is shared in social media. The analysis methodology proved useful when confronted with the three data types: words, pictures, or a diagram in a single data set.
Since Johnson-Laird’s original book in 1983, the topic of mental models has evolved to represent a hypothetical structure in a person’s mind that is shaped by experience (Jones, Ross, Lynam, Perez, Leitch, 2011). For example, a child may not see the fuzzy four-legged dog as dangerous until the dog nips at the child. Then, the child’s mental model of a dog includes “something that bites.” A good mental model helps a person accomplish a goal quickly, a poor mental model can lead a person to repeated guessing in learning, communication, and system use.
Gero and Kannengiesser (2004) state that there are two worlds: the exter- nal world and the interpreted world. Within the interpreted world, a third world exists which is called the expected world. This is where mental models exist, in our expected world. When constructing our expected world models, prior knowledge, feedback, and motivation are key to align our expected world with the external world. The quality of an individual’s alignment can be examined through eliciting their expected world model or their mental model. Usually this is a drawing, diagram or verbal explanation used to examine
knowledge change, learning, communication, behavior, problem solving, and expertise. One of the challenges has been how to quantify the mental model representations to measure these changes. Researchers have developed the structure behavior function ontology to address this challenge. It is often referred to as the structure behavior function framework.
Hmelo-Silver and Pfeffer (2014) asked three groups of ‘experts’ to pro- vide an explanatory mental model drawing of how an aquarium worked. The experts were middle school students, teachers and biologists. Along with the mental model drawings, they collected verbal explanations. They found that the difference between the three groups was striking. The number of components and what types of components and processes they used in their descriptions differed by expertise level. A subsequent study by Kang, Dab- bish, Fruchter, and Kiesler, (2015) asked participants to draw their mental model of the internet. Novices represented the internet as a simple system, with only surface level structures and no underlying systems. The expert participants represented a complex network of underlying systems. Both stu- dies used the structure, behavior, function framework to organize the data. The details of how they translated the drawings into a written format and then applied the structure behavior function framework needed additional clarification.
Method
In order to clarify this methodology and develop a documented analysis path, a similar study was conducted. Participants were asked to draw the path that a posted message takes from their social network page on Facebook or Insta- gram to their friend’s page. Data were collected between 2011 and 2018 at three different universities in the United States. Undergraduate students (Mage = 25 years, s = 6 years) volunteered for course credit for this insti- tutional review board approved study. One hundred and one participants responded with words, diagrams, and pictures.
Results
Three researchers entered the written data into a spreadsheet program. The researchers interpreted the pictures and the diagrams separately, then revie- wed each other’s work. They used single words and propositions to explain what they interpreted. The data was entered and carefully reviewed multiple times for coherency and accurate representation of the original response. At this point, the superiority of a verbal explanation was clear as the research team had to interpret what different arrows and dots meant.
Only after the research team felt that the data was the truest verbal repre- sentation of the diagrams and pictures provided did we proceed by counting the number of times that a particular word appeared in the participants’ responses. After frequency counts were calculated, we could calculate a fre- quency percentage as well. Then, the researchers began applying the structure behavior function framework to the data.
Gero and Kannengiesser (2004) provided a clear teleological ontology of structures, behaviors, and functions. A structure is a component that can be
Measuring Mental Models 369
Figure 1: Accurate and inaccurate mental model components.
Figure available in the original PDF paper.
thought of as an object. In our study, people, comments, buttons, form boxes, pictures, were all structures. Behaviors are attributes of a structure; what a structure does. In our study, reactions, emojis, commenting, and typing were all behaviors of the people or the social network. A function accounts for the users’ goals as a function of the structure or its effects. In our study, “detrimental effect of social media” and embodied effects were functions. Once the concepts were categorized then we could compare the number of structures, behaviors, and functions that respondents described. A colleague who teaches computer science assessed the answers for the probability of accuracy.
Structures. Nearly all participants had structures such as a person, a com- ment/post, an interface, and a receiving person in their model. Of the 474 structures that participants indicated in their mental models, only two were judged with a low potential accuracy. Less than five participants described underlying structures such as packets, handshakes, and servers in their model.
Behaviors. Of the 530 behaviors that participants indicated in their mental models, five were judged with a low potential accuracy. Several participants described being able to see all data associated with a comment, advertisers reading the comment, and a human in the social network company che- cking each comment individually before it is sent. Some participants had experienced blocking from their friends as they described it in their model.
Functions. Of the 76 functions that participants indicated in their mental models, six were judged with a low potential accuracy. Figure one shows the distribution of the mental model components according to the SBF framework.
CONCLUSION
There seems to be a universal mental model of how the social network pro- cess works from a superficial level. While the hope was that more underlying
370 Elliott and Janney
structures would be included in the participants’ descriptions, that was not the case. This supports many other studies of participants’ mental models of technology. The interaction with the interface conveys enough informa- tion for the person to understand how to use the system but hides much of the complexity of how the system works. For example, Wash, Rader, Vaniea and Rizor (2014) found that most of their users misunderstood what their computer is doing. Quite a few studies have applied this approach to studying mental models of privacy and security (i.e. Wu & Zappala, 2018; Asgharpour et al., 2007; Almuhimedi et al., 2014; Yee, 2004). Other studies have implemented this approach for training and knowledge change purposes (i.e. Farr et al., 2018; Krombholz et al., 2019; Hmelo-Silver & Pfeffer, 2004). In these studies, a small number of responses were gathered and analyzed.
Our approach took a large number of responses and applied the SBF fra- mework. A future step may be to correlate these frequencies with the number of social networks that a person actively uses or their understanding of secu- rity settings within the application. The number of responses did make the analysis more complex. A predetermined list of structures, behaviors, and functions with distractors would have made the analysis process quicker. This approach may be a useful tool for assessing interfaces in the future.
ACKNOWLEDGMENT
The authors would like to acknowledge Angela Fields, Taylor Sterrett, Morgan Atterholt, and Nur Othman Fadilla for their help on this project.
REFERENCES
Almuhimedi, H., Felt, A. P., Reeder, R. W., & Consolvo, S. (2014). Your reputation precedes you: History, reputation, and the chrome malware warning. In 10th Symposium on Usable Privacy and Security (SOUPS 2014), (pp. 113–128). https: //www.usenix.org/conference/soups2014/proceedings/presentation/almuhimedi Asgharpour, F., Liu, D., & Camp, L. J. (2007). Mental models of security risks.
In S. Dietrich & R. Dhamija (Eds.), Financial Cryptography and Data Security (pp. 367–377). Springer. https://doi.org/10.1007/978-3-540-77366-5_34 Farr, E. R., Stoll, J. S., & Beitl, C. M. (2018). Effects of fisheries management on local ecological knowledge. Ecology and Society, 23(3). https://www.jstor.org/stable/26799138 Gero, J. S., & Kannengiesser, U. (2004). The situated function–behaviour–structure framework. Design Studies, 25(4), 373-391. Hmelo-Silver, C. E., & Pfeffer, M. G. (2004). Comparing expert and novice under-
standing of a complex system from the perspective of structures, behaviors, and functions. Cognitive Science, 28(1), 127-138. doi:10.1207/s15516709cog2801_7 Johnson-Laird, P. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness (No. 6). Harvard University Press. Jones, N. A., H. Ross, T. Lynam, P. Perez, and A. Leitch. (2011). Mental models: an
interdisciplinary synthesis of theory and methods. Ecology and Society 16(1): 46.
Measuring Mental Models 371
Kang, R., Dabbish, L., Fruchter, N. & Kiesler, S. (2015). “My data just goes everywhere”: user mental models of the internet and impli- cations for privacy and security. In Proceedings of the Eleventh USE- NIX Conference on Usable Privacy and Security (SOUPS ‘15) (pp. 39–52). https://www.usenix.org/conference/soups2015/proceedings/presentation/kang Krombholz, K., Busse, K., Pfeffer, K., Smith, M., & von Zezschwitz, E. (2019).
“If HTTPS were secure, I wouldn’t need 2FA” - End user and administrator mental models of HTTPS. 2019 IEEE Symposium on Security and Privacy (SP), (pp. 246–263). https://doi.org/10.1109/SP.2019.00060 Wash, R., Rader, E., Vaniea, K., & Rizor, M. (2014). Out of the loop: How automated software updates cause unintended security consequences. In 10th Symposium on Usable Privacy and Security (SOUPS 2014) (pp. 89–104). https://www.usenix.org/conference/soups2014/proceedings/presentation/wash Wu, J., & Zappala, D. (2018). When is a tree really a truck? Exploring mental models
of encryption. In 12th Symposium on Usable Privacy and Security (SOUPS 2018), (pp. 395–409). https://www.usenix.org/conference/soups2018/presentation/wu Yee, K.-P. (2004). Aligning security and usability. IEEE Security Privacy, 2(5), 48–55.
https://doi.org/10.1109/MSP.2004.64
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Training, Education, and Learning Sciences
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-35-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional