Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Basic Study on Prevention of Human Error- How Cognitive Biases Distort Decision Making and Lead to Crucial Accidents -

Graduate School of Natural Science and Technology, Okayama University, Okayama, Japan ↗
Publication year
Digital Object Identifier10.54941/ahfe100191

Abstract

On the basis of the analysis of past case studies of accidents, it was examined how cognitive biases were ubiquitous in the process of accident outbreak, distorted decision making, and led to crucial accidents. We made an attempt to point out that cognitive biases distort decision making and are potentially related to crucial accidents. Using a few cases of crucial accidents (Challenger space shuttle disaster, collision accident between the Japanese Aegis-equipped destroyer “Atago” and fishing boat, and Three Mile Island nuclear power plant accident), it was demonstrated how cognitive biases are related to these accidents. It was demonstrated that heuristic-based biases such as confirmation bias, groupthink, and social loafing surely appeared in the process of accident breakout. Overconfidence-based biases such as illusion of control, fallacy of plan, and optimistic bias were also ubiquitous in the route to a crucial accident. Moreover, framing biases was found to contribute to the distorted decision making, and eventually turn into the main cause of crucial accident. In such a way, as well as human factors or ergonomics approaches, the prevention and the deletion of cognitive biases were indispensable for preventing crucial accidents from occurring.

INTRODUCTION

Different from the traditional economics, the bounded rationality is commonly assumed in behavioral economics (Kahneman, 2011, Tversky and Kahneman, 1974, Kahneman and Tversky, 1984, Altman, 2012, Angner, 2012, Bazerman and Moore, 2001, Ariery, 2009, Ariery, 2010, and Ariery, 2012). Due to bounded rationality, we generally cannot make decision rationally, and thus suffer from cognitive biases pointed out by Kahneman (2011), Tversky and Kahneman (1974), and Kahneman and Tversky (1984). Kahneman and Tversky (1984) states that our cognitive information processing is conducted by System1 or System2. While System2 requires us to conduct effortful, demanding and deliberate mental activities, System1 operates quickly, automatically, without time consuming, and intuitively with little or no efforts. Although heuristic approaches that we adopt when we have no time to deliberate are based on System1, and are very simple and intuitive, such approaches constantly suffer from cognitive biases.

One of the major causes of the Challenger space shuttle disaster (Reason, 1990 and Vaughan, 1997) is regarded to

Cognitive bias

Escalation of commitment to biased decision and behavior

Hindsight bias Confirmation bias Conformity Groupthink Social loafing Illusion of control Fallacy of plan Optimistic bias etc.

Bias of Decision Making (DM)

Human error (in judgment,DM,behavior) (intentional)Violation

Unsafe behavior Accident

Figure 1. Relational model between cognitive biases and unsafe behaviors or accidents.

Figure available in the original PDF paper.

be due to groupthink, especially illusion of unanimity (Janis, 1982). In this case, although the manufacturer of O- ring recognized the risk of malfunction of O-ring under the severely cold temperature, the manufacturer agreed with the launch of the Challenger space shuttle because of illusion of unanimity. After some serious accident occurred, one tends to overestimate the occurrence probability of such an accident. For example, we hesitate to use an airplane immediately after a serious aviation accident due to the overestimation of a fatal aviation accident. This property is one type of cognitive biases called hindsight bias. It is suggested that such a bias becomes an obstacle to the objective survey of an accident.

Therefore, it is important to get insights into how the cognitive biases are related to, and lead to crucial accidents. In this study, it was discussed how cognitive biases induce crucial accidents by distorting decision making. We made an attempt to point out that cognitive biases distort decision making and are potentially related to crucial accidents. Using a few example of crucial accidents, it was demonstrated how cognitive biases are related to these accidents.

HOW COGNITIVE BIASES LEAD TO UNSAFE BEHAVIORS OR ACCIDENTS As shown in Figure 1, it is hypothesized that cognitive biases cause the distortion of decision making, and thus that this leads to human errors in judgment, decision making, and behavior and eventually (at the worst case) triggers crucial accidents if the commitment to the biased judgment, decision making, and behavior is escalated.

According to Bazerman and Moore (2001), we summarized how cognitive biases are induced. It is hypothesized that the heuristics such as availability, representativeness, confirmation, or affect cause the biases such as confirmation biases, and anchoring and adjustment. In Figure 2, not only heuristics but also overconfidence and framing are shown as causes of biases. Moreover, it is assumed that our bounded awareness and uncertain (risk) situations form the basis of heuristics, overconfidence, and framing. Due to such bounded ability, it is valid that we humans cannot behave rationally but irrationally.

Figure 2. Mechanism of cognitive biases due to heuristics, overconfidence, and framing.

Figure available in the original PDF paper.

As mentioned above, we frequently tend to behave irrationally, and are in most cases unaware of how and to what extent these irrational behaviors influence us. Such irrational tendencies are sure to distort our decisions, and in the worst cases this leads to crucial accidents according to the model in Figures 1 and 2. Without consideration of our bounded rationality (irrationality), we cannot approach the prevention of crucial accidents and analyze the true (genuine) cause (source) of accidents.

In the next chapter (Chapter 3), it is demonstrated how the cognitive biases form the main cause of crucial accidents using three case studies of crucial accidents.

ANALYSES OF ACCIDENTAL CAUSE FROM THE PERSPECTIVE OF COGNITIVE BIASES Challenger space shuttle disaster (Reason, 1990 and Vaughan, 1997)

One of the major causes of the Challenger space shuttle disaster is regarded to be due to groupthink, especially illusion of unanimity. As shown in Figure 2, groupthink stems from confirmation heuristic. In this case, although the manufacturer of O-ring recognized the risk of malfunction of O-ring under the severely cold temperature, the manufacturer agreed with the launch of the Challenger space shuttle because of illusion of unanimity (groupthink (Janis, 1982)). This eventually led to the crucial accident. The summarized analysis of this accident is shown in Figure 3.

Collision accident between the Japanese Aegis-equipped destroyer “Atago” and fishing boat It is possible that social loafing, which stems from confirmation bias or optimistic bias, leads to a crucial accident. On 19, February in 2008, the Japanese Aegis-equipped destroyer “Atago” that belonged to Japanese Ministry of Defense collided with the fishing boat “Seitokumaru.” Consequently, two crews of the fishing boat were missing

The manufacturer of O-ring recognized the risk of malfunction of O-ring under the severely cold temperature.

However

Heuristic (c) confirmation Overconfidence Framing

Bias

The manufacturer agreed with the launch of the Challenger space shuttle.

(c)Groupthink

Challenger space shuttle disaster

Figure 3. Summarized analysis of Challenger space shuttle disaster.

Figure available in the original PDF paper.

and thereafter identified as dead. One of the main causes of this accident is inferred as follows. Although twenty- four crews were working (on duty) on the Aegis-equipped destroyer when the accident occurred at about 4 a.m., nobody properly noticed the fishing boat, and thus could not take a proper countermeasure against the collision. In spite of many crews on duty, every crew must optimistically reckon that someone would notice the fishing boat, if any. This corresponds to the social loafing phenomenon which stems from cognitive bias (confirmation or optimistic bias). The summarized analysis of this accident is shown in Figure 4.

Three Mile Island Nuclear Power Plant Accident The main cause of this accident is that the operators forget to open the valve of the auxiliary (secondary) water feeding pump after the maintenance task of the auxiliary water feeding (emergency feedwater) pump, and did not notice this error for longer hours. This is related to the confirmation bias (see Figure 2) to make operators believe that such a subtle error actually should not be the cause of criticality accident. In spite of automatic operating of ECCS (Emergency Core Cooling System), the operators could not notice the malfunction of the nuclear reactor due to the availability heuristic (especially Halo effect).

On the basis of the past report on the malfunction of the pilot operated relief valve and the past accident related to this malfunction, it should be expected that the pilot operated relief valve cannot be closed in emergent situations. The confirmation bias that such a trouble would not occur in Three Mile Island nuclear power plant also prevented operators from noticing and identifying the malfunction of pilot operated relief valve.

It took long time to identify the cause of the meltdown of the reactor core due to the rapid and abrupt increase of the reactor core temperature. This must be due to the framing effect (bias) that the operators generally can not identify the cause from the multiple perspectives especially under emergent situations. The operators adhered to the narrow frame which they usually have, and could not apply another frame to the problem solving (identification of the cause).

The preparedness or provision for the accident must be enough due to the engagement of job by many (24) crews.

However

Heuristic (c) confirmation Overconfidence Framing

Bias

The crew was trapped into a bias that someone will notice the risk (existence of fishing boat).

(c)Social loafing

Collision accident between the Japanese Aegis- equipped destroyer “Atago” and fishing boat

Figure 4. Summarized analysis of collision accident between the Japanese Aegis-equipped destroyer

Figure available in the original PDF paper.

“Atago” and fishing boat.

The design of the central control room must be also a main cause of the criticality accident. The display system of the central control room was not designed so that the states of the nuclear power plant such as the pilot operated relief valve, the emergency feedwater pump, and the drain tank were easy to recognize and notice, and eventually the cause of the rapid and abrupt increase of the increase of the reactor core temperature could not be identified until the meltdown of the reactor core occurred. The optimistic bias, the fallacy of plan, and the fallacy control (in more detail, see Figure 2) lie here. The designers of the control room and the plant operator must optimistically predict that any situations can be noticed and recognized by the system in status quo. Therefore, they did not take the worst situations into account in designing the central control room due to both fallacies of plan and control. The summarized analysis of this accident is shown in Figure 5.

CONCLUSIONS

As seen and recognized from three examples, many crucial accidents include cognitive biases as one of main causes of the accident. The correction or modification of bias in decision making must be one of the promising measures for preventing crucial accidents.

When the designers, the engineers, and the managers of modern technologies such as transportation systems, nuclear power plants, and social inflation systems don’t understand humans’ fallibility (error-prone properties) related to our irrational mind, we tend to design new and improved systems that don’t take our limitation (irrationality) into account.

The operators forget to open the valve of the auxiliary (secondary) water feeding pump after the maintenance task

Bias

Framing

Heuristic (a) availability (c) confirmation

Overconfidence illusion of control fallacy of plan

The designers of the control room and the plant operator must optimistically predict that any situations can be noticed and recognized by the system in status quo.

The operators generally can not identify the cause from the multiple perspectives especially under emergent situations.

They did not recognize that (i) a subtle lapse lead to a serious accident, and that (ii) the past accident related to the malfunction of pilot operated relief valve should be noticed as a lesson.

The operator could not identify the cause of rapid and abrupt increase of core reactor.

Meltdown of core reactor at Three Mile Island nuclear power plant

Figure 5. Summarized analysis of Three Mile Island nuclear power plant accident.

Figure available in the original PDF paper.

Consequently, we inevitably distort our decisions, and make our errors or mistakes serious, and these distortions or errors lead to magnificent and crucial accidents such as analyzed in the previous section. Without such an understanding to our irrationality, we unwillingly repeat crucial accidents, and cannot get out of vicious circles of similar crucial accidents. The understanding of how cognitive biases distort decision making and lead to crucial accidents is essential in order to avoid such vicious circles as pointed out by Dekker (2006).

REFERENCES

Altman, M. (2012), “Behavioral Economics for Dummies”, John Wiley & Sons Canada, Ltd. Angner, E. (2012), “A Course in Behavioral Economics”, Palgrave Macmillan. Ariery, D. (2009), “Predictably Irrational -The Hidden Forces that Shape our Decisions-“, Harper. Ariery, D. (2010), “The Upside of irrationality -The Unexpected Benefits of Defying Logic at Work and at Home-“, Harper. Ariery, D. (2012), “The (Honest) Truth about Dishonesty -How We Lie to Everyone-Especially Ourselves”, Harper. Bazerman, M. H. and Moore, D. A. (2001), “Judgment in Managerial Decision Making”, Harvard University Press. Dekker, D. (2006), “The Field Guide to Understanding Human Error”, Ashgate Publishing. Janis, I. L. (1982), “Groupthink: Psychological Studies of Policy Decisions and Fiascoes”, Cengage Learning. Kahneman, D. (2011), “Thinking, Fast and Slow”, Penguin Books Kahneman, D. and Tversky, A. (1984), “Choices, Values, and Frames”, American Psychologist, Vol.34. Reason, J (1990), “Human Error”, Cambridge University Press. Tversky, A. and Kahneman, D. (1974), “Judgment under Uncertainty: Heuristics and Biases”, Science, Vol.185. Vaughan, D. (1997), “The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA”, University of Chicago Press.

Cover of the proceedings volume Advances in Cross-Cultural Decision Making

Explore the complete digital volume

Advances in Cross-Cultural Decision Making

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-4951-2095-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →