Minna Rantala, Maria Lindholm, Sari Tappura (2021). Assessing the Success of Risk Assessment and Factors Affecting Its Success. In: Human Systems Engineering and Design (IHSED2021): Future Trends and Applications. AHFE Open Access, vol. 21. https://doi.org/10.54941/ahfe1001111
Even though risk assessment is an important factor in occupational health and safety (OHS) promotion and accident prevention, there are no clear definition of when the risk assessment is successful nor instructions on how to evaluate it. Inadequate risk assessments lead to ineffective safety prevention programs, inefficient use of resources, and criticism towards risk assessment. If decisions are based on misleading results, significant sources of risks are failed to eliminate or reduce.Certain factors have been recognized in the literature to affect the quality of a risk assessment, including initial assumptions, chosen method, risk assessment team, resources, and consultation and collaboration of relevant stakeholders. In evaluating the quality of risk assessment, it is suggested to examine the completeness, accuracy, fidelity, and fitness of the risk assessment results. Different approaches have been identified to be used in the evaluation, such as carrying out a parallel analysis of the system, comparing the results with occurring accidents in corresponding systems, and examining the process. This study aims to add knowledge of when a risk assessment is successful and which factors influence the success. The data was collected from OHS specialists and managers (n = 10) by interviewing them. In addition, the Delphi survey was emailed to 17 OHS specialists and managers resulting in 8, 12, and 13 responses in three rounds. The respondents represented four large companies (manufacturing, transportation and storage, and other technical testing and analysis), one medium-sized company (electrical power generation, transmission, and distribution), consulting business companies, and public authority. The interviews and the Delphi survey focused on when a risk assessment can be said to be successful and which individual and organizational features affect its success. According to the interviews, companies do not systematically evaluate the quality of the risk assessment or its success but rather control if the risk assessments are completed. However, they did acknowledge the importance and need for evaluating the risk assessment processes and results. Based on the Delphi survey, individual features affecting the risk assessment are, for example, understanding of the entity, communication and interaction skills, and understanding of the interdependencies. Organizational features influencing risk assessment include organization’s commitment to safety and safety management, risk assessment being a part of the safety management system, and line organization understanding its responsibilities and operating accordingly. Finally, a successful risk assessment fulfills, among others, the following points; the risk assessment represents reality, the results are utilized in planning and developing work tools, instructions and operations, the employees are aware of the residual risks and able to acknowledge them while working, the risk assessment is utilized in educational and orientation material, and that the measures have been focused and carried out so that the risk is removed or reduced. Further research is needed to formulate a model guiding the evaluation of the quality and success of risk assessment.
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
INTRODUCTION
In promoting occupational health and safety (OHS) and preventing accidents, risk assessment is an important factor (Työturvallisuuslaki 23.8.2002/738, 2002; ISO- 45001:2018, 2018). However, there is no clear definition of when a risk assessment is successful or instructions for how to evaluate it. Inadequate risk assessments lead to ineffective safety prevention programs, inefficient use of resources, and criticism of risk assessment (Pinto et al., 2013). If decisions are based on misleading results, or the recognized risks are not actual, significant sources of risks may not be eliminated or reduced (Backlund and Hannu, 2002; Nenonen et al., 2018).
In literature, certain factors affecting the quality of a risk assessment (including factors related to the process and methods, and to the team and individuals) have been recognized. Among others, the design phase, quality of the background knowledge, chosen method, hazard identification, accuracy of frequency and consequence analysis, resources, documentation completeness, and consultation and collaboration of relevant stakeholders, affect the quality of risk assessment results (Arunraj and Maiti, 2007; Kumasaki and Shoji, 2013; Pinto et al., 2013; Langdalen et al., 2020). Furthermore, the composition of the assessment team is important; the number of participants and their competence, knowledge, and skills influence the results (Pinto et al., 2013). Individuals on the risk assessment team have initial assumptions, and individual qualities (Pinto et al., 2013; Hrica and Eiter, 2020).
In evaluating the quality of a risk assessment, it is suggested in previous studies to examine the completeness (having all the required characteristics), accuracy (being correct and exact), fidelity (representing the workplace system), and fitness (satisfying the formulated objectives and requirements) of the risk assessment results (Pinto et al., 2013). Different approaches have been identified for use in the evaluation, such as carrying out a parallel analysis of the system, comparing the results with accidents occurring in corresponding systems, and examining the process (Pinto et al., 2013).
This study aims to add knowledge of when a risk assessment is successful and which factors influence its success.
METHODS AND MATERIALS This qualitative study was conducted to broaden understanding of the success of a risk assessment. Data were collected from OHS specialists and managers (n = 10) in interviews. In addition, a Delphi survey with three rounds was emailed to OHS specialists and managers (n = 17). The interviews and the Delphi survey focused on when a risk assessment can be said to be successful, and which individual and organizational features affect its success. The respondents represented companies from the following industries and authorities: a) manufacturing, b) transportation and storage, c) electrical power generation, transmission and distribution, d) other technical testing and analysis, e) public authority, and f) consulting. The distribution of the companies, interviews, and Delphi survey are summarized in Table
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
1. The semi-structured form of the interview was based on a previous study (Nenonen et al., 2018) and current research questions. The topics included questions such as how to evaluate the success of a risk assessment, how the residual risks have been assessed, and who evaluates or could evaluate the success of the risk assessment. Interviews were recorded for transcription and analyzed by classifying themes. The job titles of the interviewees varied from safety engineers to health, safety, environment, and quality managers. All interviews were conducted in autumn 2020.
A Delphi survey (Linstone and Turoff, 1975; Stewart, 1987) was used to gather opinions from OHS specialists and managers about when a risk assessment is successful, and which personal and organizational factors affect the success of the risk assessment. It was assumed that OHS specialists and managers are, by definition, aware of the risk assessment principles. Requests to participate was emailed to 17 individuals; eight participated in the first round, 12 in the second round, and 13 in the third round, yielding a response rate of 47.1%, 70.6%, and 76.5%, respectively.
In the first round, participants were asked to describe when a risk assessment can be said to be successful and to describe individual features that affect the risk assessment. A qualitative analysis was then performed, and the participants (n = 8) received feedback. Thematic categorization of the first-round qualitative material was conducted by applying open coding analysis (Flick, 2009) identifying 10 factors that indicate a successful risk assessment and 10 individual features affecting risk assessment. In the second round, the respondents were asked to comment on the results and add factors missing from the list or remove factors if they were not necessarily based on the respondents’ opinion. In addition, they were asked to describe organizational features affecting risk assessment. After the second round, a qualitative analysis was performed to modify the lists based on the answers (n = 12), and the participants again received feedback. In the third round, respondents were asked to choose at most eight factors that indicate a successful risk assessment, at most five individual features affecting risk assessment, and at most eight organizational features affecting risk assessment. Before each round ended, reminders were sent. The third round answers were analyzed using IBM SPSS 27.
Table 1: Background information about the interviews (n = 5) and the Delphi survey (n = 17) Interview
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
RESULTS
According to the interviews, companies do not systematically evaluate the quality of risk assessments or their success. Instead, the companies check whether risk assessments have been completed. However, they acknowledged the importance of and need for evaluating the risk assessment processes and results. Residual risks are mainly assessed at the same time as the risk assessment and planning of the corrective actions. The interviewees noted this deficiency, but at present, there is no systematic return to risk assessment after the actions have been taken. To improve the treatment of residual risks, the interviewees stated that the residual risks should be assessed after corrective actions are completed. In addition, employee awareness of residual risks was weak in some companies; therefore, adding awareness was seen as an area that clearly needs improvement.
Because in the companies the quality of neither the risk assessment nor the corrective
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0 actions were evaluated, a third question was formulated: “Who could evaluate the success of the risk assessment?” Interviewees described the line organization’s responsibility and suggested that the organization could evaluate risk assessments like safety audits. If the quality of risk assessments is ensured through audits, it could save resources compared to going through all risk assessments individually.
In the Delphi survey, the following points were included in a successful risk assessment as presented in Table 2; the risk assessment represents reality, the results are utilized in planning and developing work tools, instructions, and operations, the employees are aware of the residual risks and able to acknowledge them while working, the risk assessment is utilized in educational and orientation material, and the measures have been focused and carried out so that the risk is removed or reduced.
Table 2. Factors included in a successful risk assessment Risk assessment is successful when % the risk assessment represents reality 92 the results are utilized in planning and developing work tools, instructions, and operations 77 the employees are aware of the residual risks and able to acknowledge them while
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
Finally, the most important characteristics of the individual features affecting the risk assessment were understanding of the entity, communication and interaction skills, and understanding of the interdependencies (Table 3). Organizational features influencing risk assessment included the organization’s commitment to safety and safety management, risk assessment being a part of the safety management system, and the line organization understanding its responsibilities and operating accordingly (Table 3).
Table 3. The most important characteristics of the individual and organizational features affecting the risk assessment
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
DISCUSSION
In this study, the success of a risk assessment was examined. When asked to choose a maximum of eight factors in a successful risk assessment, respondents were almost unanimous on the first factor. Otherwise, the scores were evenly distributed, with the latter selected no more than once. “The risk assessment represents reality” was chosen by 92% of the respondents, which is in line with literature, as the quality of the occupational safety risk assessment depends on real data, in other words, actual workplace data instead of assumptions that do not reflect reality (Pinto et al., 2013). However, a question arises: How is it verified that the risk assessment on paper corresponds to the practical operation? If the factor was not chosen, are there eight more important points on the list than the fact that risk assessment is in line with practice? In addition, two factors, “the risk assessment is utilized in educational and orientation material” and “the employees are aware of the residual risks and able to acknowledge them while working”, were selected as the third most important. Similarly, the interview results showed that adding awareness of residual risk is a major development target.
When selecting a maximum of the five most important characteristics of individual features affecting the risk assessment, after the two most notable factors, the following options received each more than 50% of the votes: “Understanding of the dependencies,” “Competence and expertise,” and “Identifying and understanding human factors”. Similarly
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0 to the Delphi survey, in the literature, relevant competence, knowledge about methodologies for identifying hazards, “real-world” tasks and the relevant technical systems, and experience have been emphasized as individual features affecting risk assessment (Kumasaki and Shoji, 2013; Pinto et al., 2013). While respondents chose a maximum of eight organizational features influencing risk assessment, the seven most frequently selected options received each more than 50% of the votes. Furthermore, the last eight received each 0–15% of the votes which indicates quite wide variation in the answers. Compared with the question of individual features, “communication and interaction skills” were considered an important factor influencing risk assessment, in contrast to the organizational level where “information flow” and “work atmosphere/social factors” were considered less important characteristics.
Regarding the response rate for the Delphi survey, response rates for e-mail surveys vary widely (Leong and Austin, 2005; Dillman et al., 2014)In this case, the Delphi survey was sent to OHS specialists and managers in different industries and authorities, and sending reminders in each round increased the number of responses. Slight variations in the words used in a questionnaire or the context of a question can also affect the results which became concrete; therefore, a question was added to the second round to clarify organizational factors in addition to individual factors (Strauss and Corbin, 1998). Carelessly executed, poorly worded, and ambiguous questionnaires and superficial analysis of responses are likely to undermine the Delphi method (Linstone and Turoff, 1975; Stewart, 1987). In the present study, the fact that respondents continued to answer throughout the three rounds reflects well on the questions and analysis, with 8 responding in the first round, 12 in the second, and 13 in the third.
As can be seen from the results, the opinions regarding factors affecting risk assessment and factors included in a successful risk assessment vary, although some options unanimously received the most votes. Further research is needed to formulate a model guiding the evaluation of the quality and success of risk assessment.
ACKNOWLEDGMENTS
The authors would like to acknowledge the Finnish Work Environment Fund for the funding of this research. Moreover, the authors thank the companies and interviewees for participating in the study.
REFERENCES
Arunraj, N. S. and Maiti, J. (2007) ‘Risk-based maintenance-Techniques and applications’, Journal of Hazardous Materials, 142(3), pp. 653–661. doi: 10.1016/j.jhazmat.2006.06.069.
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
Backlund, F. and Hannu, J. (2002) ‘Can we make maintenance decisions on risk analysis results?’, Journal of Quality in Maintenance Engineering, 8(1), pp. 77–91. doi: 10.1108/13552510210420603. Dillman, D. A., Smyth, J. D. and Christian, L. M. (2014) Internet, Phone, Mail, and Mixed-
Mode Surveys: The Tailored Design Method. 4th edn. John Wiley & Sons. Flick, U. (2009) An introduction to qualitative research. 4 th ed. London: SAGE Publications Ltd. Hrica, J. K. and Eiter, B. M. (2020) ‘Competencies for the Competent Person: Defining
Workplace Examiner Competencies from the Health and Safety Leader’s Perspective’, Mining, Metallurgy & Exploration, 37, pp. 1951–1959. doi: 10.1007/s42461-020-00275-w/Published. ISO-45001:2018 (2018) International Organization for Standardization: Occupational Health and Safety Management Systems. Requirements with Guidance for Use (ISO Standard No 45001:2018). Geneva. Kumasaki, M. and Shoji, T. (2013) ‘A Logical Hazard Identification Method in
Workplace’, Chemical Engineering Transactions, 31, pp. 259–264. Available at: www.aidic.it/cet. Langdalen, H., Abrahamsen, E. B. and Abrahamsen, H. B. (2020) ‘A New Framework To Idenitfy And Assess Hidden Assumptions In The Background Knowledge Of A Risk Assessment’, Reliability Engineering and System Safety, 200. doi: 10.1016/j.ress.2020.106909. Leong, F. T. L. and Austin, J. T. (2005) The Psychology Research Handbook: A Guide for
Graduate Students and Research Assistants. 2nd edn. SAGE Publications. Linstone, H. A. and Turoff, M. (1975) The Delphi method : techniques and applications. 3.
pr. London: Addison-Wesley. Nenonen, N., Anttila, S. and Kivistö-Rahnasto, J. (2018) Kuinka löytää ja hallita oikeat riskit? (How to find and manage actual risks?). Tampereen teknillinen yliopisto. Tuotantotalous ja tietojohtaminen. Available at: http://www.tut.fi/tutcris. Pinto, A., Ribeiro, R. A. and Nunes, I. L. (2013) ‘Ensuring the quality of occupational
safety risk assessment’, Risk Analysis, 33(3), pp. 409–419. doi: 10.1111/j.1539- 6924.2012.01898.x. Stewart, T. R. (1987) ‘The Delphi technique and judgmental forecasting’, Climatic Change, 11(1–2), pp. 97–113. doi: 10.1007/BF00138797. Strauss, A. L. and Corbin, J. M. (1998) Basics of Qualitative Research: Techniques and
Procedures for Developing Grounded Theory. 2nd edn, Sage. 2nd edn. Thousand Oaks, CA: SAGE Publications. Työturvallisuuslaki 23.8.2002/738 (2002) Työturvallisuuslaki 23.8.2002/738 (Occupational Safety and Health Act 2002/783), Finlex. Available at: https://www.finlex.fi/fi/laki/ajantasa/2002/20020738 Accessed on 15 December 2021.
Human Systems Engineering and Design (IHSED 2021) https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8987-0
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Systems Engineering and Design (IHSED2021): Future Trends and Applications
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-7923-8987-0
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional