Yu Shibuya, Yusaku Okada (2024). Methods of Measuring the Effectiveness of On-site Human Error Response Training Based on Employee Engagement Indicators. In: Human Factors in Design, Engineering, and Computing. AHFE Open Access, vol. 159, pp. 2289–2297. https://doi.org/10.54941/ahfe1005797
One of the important issues in the field of safety management is effective human error prevention education for on-site staff. Currently, many sites not only provide basic human error response training such as confirmation and thorough implementation of basic actions, but also various education methods such as work improvement that takes human factors into consideration, active follow-up with team members, and communication methods that lead to accurate reporting, contact, and consultation. However, the effectiveness of such human error prevention education has only been measured qualitatively.Therefore, this study focused on the engagement of workers and examined a method to multifacetedly evaluate each trainee's attitude toward safety activities before and after the course, including the individual characteristics of the workers (personality such as personality analysis). A questionnaire was designed with work engagement, personal engagement, burnout, employee engagement, psychological safety, personality (Big Five theory), and attributes (job type, position, years of experience) as basic parameters, and a model was created using machine learning to evaluate the following four main factors based on the answers. The four main factors are: (i) loyalty to work, (ii) desire for growth, (iii) desire to contribute to the company, and (iv) sense of happiness (well-being). These indicators were evaluated on a 10-point scale. The effectiveness of the method for measuring the effectiveness of on-site human error prevention training based on the employee engagement index obtained in this study was verified through a survey at several hospitals in Japan. However, there are cases where the effectiveness measurement is unclear, and we are currently continuing to improve the accuracy by expanding the data.
Many companies provide safety education, such as safety management training. The two purposes of education mentioned here are to improve safety-related knowledge and to develop human resources who understand the essence of safety education so that they can actively participate in safety activities and apply it to their practical work. The former is measured quantitatively using assignments, reports, and tests in the course. The latter is particularly important, but it is mostly measured qualitatively based on the
instructor’s subjectivity, and no quantitative measurement method has been established.
The purpose of this study was to assist in measuring the effectiveness of education and the subsequent positive attitude toward safety activities, and a model was created that can assess the following four main factors: (i) loyalty to work, (ii) desire for growth, (iii) desire to contribute to the company, and (iv) sense of happiness (well-being). These indicators were evaluated on a 10-point scale based on the results of the questionnaire responses regarding the personal characteristics and engagement.
In human resource development, an important perspective is the spirit of company loyalty. This is because even when safety training is provided, the response to accidents and emergencies varies greatly depending on the degree of company spirit. This is due in large part to engagement, which is the connection between the company and the individual. Therefore, I wanted to use engagement to measure the part related to human resource development, which is the purpose of education, and created a questionnaire on engagement.
METHOD
Below are details on personality and engagement, and the sequence of steps from measuring personality and engagement on the questionnaire to calculating the main factor scores.
Extraction of Factors Personality and engagement were categorized in detail.
Personality was based on the Big Five theory. It is a common trait theory model that attempts to comprehensively understand personality through five trait factors.
The five traits in the Big Five Theory are mainly Extraversion, Emotional Stability, Intelligence, Agreeableness, and diligence.
In addition, the Big Five theory has been expanded to seven factors by incorporating the perspective of the relationship between personality and human behavior at work into the Big Five theory, which is called Ariki’s Seven Factors. Among the five factors, “extroversion”is classified into “sociability” and “proactivity”, and “emotional stability” is classified into “stability” and “optimism”. From the above, personality was classified into seven factors.
Also, several types of engagement exist. Mainly, there are three typical systems of engagement at work: work engagement, personal engagement, and employee engagement.
In addition, burnout exists as a counter-concept to engagement. I used these four in this study because each engagement and burnout has a slightly different meaning, and the use of multiple engagements allows us to measure engagement from different angles.
The eleven elements of personality and engagement were used to measure the four main factors. Table 1 lists the eleven factors and their descriptions.
Methods of Measuring the Effectiveness of On-Site Human Error Response Training 2291
Table 1. Analysis categories.
Table available in the original PDF paper.
Personality Sociability A disposition in which mental energy is directed toward an object, such as another person or thing. Lively, energetic, talkative, adventurous, and aggressive personality Proactivity Agreeableness Indicates the degree of cooperation with one’s surroundings. Warm, friendly to all, pleasant, humane, generous, and cooperative personality. Diligence It is a dimension related to self-discipline, which indicates solidity in planning and execution of things. Responsible, conscientious, energetic and diligent in work and study. Optimism A tendency to show emotional irritability. A tendency to be less likely to cause mental confusion in response to stress. An ideal personality that is stable in mood, free of complaints, easy going, and not jealous. Stability Intelligence Curious, has a wide range of knowledge, analyzes and thinks things through. Thoughtful, creative, and intelligent. Engagement Work engagement
A positive and fulfilling state of mind in which one is energetic (willing to persevere), enthusiastic (ambitious and challenging in one’s work), and absorbed (focused on one’s work) in one’s work. Personal engagement
A state of feeling a sense of meaningfulness (a sense of being able to expect a return on one’s investment in performing one’s role), security (a sense of being able to use and express oneself), and availability (a sense of having the necessary resources to perform one’s role) when performing one’s role at work. Employee engagement
The voluntary attitude and behavior of employees who are willing to contribute to the organization. Burnout Burnout with symptoms of emotional exhaustion (feeling physically and mentally drained and not wanting to do anything), depersonalization (no longer caring about others), and decreased sense of personal accomplishment (unable to take pleasure in their work and underestimating the importance of their work).
Structuring the Factors Next, a questionnaire was administered regarding the relationship between the 11 elements and each of the four primary factors. In this case, the questionnaire was administered to one subject who had sufficient knowledge of personality and engagement. The questionnaire was a pairwise comparison of the elements, asking which was more related to the main factor, element
2292 Shibuya and Okada
i or element j. This questionnaire was taken for all elements and the results were put into the ISM(Interpretive Structure Modeling) and structured.
Figures 1∼4 show the ISM structural model derived from the questionnaire results for each of the four main factors: (i) loyalty to work, (ii) desire for growth, (iii) desire to contribute to the company, and (iv) sense of happiness (well-being).
The factors at the top indicate that they were rated as having a greater impact on the main factor.
Figure 1: Structuring the factors that form “loyalty to work”.Figure 2: Structuring the factors that form “desire for growth”.
Methods of Measuring the Effectiveness of On-Site Human Error Response Training 2293
Figure 3: Structuring the factors that form “desire to contribute to the company”.Figure 4: Structuring the factors that form “well-being”.
To score the main factors, an influence score was given. Elements of the top row of the figure 1∼4, which was more related to each main factor, was assigned a score of 1, and the elements of the second upper row was assigned 0.5, and this score was used as the influence score. After the third upper row, the main factor was not considered relevant, and an influence score was assigned to the second upper row.
2294 Shibuya and Okada
Calculation of Main Factors Score The relationship between the 11 factors and the main factor is quantified. The scores for each factor are obtained by taking a questionnaire.The questionnaire consists of 70 personality questions and 43 engagement questions. After each person completes the questionnaire, they receive a radar chart showing their personality and engagement scores, and comments corresponding to their scores on the personality factors.
These comments were prepared after organizing each of the personality traits and the behaviors that are considered likely to result from various psychological perspectives.
For the personality analysis, the factor structure of the results of Murakami’s main 5-factor personality test was used to score everyone’s personality building factors, with a minimum score of 0 and a maximum score of 100.
In addition, there are different ways to measure engagement. The Utrecht Work Engagement Scale was used for work engagement, the 13-item measurement tool for personal engagement, the Q12 for employee engagement, and the JBS scale for burnout. For this survey, all responses were scored with a minimum score of 0 and a maximum score of 100. For burnout, a score closer to 100 indicates that the employee is not in a state of burnout.
The figure 5∼7 shows a portion of the questionnaire, the radar chart that appears after the survey, and comments.
Figure 5: Questionnaire sheet.
The personality trait scores and engagement scores (xi) obtained from the questionnaire responses were multiplied by their corresponding impact scores (αi) and summed (y = P αixi). All the scores were then re-scaled to a 10-point scale, which was used as the main factor score.
Methods of Measuring the Effectiveness of On-Site Human Error Response Training 2295
Figure 6: An example of inidividual score evaluated by the proposed method.Figure 7: An example of advice document according to the individual score.
Verification In this study, 21 employees of a company that conducts human error education and human error response activities were surveyed about their personality and engagement, and their main factor scores were calculated.
The employees who completed the questionnaire were divided into two groups: those who had received individualized active learning and those who had not. The average of the main factor scores calculated for each group was averaged and a graph was created. In the figure5, the blue bars indicate employees with individualized active learning (8 employees) and the red bars indicate employees without it (13 employees). Figure 5 shows the results.
The vertical axis is the average of each main factor score, on a 10-point scale, with a score closer to 10 indicating higher loyalty to work, desire for growth, desire to contribute to the company, and sense of happiness (well- being).
2296 Shibuya and Okada
Figure 8: Average of main factor scores.
The figure shows that the group receiving individualized active learning had higher overall scores than the group not receiving this form of education. This suggests that this form of education may increase the spirit of company loyalty. In fact, interviews with the employees receiving the training revealed that they were positive about their work and the company, commenting that they were able to rely on the organization when they had problems in their fulfilling work, indicating that the measurement was effective.
In this study, we attempted to measure educational effectiveness through the scores of the four main factors. In the future, I intend to further analyze the relationship between educational methods and educational effects in more detail. First, I would like to characterize each educational method by evaluating its involvement in the following seven axes. The seven axes are (1) expressiveness, (2) logic, (3) diversity, (4) comprehensiveness, (5) ripple effect, (6) adaptability to the field, and (7) versatility. The method of characterization was determined by morphological analysis of the explanatory documents of the educational methods.
Finally, I would like to create a matrix of “advice for future training plans” based on the relationship between “each worker’s four main factors” and “each worker’s degree of involvement in the seven axes with respect to the training content already learned.”
This matrix allows each worker to understand their current achievement status and obtain the outline of the next step of training.
CONCLUSION
One of the most important issues in safety management is effective human error prevention education for on-site personnel. A sense of company loyalty and volunteerism are essential characteristics for personnel who play a central role in safety activities. In this study, we measured these characteristics
Methods of Measuring the Effectiveness of On-Site Human Error Response Training 2297
of belongingness to the company, work engagement, personal engagement, burnout, employee engagement, psychological safety, personality (Big Five theory), and attributes (job title, position, and years of experience) to determine the following: (1) loyalty to the job, (2) desire to grow, (3) desire to contribute to the company, and (4) happiness (wellbeing). The effectiveness of the developed method was verified through surveys at several hospitals and companies in Japan. However, the practicality of the method cannot be achieved unless it is connected to the proposal of guidelines for improvement of educational methods through effectiveness measurement. In the future, we plan to establish a method for characterizing educational methods based on the seven characteristic indices described in the previous chapter and develop it into a method that can evaluate and improve competency education practiced in companies.
REFERENCES
Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of school psychology, 44(5), 427–445. Attri, R., Dev, N., & Sharma, V. (2013). Interpretive structural modelling (ISM)
approach: an overview. Research journal of management sciences, 2319(2), 1171. Barlett, C. P., & Anderson, C. A. (2012). Direct and indirect relations between the Big 5 personality traits and aggressive and violent behavior. Personality and individual differences, 52(8), 870–875. Chika Ariki (Keio University), Yusaku Okada (Keio University). An Evaluation on
the Working Adaptability by Analyzing User’s Personality. 2006 International Ergonomics Association 16th World Congress on Ergonomics. Gallup. Gallup’s Q12 Employee Engagement Survey. Gallup. Available: Gallup’s Q12 Employee Engagement Survey - Gallup. Hayes, N., & Joseph, S. (2003). Big 5 correlates of three measures of subjective
well-being. Personality and Individual differences, 34(4), 723–727. Kular, S., Gatenby, M., Rees, C., Soane, E., & Truss, K. (2008). Employee engagement: A literature review. May, D. R., Gilson, R. L., & Harter, L. M. (2004). The psychological conditions of
meaningfulness, safety and availability and the engagement of the human spirit at work. Journal of occupational and organizational psychology, 77(1), 11–37. Murahashi, M., Okada, Y. (2023). Method for Enhancing Evaluation of the Human Error Probability in Disaster Risk Assessment for Industrial Plants. In: Ronald Boring (eds) Human Error, Reliability, Resilience, and Performance. AHFE (2023) International Conference. AHFE Open Access, vol 82. AHFE International, USA. http://doi.org/10.54941/ahfe1003556 Okada, Y., Aoyagi, T. (2023). A Failure Event Virtual Learning Method that Replaces
Field Experience and Its Effectiveness Measurement. In: Ravindra S. Goonetilleke and Shuping Xiong (eds) Physical Ergonomics and Human Factors. AHFE (2023) International Conference. AHFE Open Access, vol 103. AHFE International, USA. http://doi.org/10.54941/ahfe1003033 Ruriko Yano (Keio University), Yusaku Okada (Keio University).2nd International Conference on Applied Human Factors and Ergonomics. A Proposal of Human Error Tendency Estimation Method on Human Personality - For Nurse’s Work in Hospital-. Saks, A. M., & Gruman, J. A. (2014). What do we really know about employee
engagement?. Human resource development quarterly, 25(2), 155–182.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Design, Engineering, and Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-35-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional