Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Selection and implementation of Evidence based Safety Performance Indicators in Aviation Training

Purdue University, School of Aviation and Transportation Technology, West Lafayette, IN 47907, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1002629

Abstract

Safety management in Aviation training is shaped by identification, evaluation, and measurement of the safety risks. The International Air Transportation Authority (IATA) Technology Roadmap (IATA 2022) offers a synopsis and evaluation of ongoing technology opportunities, which change the aviation environment with the implementation of Artificial Intelligence (AI) and introduction of enhanced Minimum Crew Operations (eMCO) and Single Pilot Operations (SiPO). Change management (airplane design philosophy /ergonomics) affects aviation training design philosophy. A hybrid competency-based education approach in aviation needs interaction with the aviation industry and changes in flight operations (ICAO, 2022). The performance gap of these changes concerning the aviation industry, flight operations, and training is assessed and measured through Evidence-Based Training (EBT) pilot competencies (ICAO, 2016). Safety management systems (SMSs) in aviation training present a lag in identifying and implementing Safety Performance Indicators (SPIs). Implementation of Evidence-based Safety performance indicators as conceptualized in the aviation training through SPI’s (data-based parameters from students' flights) at a theoretical and practical level, act as safety (SMS), personal development (identify pilot weakness – Human Factors), and assessment (Competency-based education) tool (ICAO, 2018).Keywords: Evidence-Based Training (EBT), Safety Management Systems (SMSs), Safety Performance Indicators (SPIs), Human Factors (HF), ergonomics, aviation training.

INTRODUCTION

Adaptation of new technology in aviation depends on training systems design. Evidence-based Training (EBT), according to the International Civil Avia- tion Organisation (ICAO), develops “a new paradigm for competency-based training and assessment of airline pilots based on evidence” (ICAO, 2006). Competency-based training emphasizes new technologies (i.e., AI and ergo- nomics) and the identification of target interventions focusing on students’ learning needs (Keller et al., 2020). In the aviation environment, among the eight critical considerations of the Competency-Based Education Network ([C-BEN] 2018), the selection of “Evidence - driven Continuous impro- vement” and “Collaborative Engagement with External Partners” targets, links collegiate aviation education system and the airline operational. EBT

© 2022. Published by AHFE Open Access. All rights reserved. 63

programs in airline operations rely on the maturity of the operator’s safety management system (ICAO, 2016).

Safety Data Collection and Processing Systems (SDCPS) affect safety per- formance management maturity. Furthermore safety analysis focuses on the development of the organization’s safety objectives -Safety Performance Indi- cators (SPIs) (Maurino, 2017). Safety objectives measurement is achieved by identifying SPIs used to monitor and measure safety performance (ICAO, 2016). Accidents, incidents, flight operations, and training data collection reviewed and updated continually validate course development Safety Perfor- mance Indicators. The lagging indicators focus on long-term trends, events that have occurred, and organization targets as prevention goals. Leading SPI’s serve as a weaknesses and vulnerabilities’ prediction and adaptation to changes maker tool (ICAO, 2018).

Aviation programs following a competency-based approach offer the resulting advantages: (a) aviation safety; (b) advanced training based on knowledge, skills, and abilities; (c) satisfy personnel safety standards; and (d) follow a qualitative (student learning objectives) with quantitative appro- ach accumulating flying hours, (Keller et al., 2020). In order to measure safety risk in training based on the above-described concept, an accurate system model is necessary utilizing data-driven SPIs. SPIs must be speci- fic, measurable, achievable, relevant, and timely oriented (SMART). The School of Aviation and Transportation Technology (SATT) at Purdue Uni- versity has selected CloudAhoy data visualization and analysis technology to integrate students’ flight data from multiple sources. This selected method analyzes, combines, and transforms the data into meaningful information within the safety management system’s flight operations and develops an aviation training approach that utilizes an evidence-based training database from airline operators and students’ flight data. The following design level focuses on implementing machine learning in students’ flight data integration to SPIs. Machine Learning (ML) is a system’s capability to acquire know- ledge through data rather than software elaboration in a critical decision path. However, this could be considered a significant opportunity for the aviation industry to shorten development cycles (EASA, 2020).

DEVELOPING EVIDENCE-BASED SAFETY PERFORMANCE INDICATORS TO SUPPORT AVIATION TRAINING DESIGN This study aims to present, identify and implement evidence-based Safety Performance Indicators in Aviation Training following an interpretivist phi- losophy on the Purdue University case study. Specifically, this study identified and examined safety performance thematic areas chosen by discrete, observa- tional experiences related to the selected aviation university database (EBT), literature review, questionnaires, and current SPIs. Figure 1 presents the phi- losophical rationale that assured the adapted research method (Saunders et al., 2019).

The research followed the single case study approach. Initially, the rese- arch team apprehended the aviation university students’ pilots’ perceptions

Selection and Implementation of Evidence-Based Safety Performance Indicators 65

Figure 1: Research framework schematic presentation.
Original PDF Figure 1:

through an online questionnaire (Yin, 2014) and literature review under the research questions framework:

  • How do students understand SPI’s?
  • How could the proposed SPI’s introduction affect the existing safety management practices?

Through a practical and controlled analysis, the questionnaire results lin- ked the variable indicators and acknowledged the relevance and experiences relating to the research questions and literature review findings for the SPI’s (Coolican, 2019).

Following a thematic coding technique, the research team applied an analytical approach representing the flight training areas related to the rese- arch question framework. The interpretive stage followed the triangulation method (Merriam, 2015). The results of each question are linked with central themes and categorized (Coolican, 2019). Then, a frequency analysis was per- formed for the derived themes to assess repetition in the entirety of the group feedback and to formulate a model of the major themes and the associated responses (King and Horrocks, 2010; Merriam, 2015). This thematic code method oriented the analytical process. The yielded thematic outputs repre- sented the consensus of the grouped responses, whereas any deviations were critically discussed and interpreted (Merriam, 2015).

In designing the aviation training, the School of Aviation and Transpor- tation Technology at Purdue University has focused on the existing SPIs from the EBT airline operator’s database, flight-training organization’s SMS, and selected proposed safety indicators from SATT students flights thematic analysis. Additionally, the research team selected CloudAhoy data visualiza- tion and analysis technology to integrate students’ flight data from multiple sources. This selected method analyzes, combines, and transforms the data into meaningful information within the training syllabus. Debriefs performed using CloudAhoy are used in a variety of training scenarios and mainly in the thematic analysis for the implementation of safety performance indicators of the SATT safety management system. CloudAhoy’s rule-based knowledge engine uses an updated airplanes flight envelope database. Once the fli- ght data has been collected, a structured and controlled thematic analysis - following Evidence-Based Training (EBT) criteria - compared the varying perspectives and identified the qualitative and quantitative SPIs. The analysis

Figure 2: ADDIE approach proposal.
Original PDF Figure 2:

used multiple thematic results indicating the consent of the clustered replies, and a t-test is used as a hypothesis-testing tool, which tests any statement applicable to the population. The analysis part includes a hierarchical skills analysis which provides a bridge between the classical skills taxonomies (such as Blooms) and competencies and can be used as a practical instrument for training design focusing on Human Factors aspects (ICAO, 2006).

SPIs AS A SUPPORTIVE FRAMEWORK FOR AVIATION TRAINING DESIGN The proposed aviation design philosophy follows the ADDIE (Analyze- Design – Develop – Implement – Evaluate) approach, implementing evidence- based safety performance indicators (SPI’s) (Figure 2):

  • Analyze the Flight school SMS – collected data related to EBT ICAO analysis.
  • Design the areas of learning interest related to operations – evidence-based – SMS / (SPI’s) – thematic approach. Introduction of ML/DL software.
  • Develop the courses - learning objectives using the EBT ICAO database (national database option) – develop scenarios.
  • Implement the learning objectives – scenarios.
  • Evaluate using CBTA competencies as described in ICAO documents. Use of ML/DL software capabilities (EASA,2020).

IATA recommendation for aviation training is the implementation of the latest ICAO provisions for competency-based training and assessment. AOCs and ATOs should use the nine (9) Pilot competencies (eight (8) Pilot compete- ncies as proposed by ICAO and the competency “Application of Knowledge” recommended by EASA for EBT.

Based on the current vision of Purdue University on competency-based training (Keller et al. 2020) and the evaluation of ongoing technology oppor- tunities in aviation training, a holistic aviation training approach covers

Selection and Implementation of Evidence-Based Safety Performance Indicators 67

Table 1. SPI’s implementation in the Aviation training system.

Table available in the original PDF paper.

CBTA com- petencies / SPI’s

Pilot competencies Evidence- Based Training Competencies

Purdue Collegiate Competencies

PC 0 / SPI 0 Application of Knowledge

Application of Procedures

Technical Excellence

Technical Excellence PC 1/ SPI1 Application of Procedures and Compliance with Regulations Technical Excellence PC 2/ SPI2 Communication Communication Communication PC 3/ SPI3 Aeroplane Flight Path Management, automation Flight Path Management, automation

Technical Excellence

PC 4/ SPI4 Aeroplane Flight Path Management, manual control

Flight Path Management, manual control Leadership

PC 5/ SPI5 Leadership & Teamwork

Leadership & Teamwork Decision Making

PC 6/ SPI6 Problem Solving & Decision Making

Problem Solving &Decision Making Resilience Situation Awareness PC 7/ SPI7 Situation Awareness and Management of Information

Teamwork

PC 8/ SPI8 Workload Management

Workload Management

under the six collegiate competencies, the proposed pilot competencies from ICAO / EASA/ CBTA, evidence-based SPI’s (Table 1).

The suggested training concept shift under EBT should be regarded not as a mere replacement set of the outdated set of critical events but rather as the mean for developing and assessing crew performance by using human factors aspects and well-defined Safety Performance Indicators (SPI’s). The thematic analysis for identifying evidence-based Safety Performance Indicators in Avia- tion Training and qualitative analysis of the collegiate competencies prove the dominance of human factors and ergonomics opposing the technical aspects.

To conclude, a complex learning training program should hone the more significant cognitive areas (i.e., integrated objectives) and simultaneously sup- port nonrecurrent aspects’ schematic structure and recurrent aspects’ rule automation (Harris, 2011).

CONCLUSION

Safety performance measures the ability to manage safety risk in training effectively. This review assesses the selection and implementation of Safety

Performance Indicators (SPIs) in a Safety Management System and evaluates the perceptions and satisfaction of pilots at the selected case study of Pur- due University School of Aviation and Transportation Technology (SATT). In order to measure safety risk in training, an accurate system model is neces- sary utilizing data-driven SPIs. Results of this study propose recommended practices for SPI’s amalgamation at the Purdue University - SATT, and ada- ptation policies of how other ATOs might follow to enhance the role of SPIs. SATT uses leading and lagging SPI’s as advanced markers of evidence (EBT) for selected types of incidences/accidents or locations affecting the training process.

Furthermore, leading SPIs are used to identify, monitor, and evaluate condi- tions that cause or contribute to a specific training outcome – student learning objective (SLO). Finally, the leading SPIs with the Competency-Based Trai- ning Assessment under ML/DL software are researched by Purdue University SATT faculty. Machine Learning (ML) is a system’s capability to learn from a set of data rather than software elaboration in a critical decision path. Despite the many challenges concerning the trustworthiness of Machine Learning / Deep Learning (DL) software and its effects on aviation training – safety, this could be considered a significant opportunity for the aviation industry to shorten development cycles (EASA, 2020).

ACKNOWLEDGMENT

The authors thank faculty members of Purdue University for their invaluable feedback contributing to this work.

REFERENCES

Armstrong, P. (2010). Bloom’s Taxonomy. Vanderbilt University Center for Teaching.

Retrieved 03.02.2022 from https://cft.vanderbilt.edu/guides-sub-pages/blooms- taxonomy/. Coolican, H. (2019). Research methods and statistics in psychology. London; New York: Routledge, Taylor & Francis Group European Aviation Safety Agency (EASA). (2020). Roadmap, A. I.: A human-centric

approach to AI in aviation. Harris, D. (2011). Human performance on the flight deck. England: Ashgate International Air Transport Association (IATA). (2022). Aircraft Technology Roadmap. International Civil Aviation Organization (ICAO). (nd). https://www.icao.int/safety/

OPS/OPS-Normal/Pages/CBTA-Instructional-system-design.aspx International Civil Aviation Organization (ICAO) Annex 19. (2016). Safety Management. Montreal: ICAO International Civil Aviation Organization (ICAO) Doc 9683. (2006). Human Factors

Training Manual. Montreal: ICAO International Civil Aviation Organization (ICAO) Doc 9859. (2018). Safety Management Manual. Montreal: ICAO Keller, J., Mendonca, F., Cutter, J., Suckow, M., & Dillman, B. (2020). Justifica-

tion and development of competencies to transform a collegiate aviation flight program. The Journal of Competency-Based Education, 5(3).

Selection and Implementation of Evidence-Based Safety Performance Indicators 69

King, N., and Horrocks, C. (2010). Interviews in Qualitative Research. London: SAGE Maurino, D. (2017). Why SMS: an introduction and overview of safety manage-

ment systems. International Transport Forum Discussion Paper, No. 2017-16, Paris: International Transport Forum, Organization for Economic Co-operation and Development (OECD) Merriam, S. B. (2015). Qualitative Research: A Guide to Design and Implementation.

San Francisco, California: Jossey Bass Saunders, M., Lewis, P. and Thornhill, A. (2019). Research methods for business students. New York: Pearson Skybrary.aero Airbus (n.d.) Flight Operations Briefing Notes: Standard Operating

Procedures Operations Golden Rules. [online]. available from <https://www.skyb rary.aero/bookshelf/books/189.pdf> [20 August 2020] Tracy, B. (2013). Negotiation. New York: American Management Association Yin, R. K. (2014). Case Study Research: Design and Methods. Los Angeles: Sage

Cover of the proceedings volume Safety Management and Human Factors

Explore the complete digital volume

Safety Management and Human Factors

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-40-7

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →