Dimitrios Ziakkas, Abner Del Cid Flores, Hans Cornelius Natakusuma (2023). The Implementation of Immersive Technologies - Artificial Intelligence (AI) in Aviation Collegiate Education: A Simple to Complex Approach. In: Training, Education, and Learning Sciences. AHFE Open Access, vol. 109, pp. 162–170. https://doi.org/10.54941/ahfe1003162
Aviation and air transportation have traditionally led to technological innovation. The International Air Transportation Authority (IATA) Technology Roadmap (IATA, 2019) and the European Aviation Safety Agency (EASA) Artificial Intelligence (AI) roadmap (EASA, 2020) outline and assess ongoing technological prospects that aim to change the aviation environment with the implementation of AI from the beginning of collegiate education.Rotolo (2015) suggests five characteristics to characterize immersive technology. Immersive technologies must have radical originality, fast growth, coherence, significant impact, and low uncertainty and ambiguity. Emerging technologies like Virtual Reality (VR) are notable for their influence and coherence across applications and usage. VR technology has dramatically reduced training expenses compared to other systems. Laughlin (2008) reported that remote learning and practicing basic flying abilities could lower new pilot training expenditures by as much as 70%. Due to the COVID-19 pandemic, flight training organizations must develop new ways to teach their pupils. Distance pilot-to-student education is one. The Federal Aviation Administration (FAA) (2020) is progressively accepting non-traditional methods for regulatory training. AI, VR, AR, and MR can improve flight operation training. Human System Integration (HSI) specialists use systems engineering methods and procedures to achieve successful HSI. Simple to complicated methods meet functional and non-functional requirements. The systems engineering team uses each branch to analyze collegiate aviation program requirements. Aviation training changes will impact humans' performance and ability to make decisions. The research was thematically selected on immersive technologies implementation in collegiate aviation trainees' perception. The study was structured based on an analysis of the available literature concerning the current uses of immersive technologies - AI in aviation. The findings were reviewed and evaluated concerning the appropriateness of the implementation of immersive technologies in aviation training syllabus and the notable differences between the levels of technology.
The Implementation of Immersive Technologies - Artificial Intelligence (AI) 163
INTRODUCTION
The current and future use of immersive technologies, specifically VR, for training purposes in aviation is promising. For example, the US Air Force announced that “experimental virtual reality fighter pilot training works best for students who want to fly the service’s most advanced stealth plat- forms” (USAF, 2021). The US Army determined that low-cost, low-fidelity commercial-off-the-shelf virtual reality (VR) technologies could be used effe- ctively to train pilots in the “Aviator Training Next” experiment (ARMY AVIATION, 2020). Several airlines worldwide have announced using virtual reality technology for training purposes.
VR immerses oneself in a computer-generated environment that includes visual, auditory, and optionally haptic representations of the environment. This environment, which could be a room, landscape, or cockpit, could be shown on a screen or head-mounted display (headset). For example, the user may interact with the environment through gestures or physical buttons, or levers.
Because subjects can be taught in ways that reading a textbook or listening to a lecturer cannot, virtual reality offers new opportunities to significantly improve teaching and learning. One of the most critical aspects of learning is knowledge retention, or the ability to recall what has been learned (Diamond, 1989). This is especially true for flight crew, who must remember details about aircraft systems, procedures, and rules. According to research, VR- assisted learning can increase knowledge retention by up to 400%, thereby increasing the value of study time. Better knowledge retention results in less retraining and better pilots who remember the emergency checklist better due to VR-assisted learning.
According to Edgar Dale’s Cone of Experience and the technology pyramid of the Purdue Virtual Reality Research Laboratory, students remember only 10% of what they read but 90% of what they do (Dale, 1969). This proves the adage “you learn by doing” to be true. Purdue SATT proposed aviation pro- gram advocates the use of immersive technologies - AI to a four-step simple to complex collegiate instructional design:
1. Identify outreach, personnel recruitment, and pilot selection tasks. 2. Introducing newly enrolled students to the PFP (Professional Flight Program). 3. Additional training in addition to fundamental and advanced jet maneu- vers. 4. Research aimed at mastering pilot competencies through immersive technologies - AI implementation, increasing student self-efficacy, and reducing crew training delays.
In terms of knowledge and skills, the project team has an extensive safety risk assessment and monitoring experience accumulated over the last four decades. Purdue University is leading the implementation of immer- sive technologies (VR -SATCE) in aviation training (basic pilot certification, aircraft/rotorcraft type ratings, general aviation, rotorcraft, powered-lift, and advanced air mobility pilot training) with a particular emphasis on leveraging
research results and effectively developing standards and guidance to certify the use of new immersive flight simulation technologies for pilot training with National Test Pilot School, (NTPS). Purdue SATT research team has the technological capacity, methodology, and plan to establish a standard set of research goals and collaborate with FAA partners to evaluate the perfor- mance of immersive flight simulation technologies and implement them in collegiate aviation training.
IMMERSIVE TECHNOLOGIES SELECTION IN AVIATION TRAINING Immersive technologies – AI are being used in many different ways. In in- flight training, one of the most important things to learn is how to judge distances correctly. This is called “depth perception.” A pilot with much expe- rience may be better at judging distances because they have done it so many times before. A new pilot student does not have this experience yet, so they must get it by flying. This part of flight training is hard to practice in traditio- nal simulators due to space perception. This is because the typical screen onto which the outside world is projected is placed at a fixed distance from the eyes of the student pilot. Every object projected onto the screen would appear at the same distance from the pilot, whether it is the runway 5 meters away or a tower 5 kilometers away. VR goggles have stereoscopic screens showing the same scene in slightly different ways. This gives a sense of depth and distance like our natural, stereoscopic vision (our two eyes) lets us judge distance. So, VR can accurately and naturally show distances in a flight simulator.
Figure 1 and Figure 2 show the difference between how a curved screen, like the kind used in most traditional flight simulators, and VR goggles make students feel about distance. On a curved screen, three targets marked with red, green, and blue Xs are shown at the same distance, but in VR, they appear to be at the right distance from the viewer. A proper lookout is another essential skill that every pilot must learn.
A proper lookout is another important skill that every pilot must learn. The lookout must be done when making a landing circuit or keeping an eye out for traffic. Most traditional flight simulators only let pilots see up to 180 degrees (Figure 3), which makes it very hard to do a good lookout. Pilots under training on these simulators often have to use different ways to find their way around, like timing their turns, because they cannot use the lookout procedures they would use in a real plane. The student pilot can look in any direction with VR goggles (Figure 4). This means that the student can practice a lookout the way he or she would in a real plane.
Figure 1: Perceived distance of three targets (red, green, blue) seen on a curved screen.
The Implementation of Immersive Technologies - Artificial Intelligence (AI) 165
Figure 2: Perceived distance of three targets (red, green, blue) seen through VR goggles.Figure 3: Field of view on a curved screen.Figure 4: Field of view with VR goggles.
Even though Virtual Reality pilot training has become very popular in recent years because it allows pilots to interact through sight and sound, it could not give the same level of immersion through touch. This is because the
VR touch controllers, the current state-of-the-art solution, stop users from using their fingers naturally and intuitively (Figure 5). The problem that has been found is that touch sense needs to be integrated with the human hand, which is a very complex system. The hand has 16 joints that give it 27
of freedom, which lets the fingers and wrist do every possible movement and motion. Typical VR touch controllers have a single joint with only 6
of freedom (Figure 6).
Purdue SATT and aviation industry immersive technologies partners (VR Pilot – Magos) focused on implementing a solution to simulate the touch sensation inside a virtual cockpit. Purdue research team uses Magos Virtual Reality Pilot Training (MVRPT) solution. It is a mix of software (VRflow from VRpilot) and hardware (Magos Gloves from the Magos team). It offers all of the above-mentioned innovative features, such as depth perception, 360-degree vision, and touch sensation, and aims to completely change the current state of the art in pilot training. MVRPT enables remote (at home) procedural training in which the trainee can interact with a digital cockpit using their fingertips while wearing Magos gloves. Additionally, the Purdue research roadmap is focused on the VR certification process with NTPS (FAA, EASA, UK CAA), the implementation of a VR training syllabus following a change management approach, and the introduction of VR standardization principles in the global VR aviation ecosystem.
Figure 5: Interaction with controllers.Figure 6: Hand degrees of freedom.
The Implementation of Immersive Technologies - Artificial Intelligence (AI) 167
THE IMMERSIVE TECHNOLOGIES - AI CONTRIBUTION TO THE COLLEGIATE TRAINING COURSE REVISION The Purdue human systems integration team is developing a research proto- col with VR PILOT - MAGOS -NTPS that could be easily incorporated into the systems engineering test plan, implement VR in aviation training globally, and evaluate the results. Initial Purdue aviation blended learning context was a Multichannel Method education program incorporating tutor-led activities, images, videos, digital tasks, flight training, simulator training, and self-study (Hillman, 2021).
From September 2021, Purdue SATT entered the hybrid stage combining new technology with old technology aiming to use “blended learning” as the instructional model for experimenting with immersive technologies (Vir- tual Reality-VR/ Mixed Reality - MR/Augmented Reality - AR, artificial intelligence instructor, SATCE, eye tracking, and monitoring capabilities). A cloud-based SATCE that delivers live ATC service from real people dramati- cally increases the impact and range of uses for SATCE. Additionally, an AI fully featured ATC environment for all phases of flight support gate-to-gate flights between any airport in the world, accurate ATC behaviors and other traffic aircraft, and IFR - VFR flight operations for fixed and rotary-wing aircraft.
The suggested aviation research philosophy adheres to the ADDIE (Analyze-Design-Develop-Implement-Evaluate) methodology, which invo- lves the implementation of immersive technologies - AI Student Learning Objectives (SLO).
Analyze
Flight school student learning objectives using the data acquired on the Evidence-Based Training (EBT) – Competency Based Training Assessment (CBTA) ICAO analysis and VR elements.
Analysis of FAA 14 CFR Part 60 Flight Simulation Training Device Ini- tial and Qualification and Use; ICAO 9625 standards and training needs. Training Needs Analysis (TNA).
TNA breaks down a complex task into more straightforward sub-tasks, each subtask with a well-defined learning objective (Honour, 2006). Specify for each subtask the skills needed to assimilate the sub-task (reproductive, decision-making, behavior, etc....…) and the minimum levels of fidelity required to reach the learning objectives (Chapanis, 1996).
Design
Learning areas of interest connected to operations using an evidence- based, CBTA - AI-based, thematic approach. Machine Learning/Deep Learning ML/DL software is now being introduced. Learning areas follow Aviation Accreditation Board International (AABI) guidelines.
Strengthen the link between training tasks and tool.
Cater for innovative training tools.
Enable better standardization of training tools used.
Place the learning objectives in scenario form and implement them.
Training syllabus implementing new technologies in aviation training: (Virtual/Mixed/Augmented Reality, artificial intelligence instructor, eye tracking, for Aviation training from Ab-initio to Type Rating and recurrent level.
Advanced Air Mobility features with aviation partners and regulators.
Develop Purdue research protocol offers VR applications for both civil and military use as the first phase of introduction in VR. Focusing on Professional flight VR training courses, SATT with VRpilot offers the following VR learning experience to Purdue collegiate students - specific training objectives that can be implemented fast:
Normal Procedures Training (A-320/B-737)
Abnormal – Emergency Procedures / Memory Items Training
Memory Training
Cockpit familiarization module
Flow visualizations
Step-by-step explanations
Multi-crew remote training
Automatic crewmember for solo training
Competency-Based Training and Assessment (CBTA) - Exam
Dynamic Pilot Flying (PF) – Pilot Monitoring (PM) task sharing
Eye tracking monitoring-Errors monitoring
Debriefing.
Implement
The content efficiency, teaching experience, and aviation market needs are implemented through the competencies and the standardization of immer- sive technologies at Purdue. SATT revised the AT-38800 course (Large Aircraft Systems) to familiarize the students with VR training courses focusing on A-320/B737.
Supplementary VR training syllabus for airliners – operators of A-320 / B-737.
Implementation of CBTA in VR training environment -TNA.
Evaluate The Purdue HSI team developed the research protocol that could be easily incorporated into the systems engineering test plan to implement AI in avia- tion training globally and evaluate the results. SATT experts established and are working with NTPS within a framework of systems engineering proces- ses and methodologies to ensure successful human systems integration. The framework was based on the simple to complex approach to meet functional and non-functional requirements:
Professional flight VR training courses using the CBTA competencies outlined in the ICAO and ABBI guidelines.
The Implementation of Immersive Technologies - Artificial Intelligence (AI) 169
New technologies in aviation training: (Virtual/Mixed/Augmented Rea- lity, artificial intelligence instructor, eye tracking, Simulator Operations Quality Assurance.
Operational -Human Factors evaluation of training with new techno- logies (VRPILOT -Magos, Purdue VRLab) Vs. traditional ways Flight Simulators – FSTD A-320/B737 in Purdue Niswonger Simulator Center.
Make use of the ML/DL capabilities of the software.
Feasibility of introducing VR training to Purdue training syllabus from the ab-initio phase.
Purdue studies have demonstrated a significant reduction in training time by utilizing immersive technologies for aviation training applications. The identified benefits are:
Reduced instructor hours
Competency-based training – simulator preparation
Training – scheduling flexibility
Improved System Knowledge
Faster Learning rate
Better knowledge retention
Reduced negative training.
CONCLUSION
The Purdue research certification roadmap is focused on the immersive technologies’ certification process with NTPS (FAA, EASA, UK CAA), imple- mentation of a revised training syllabus following a change management approach, and introduction of new technologies standardization principles in the global aviation ecosystem.
The Purdue SATT research approach provides a crew-centric, systemic, socio-technical risk model, which will support the detailed risk assessments. The baseline risk assessment relies on a combination of STAMP/STPA- Secu- rity - Cyber Security systems analysis, defining the risk parameters, leading to risk ratings, and then synthesizing the different elements of the Bow-tie model (Folds, 2008). Additionally, the workbenches include a discussion on certifi- cation of the immersive technologies following (FAA 14 CFR Part 60 Flight Simulation Training Device Initial and Qualification and Use; and ICAO 9625 Manual of Criteria for the Qualification of Flight Simulation Trai- ning Devices (FSTDs) and organizational requirements, (Competency-based Training and Assessment, CBTA).
ACKNOWLEDGMENT
The authors thank faculty members of Purdue and Coventry Universities for their invaluable feedback contributing to this work.
REFERENCES
Chapanis, A. (1996). Human factors in systems engineering. Wiley Series in Systems Engineering and Management. Andrew Sage, series editor. Hoboken, NJ: Wiley.
EASA. (2020). Artificial Intelligence Roadmap: A human-centric approach to AI in aviation. Folds, Dennis. Gardner, Douglas and Deal, Steve. (2008). Building Up to the Human
Systems Integration Demonstration, INCOSE INSIGHT Volume 11 No. 2. Hillmann, C. (2021). UX for XR: User Experience Design and Strategies for Immersive Technologies. Apress L. P. Honour, E. C. (2006) “A Practical Program of Research to Measure Systems Engi-
neering Return on Investment (SE-ROI),” proceedings of the Sixteenth Annual Symposium of the International Council on Systems Engineering, Orlando, FL. IATA. (2019). Aircraft Technology Roadmap to 2050. Laughlin, B. (2018). XR Drives Aerospace Excellence at Boeing. Manufa- cturing Engineering, p. 82. https://www.proquest.com/trade-journals/xr-drives- aerospace-excellence-at-boeing/docview/2164486002/se-2?accountid=13360 Rotolo, D., Hicks, D., & Martin, B. R. (2015). What is an emerging technology? Research Policy, 44(10), 1827–1843. https://doi.org/10.1016/j.respol.2015.06.006 Ziakkas, D., Plioutsias, A., Pechlivanis, K., 2022. Artificial Intelligence in the aviation decision-making process. The transition from extended Minimum Crew Operations to Single Pilot Operations (SiPO). Artificial Intelligence and Social Computing. AHFE (2022) International Conference. AHFE Open Access, vol. 28. AHFE International, USA. http://doi.org/10.54941/ahfe1001452
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Training, Education, and Learning Sciences
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-85-8
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional