Dimitrios Ziakkas, Linda Tomko, Dimitra Eirini Synodinou (2024). Cognitive Systems Challenges of Virtual Reality (VR) and Simulated Air Traffic Control Environment (SATCE) in Flight Training: The Purdue Case Study. In: Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems. AHFE Open Access, vol. 119, pp. 99–109. https://doi.org/10.54941/ahfe1004473
Adaptive training capabilities based on AI can provide learners with a personalized learning path. It is a capability that customizes the trainee's learning experience to their identified learning preference while providing the quickest route through the pilot training program. To accomplish this, the training design and process is supported by cognitive theories, providing a succession of contextualized recommendations based on the training program goals and learner performance. The aviation industry seeks novel methods for pilot training that are more efficient. Competency-Based Training and Assessment (CBTA) is a method that proposes an assessment process to help understand how a flight crew manages both foreseen and unforeseen incidents, and uses this data to help the crew achieve a higher level of efficiency and performance. By training pilots in a virtual environment, instructors introduce evidence-based scenarios testing the pilot's performance while collecting relevant data. Biometric data allows for accurate training and assessment of pilot behaviors and performance parameters in competencies like, but not limited to, application of procedures, proper use of automation, manual flying, communication, workload management, situation awareness, decision making, and resilience. Considering communication competencies from a training perspective, AI (Virtual Reality - Simulated Air Traffic Control Environment; VR-SATCE) would allow pilots to improve their communication skills, enable pilots to ask questions with a specifically trained Generative Pre-Trained Transformer (GPT) model, and receive a validated answer. The virtual instructor updates the training scenarios in real-time and corrects the trainees instantly during the training session – in the same or better and safer way an experienced Type Rating Instructor would. Moreover, the same AI crewmember – a virtual instructor – can also function as an uncooperative co-pilot, which will enhance the student's training in managing difficult situations when lacking support from team members. The Purdue School of Aviation and Transportation Technology (SATT) case study focuses on the cognitive aspects of flight training using immersive technologies. This research aims to improve training effectiveness by incorporating immersive technologies in aviation training. Dynamic real-time visualization, automatic human (pilot) profile assessment, and adaptive training system technologies can potentially improve flight training's overall efficacy and efficiency. By using these technologies, all persons participating in flight training will obtain comprehensive insight into the participants' performance and cognitive limitations, ultimately optimizing the training lifecycle.
Flight simulators designed to offer the utmost realistic encounter endeavor to emulate the actual aircraft setting, embracing all facets of human sen- sory perception, to effectively train crew members. Visual displays and readouts demonstrate a level of responsiveness comparable to that of oper- ational cockpits, regulated by complex simulation models. The simulation is augmented by high-fidelity techniques for generating images, replicating platform motion and vibration signals, and accurately reproducing the audi- tory environment of the cockpit (Harris, 2018). Virtual Reality (VR) and Simulated Air Traffic Control Environment (SATCE) are innovative technolo- gies that have gained prominence in flight training. They offer immersive, realistic, cost-effective training experiences for pilots and air traffic con- trollers. The introduction of VR and SATCE is still under consideration in flight training without a commonly accepted certification–implementation plan (Ziakkas et al., 2023).
Virtual Reality (VR) in Flight Training (Ziakkas et al., 2023):
Immersive Cockpit Experience: VR provides an immersive cockpit expe- rience for pilots, replicating controls, displays, and instruments.
Flight Maneuver Training: Pilots can practice a wide range of flight maneuvers and procedures in a safe and controlled VR environment. This includes takeoffs, landings, stalls, emergencies, and more.
Scenario-Based Training: VR allows for scenario-based training, including simulated adverse weather conditions, system failures, and emergencies, developing decision-making and problem-solving skills.
Navigation and Route Planning: VR systems often include realistic nav- igation tools, allowing pilots to practice route planning, navigation, and instrument approaches, especially valuable for advanced avionics systems.
CRM and Communication: VR can simulate multi-crew environments, enabling pilots to practice Crew Resource Management (CRM) and effective communication within the cockpit.
Situational Awareness: VR enhances pilots’ situational awareness by offer- ing a 360-degree view of the virtual environment. This is particularly useful for spatial awareness and navigation training.
Safety and Emergency Procedures: Pilots can rehearse safety and emer- gency procedures, such as engine failures, fire drills, and evacuations, in a realistic virtual environment without risk to life or property.
Simulated Air Traffic Control Environment (SATCE) in Flight Training (Ziakkas et al., 2023):
Realistic ATC Interaction: SATCE replicates a realistic ATC environment, allowing pilots to interact with virtual ATC personnel.
Traffic and Airspace Management: SATCE can simulate various traffic scenarios, airspaces, and ATC instructions, helping pilots develop skills for controlled and uncontrolled airspace operations.
Multi-Aircraft Scenarios: SATCE can introduce multiple aircraft into the training scenarios, enabling pilots to practice flying in busy airspace and dealing with traffic separation and sequencing.
Cognitive Systems Challenges 101
Emergency and Diversion Procedures: Pilots can practice emergency response procedures and diversions based on instructions from virtual ATC, enhancing their ability to manage unforeseen situations.
ATC Phraseology and Communication: SATCE reinforces proper ATC phraseology, improving communication skills and reducing the likelihood of misunderstandings between pilots and controllers.
ATC Center and Approach/Departure Control Training: SATCE can sim- ulate different levels of ATC service, including en-route center control and approach/departure control.
Closed-loop adaptive training is a training approach in which the curricu- lum is dynamically adjusted depending on flight operation and training data. This adaptation occurs iteratively within the entire operation and training ecosystem, including both collective and individual levels of performance (Kaber, 2011). Meanwhile, the Threat and Error Management framework increases safety by identifying behaviors that can contribute to undesirable states. These necessitate a method for evaluating both technical and non- technical skills with reliability (Wäfler et al., 2021). With a data capture process centered on the pilot’s information, a pilot profile can be created to provide personalized training and advanced insight into the pilot’s learn- ing experience. By combining multiple data sources, such as flight telemetry, biometry, psychometry, flight history, learner activity, and demographic infor- mation, AI inference algorithms can create a digital twin of the pilot. Diverse profile characteristics, such as performance profile, cognitive-behavioral ten- dencies, learning preference, and aptitude profile, can provide a 360-degree view of the pilot to personalize training with highly valuable information (Rostami et al., 2023). This optimizes time and resources, allowing pilots to be trained more quickly and for less cost. Instructors can be equipped with data-driven, actionable intelligence regarding student behavior. New tech- nologies like VR training combined with biometric data like eye-tracking and facial tracking can be a powerful platform to obtain the required learner profiling dataset (Oberhauser, 2018).
Cognitive systems play a crucial role in aviation training, as they are central to how pilots process information, make decisions, and execute tasks while operating aircraft. The integration of cognitive systems in aviation training aims to enhance pilot performance, situational awareness, and safety. The Purdue SATT research team reviewed the following key aspects of cognitive systems for the implementation of immersive technologies in aviation training (Harris, 2018):
Cognitive Load Management: Cognitive load can be managed by opti- mizing the organization of information, procedures, and cockpit design to reduce mental workload experienced during flight operations.
Situation Awareness (SA): Training focuses on developing and maintaining SA, which is the pilot’s perception and understanding of the current state of the aircraft and its environment. Cognitive systems training helps pilots gather, process, and prioritize information effectively to maintain SA.
Decision-Making Skills: Cognitive training programs teach pilots how to make effective decisions, especially in high-stress and time-critical
situations. This includes understanding the decision-making process, rec- ognizing biases, and considering all available information.
Problem-Solving Abilities: Pilots are trained to develop strong problem- solving abilities, which are essential for resolving unexpected situations.
Crisis Management: Cognitive training prepares pilots to handle emer- gencies and abnormal situations by applying effective cognitive strategies under pressure.
Memory and Recall: Cognitive systems training helps pilots improve their memory and recall abilities, ensuring they can remember and apply critical procedures, checklists, and information as needed.
Human Factors (HF) and Crew Resource Management (CRM): Training programs incorporate HF principles and CRM to enhance communica- tion, teamwork, and coordination among flight crews.
Information Processing: Pilots are trained to efficiently process, under- stand, and prioritize vast amounts of data from instruments, sensors, and communication channels for decision making.
Attention Management: Cognitive training teaches pilots how to manage their attention effectively. This includes monitoring critical instruments, maintaining vigilance, and avoiding distractions.
Mental Resilience: Cognitive systems training enhances mental resilience, helping pilots remain focused, calm, and adaptive in challenging situa- tions.
Simulation Training: Flight simulators are used to replicate real-world sce- narios, allowing pilots to practice and apply cognitive skills in a controlled environment.
Continuous Learning and Improvement: Training is an ongoing process that involves continuous learning and improvement. Cognitive systems in aviation training evolve to incorporate new technology and methods.
By integrating cognitive systems into aviation programs, pilot competen- cies are enhanced, and the ability to manage complex situations and make informed decisions are improved. Competency is demonstrated and predicted by behaviors that effectively utilize the essential knowledge, abilities, and attitudes to execute activities or tasks within predetermined circumstances (Ziakkas et al., 2023).
METHODOLOGY
Although there are many valid models appropriate for developing sys- tems, the Purdue SATT research methodology follows International Civil Aviation Organization’s (ICAO; 2015) recommended ADDIE instructional system design model (Branch, 2009). The recommended model provides a structured framework for designing and developing a training system proto- type and final product through a step-by-step process considering multiple factors.
The analysis step assesses cognitive–immersive systems strengths, weak- nesses, opportunities, and threats. This includes the VR–SATCE training needs analysis in relation to cognitive targets prior to project design and
Cognitive Systems Challenges 103
development. For the design phase, in conjunction with the ADDIE frame- work, this research project utilizes the British Design Council’s Framework for Innovation Double-Diamond approach, illustrated in Figure 1 (Design Council, 2023; Mortati, 2015). The Double-Diamond approach employs a four-step framework to guide designers to approach a problem from a wide and in-depth perspective (i.e., divergent thinking) initially and then converge to provide a specific solution to meet design needs.
Figure 1: Double-diamond approach in flight training (Design Council, 2023; Mortati, 2015).
The development stage includes the VR–SATCE implementation into the existing training syllabi, and then reassessing gaps and opportunities during implementation of the immersive technologies, through the lens of cognitive theories. The last step will evaluate the success and projected impacts of the cognitive systems – immersive technologies in aviation training with regards to competency-based training and assessment (CBTA).
Figure 2: Adapted ADDIE approach (ICAO, 2015).
The purpose of this project is to explore a human-centered design approach for the implementation of cognitive VR-SATCE systems in the aviation train- ing ecosystem. Figure 2 provides an illustration of the application of cognitive systems VR-SATCE and CBTA in the Purdue flight training design case study.
Purdue SATT AT-48700 Transport Aircraft Simulation is a short (8-week) professional flight training course. Students in the course are expected to
demonstrate familiarization proficiency in normal and non-normal cockpit operations, including aircraft and system operations, flying procedures, and crew coordination and resource management. Students practice their knowl- edge, skills, and attitudes, flying the Purdue SATT Boeing - 737 and Airbus - 320 MPS Level 5 Fixed-Base Flight Training Devices. Anecdotally, many students in past offerings have been overwhelmed with the pace of the course given the novel environment of a complex commercial air transport flight deck.
This study is testing a supplement to the course, offering students the opportunity to gain early familiarization with flight deck displays, controls, and procedures before needing to rapidly demonstrate proficiency during the course simulator events. Traditionally, professional pilot trainees gain this familiarization using static manuals and “chair flying” with static posters or flat panel pictures of flight deck displays and controls. Virtual reality and AI-enhanced ATC simulation technology may afford greater realism and task immersion, and provide opportunities for dynamic and responsive just-in-time guided practice.
The Purdue research team aims to assess whether additional familiariza- tion exposure and practice improves professional flight students’ experience and performance during subsequent simulator events in the training course. We are surveying trainee reactions (e.g., perceived effectiveness, useful- ness, and satisfaction) and self-ratings of performance (e.g., confidence in knowledge and skill), and make behavioral observations during simulation exercises. We further aim to compare whether VR-facilitated familiarization training confers learning advantages beyond traditional familiarization meth- ods (e.g., “chair flying”or interacting with flight procedure training software via a tablet display without the immersive headset). Regardless of the method, we expect that student experience and performance during course simulator events will benefit from the additional familiarization exposure and enable more effective use of class time. However, we hypothesize that students in the immersive VR condition may experience greater learning efficiency.
Finally, the Purdue research team is offering an additional cockpit pro- cedure practice session to flight students within the same high-fidelity simulator(s) used throughout the course, but with a more realistic ATC communications environment. We are comparing student reactions and per- formance under AI-generated ATC communications to that of pseudo-ATC communication provided ad hoc by human instructors. This provides an opportunity for student volunteers to practice their new skills while expe- riencing realistic ATC communication. It will also allow us to evaluate the effectiveness of our existing professional flight training course design and instructional strategy, and whether these would benefit from incorporating VR and AI-enhanced training technologies.
ANALYSIS The primary focus of this study revolves around the application of a case study method, which is a widely used procedure in the realm of social sci- ences (Yin, 2018) and can also be used in aviation training (Yi and Lutte,
Cognitive Systems Challenges 105
2008). The aforementioned methodology is well esteemed for its capacity to offer an in-depth analysis of a certain study issue or institution. The incor- poration of a case study approach provides a substantial framework within an interpretivist paradigm, facilitating researchers in obtaining qualitative insights from participants (Quinlan, 2019). This study utilizes a case study methodology to analyze flight training within the Purdue SATT MPS A320 simulator device, specifically focusing on the utilization of SATCE and related cognitive theories in aviation training. The Purdue research team is collecting and analyzing the main data for this project utilizing ASTi SERA (Advanced Simulation Technologies, Inc. - Simulated Environment for Realistic ATC) SATCE cloud capabilities. The determination of study areas for examina- tion was conducted by subject matter experts. A comprehensive review of relevant literature resulted in the identification of primary areas for study and analysis. These focus areas include the evaluation of communication competency-based training, the understanding of SATCE, the examination of an AI human training systems integration technological roadmap, and the exploration of training using VR technology. In addition to performance observation and assessment and survey methods, we are also conducting focus groups and semi-structured interviews. This latter approach effectively balances the facilitation of data gathering while also allowing study partici- pants to freely express their thoughts and opinions (Bryman and Bell, 2015). The VR training will focus on conducting procedures (flows), the application of skills that reduce workload and positively affect communication, SA, and decision-making competencies.
Cognitive research in aviation communication explores how pilots and air traffic controllers acquire, process, and exchange information to ensure safe and efficient flight operations. This research is critical for understand- ing aviation communication’s cognitive aspects and improving communica- tion processes’ effectiveness. The Purdue research team identified, reviewed and analyzed the following critical areas of cognitive research in aviation communication:
1. Communication in High-Workload Environments: How pilots and con- trollers manage communication in high-stress, high-workload situa- tions. Understanding how cognitive resources are allocated in these scenarios is essential for optimizing communication protocols. 2. Cognitive Workload and Cognitive Load: How the cognitive workload associated with managing aircraft systems, navigation, and situational awareness affects communication. This research helps identify strategies to reduce cognitive load and ensure clear, effective communication. 3. Language and phraseology: The effectiveness of aviation phraseology and language in conveying critical information accurately and effi- ciently. How pilots and controllers process and interpret these linguistic elements. 4. Situational Awareness (SA) and Communication: How SA influences communication. Maintaining SA is crucial for understanding the con- text of communication, and research explores how effective SA supports clear and relevant exchanges.
5. Cognitive Decision-Making in Communication: Cognitive processes involved in decision-making during communication. This includes studying how information is gathered, processed, and used in decision- making regarding aircraft control, route changes, and emergency pro- cedures. 6. Cognitive Errors and Miscommunications: Common cognitive errors and miscommunications in aviation. Understanding the sources of these errors helps design training programs and communication protocols to reduce instances of misunderstandings. 7. Voice Communication vs. Data Communication: With the growing use of data link communication in aviation, cognitive research com- pares voice and data communication to evaluate their effectiveness and identify cognitive considerations associated with both modes of communication. 8. Crew Resource Management (CRM): Cognitive aspects of team com- munication, coordination, and decision-making among flight crews. How CRM improves teamwork and communication within the cockpit. 9. Human Factors and Communication: How cognitive human factors, such as perception, memory, and attention, influence communication effectiveness. 10. Multimodal Communication: Use of multiple communication chan- nels, including visual and auditory cues, to enhance communication in complex aviation environments. 11. Training and Simulation: How cognitive research informs the develop- ment of training and simulation programs that help aviation profession- als improve communication skills, reduce cognitive errors, and enhance safety.
FINDINGS The findings from cognitive research in aviation communication contribute to developing guidelines, protocols, and training procedures that optimize communication within the aviation industry. These insights help reduce the risk of miscommunication, errors, and accidents, ultimately enhancing the safety and efficiency of flight operations following ICAO standard phrase- ology. The existing approach to ATC training in simulators is marked by inherent limitations and the potential for substantial variation in the caliber and uniformity of instruction attributable to disparities in instructors’ ATC proficiency levels. The instructor is confronted with an augmented responsi- bility due to the introduction of Line-Oriented Flight Training (LOFT), which emphasizes crew resource management (CRM). A LOFT session’s purpose is to simulate an airline’s routine operational activities that are encountered reg- ularly. However, these sessions also focus on atypical scenarios that require effective communication, efficient management, and strong leadership skills. Therefore, it is imperative for the curriculum development team account- able for a certain airline to allocate significant resources in order to generate ATC scripts that are accurate and customized to suit each distinct situation.
Cognitive Systems Challenges 107
Accomplishing this presents a significant obstacle, especially considering the limited timeframe of a 4-hour simulation session.
This study investigated the opportunities and challenges involved in using immersive technology within the aviation industry. The industry acknowl- edged significant restrictions on the role-playing instructor and the absence of other traffic in advanced flight simulators, as outlined in the 2015 edi- tion of ICAO’s Manual of Criteria for the Qualification of Flight Simulation Training Devices. The document delineates the inclusion of “Environment ATC” as a desirable characteristic in flight simulators and aligns training goals that necessitate ATC and other traffic with the related qualifications.
ASTi SERA can be used to provide realistic and interactive communi- cation experiences for ATC and pilot training. This technology recognizes and interprets spoken commands and provides responses simulating ATC, allowing pilot trainees to interact with virtual air traffic controllers. This enhances training scenarios, making them more immersive and realistic. Our research focuses on the following key aspects and applications of ASTi SERA in aviation training:
1. Realistic ATC communication 2. Natural language speech recognition 3. Enhanced scenario-based training 4. Improved situational awareness 5. Emergency/abnormal situations communication and procedures training 6. Radio discipline: phraseology and communication protocols 7. Multi-aircraft scenarios for practicing communication and coordination 8. ATC center, approach/departure, and tower control training 9. Scalable training: ability to integrate the technology into various FTDs 10. Training data analysis and performance feedback The SATCE is a notable deviation from the conventional ATC– tower framework since it necessitates a shift in viewpoint from the tower to the cockpit. The replication of the behaviors and communication exhibited by several controllers during a flight and the coordination and supervision of additional aircraft within the airspace is necessary. The level of intri- cacy exhibited in this phenomenon far exceeds the rudimentary logic trees and characteristics commonly discovered in tower simulators. Contempo- rary SATCE solutions integrate AI methodologies to emulate the cognitive processes employed by ATC operators.
CONCLUSION
Closed-loop adaptive training is a training approach in which the curriculum is dynamically adjusted depending on flight operation and training data. Both VR and SATCE provide risk-free, high-fidelity environments for pilots to gain experience, develop critical skills, and improve their overall proficiency. They are particularly valuable for initial and recurrent training, skill consolidation, and scenario-based learning.
AI-driven SATCE technology, such as ASTi SERA, overcomes problems and limitations inherent in human-provided communications instruction. ASTi SERA contributes to developing strong communication and decision- making skills, enhances situational awareness, and provides a realistic train- ing experience that prepares trainees for real-world ATC interactions and communication in complex airspaces. As these technologies continue to advance, they are likely to play an increasingly significant role in the training and development of aviation professionals.
ACKNOWLEDGMENT
The authors thank faculty members of Purdue University, HF Horizons, and the ASTi research team for their invaluable feedback contributing to this work.
REFERENCES
Branch, R. M. (2009). Instructional design: The ADDIE approach. New York, NY: Springer. https://doi.org/10.1007/978-0-387-09506-6 Bryman, A., and Bell, E., 2015. Business research methods (4th ed.). Oxford
University Press. Design Council. (2023). The Double Diamond. Retrieved from https://www.design council.org.uk/our-resources/framework-for-innovation/. Harris, D. (Ed.). (2018). Engineering Psychology and Cognitive Ergonomics: 15th
International Conference, Proceedings of HCI International (LNCS, Vol. 10906). Springer International Publishing. https://doi.org/10.1007/978-3-319-91122-9 International Civil Aviation Organization (ICAO). (2015). Manual of criteria for the qualification of flight simulation training devices: Volume I - Aeroplanes (Doc 9625) (4th ed.). https://store.icao.int/en/manual-of-criteria-for-the-qualifi cation-of-flight-simulation-training-devices-volume-i-aeroplanes-9625-1 Kaber, D. B., and Boy, G. A. (2011). Advances in cognitive ergonomics. CRC
Press/Taylor & Francis. Mortati, M. (2015). A framework for design innovation: Present and future discussions. Design Issues, 31(4), 4–16. https://doi.org/10.1162/desi_a_00347 Oberhauser, M., Dreyer, D., Braunstingl, R., and Koglbauer, I. (2018). What’s real
about virtual reality flight simulation? Aviation Psychology and Applied Human Factors, 8(1), 22–34. https://doi.org/10.1027/2192-0923/a000134 Quinlan, K. M. (2019). Emotion and moral purposes in higher education teaching: Poetic case examples of teacher experiences. Studies in Higher Education, 44(9), 1662–1675. https://doi.org/10.1080/03075079.2018.1458829 Rostami, M., Kamoonpuri, J., Pradhan, P., and Chung, J. (2023). Development
and evaluation of an enhanced virtual-reality flight simulation tool for airships. Aerospace, 10(5), 457. https://doi.org/10.3390/aerospace10050457 Wäfler, T., Gugerli, R., and Nisoli, G. (2021). Integrating Safety-II into safety management: Generalized guidelines for a Safety-II-based tool: Measure evaluation and effectiveness assessment. VDF. https://doi.org/10.3218/4023-4 Yi, R., and Lutte, R. K. (2008). Case study methodology in the aviation industry.
Journal of Aviation Education and Research, 23, 55–66. Yin, R. K. (2018). The case study handbook: A practical guide for practitioners, researchers, and students. (6th ed.). Sage Publications.
Cognitive Systems Challenges 109
Ziakkas, D., Pechlivanis, K., and Keller, J. (2023). The implementation of Artificial Intelligence (AI) in aviation collegiate education: A simple to complex approach. Intelligent Human Systems Integration (IHSI 2023): Integrating Peo- ple and Intelligent Systems, AHFE Open Access, 69. AHFE International, USA. https://doi.org/10.54941/ahfe1002863 Ziakkas, D., Sarikaya, I., and Natakusuma, H. C. (2023). EBT-CBTA in aviation
training: The Turkish Airlines case study. Engineering Psychology and Cognitive Ergonomics: HCII 2023 Lecture Notes in Computer Science, 14018, 188–199. https://doi.org/10.1007/978-3-031-35389-5_14 Ziakkas, D., and Vink, L.-S. (Eds.). (2023). Implementation guide for artificial intelligence in aviation: A human-centric guide for practitioners and organizations.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-95-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional