Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Human-Centred Classification of Remote Operation Intervention Scenarios for Automated Vehicles

Centre for Future Transport and Cities, Coventry University, Coventry, CV12TT, UK 2dRISK Limited, London, WC1V6PX, UK ↗
Publication year
Digital Object Identifier10.54941/ahfe1007121

Abstract

The deployment of automated vehicles (AVs) offers substantial societal benefits but faces significant challenges, particularly in complex urban environments. Remote operations (ROs) enabled safety interventions serve as a critical intermediary, allowing human operators to intervene when AVs encounter limitations. However, to inform RO workstation design and understand human intervention capabilities, real-world scenario data is critical. To address this gap, we conducted semi-structured interviews with 13 local safety experts at the Birmingham National Exhibition Centre (NEC), UK, resulting in a catalogue of 105 functional scenarios. We identified prevalent scenario types, AV-related challenges, and scenario complexity. These findings inform RO workstation and human machine interface (HMI) design and highlight the need for real-world scenario generation to support RO capability assessment. This study contributes to a framework for enhancing human-AV interactions, understanding RO safety requirements, and advancing ROs.

INTRODUCTION

The deployment of highly automated vehicles (AVs) is broadly anticipated to yield significant benefits, notably a reduction in road accidents and improved mobility (Golbabaei et al., 2024). However, despite these advan tages, AVs continue to encounter safety-related challenges, particularly in complex urban environments (Koopman and Wagner, 2017). These chal lenges impede fully autonomous systems across diverse operational design domains (ODDs). To address these limitations, increase road safety, and progress toward wider deployment, Remote Operations (ROs) have emerged as an intermediary solution, empowering remote human operators to mon itor, assist, and regain control of AVs during situations surpassing vehicle capabilities (British Standards Institution, 2024).

However, realizing effective ROs brings to light both technological and human operator-related challenges. Technological challenges mainly involve issues related to data connectivity and communication latency (Kamtam et al., 2024). Conversely, human operator-related challenges centre on how Received September 26, 2025; Revised November 10, 2025; Accepted November 25, 2025; Available online February 1, 2026

Human-Centred Classification of Remote Operation Intervention Scenarios 629 operators interact with the AV through remote workstations and the HMI (Bogg and Birrell, 2025). Physical separation from the vehicle complicates situational awareness, workload management, and control input methods, making HMI design an important mediator of operator performance and safety (Wolf et al., 2025). Critical to remote operator-related challenges is identifying and understanding intervention scenarios (Tener and Lanir, 2024). Understanding the breadth and depth of intervention scenarios can help to understand remote operator limitations, identify tasks, as well as inform RO workstation requirements and HMI design.

Although there is an increased number of research papers regarding RO technical and human factors challenges, only a few papers specifically address the limited understanding of RO intervention scenarios. Previous RO scenario research has mainly focused on identifying situations where human intervention is needed. Early work by Kettwich et al. (2021) developed one of the first frameworks to classify such scenarios, outlining how vehicle, envi ronment, and operator factors interact during teleoperated driving. Later, Kettwich et al. (2022) expanded this approach by compiling a catalogue of 74 realistic cases drawn from control-centre observations, interviews, and L4 deployment trials. More recently, Tener and Lanir (2024) conducted expert interviews to define a comprehensive set of intervention scenarios and use cases for teleoperation. Together, these studies provide a foundation for understanding when and how remote operators assist AVs.

While previous research has advanced the identification of teleoperation intervention scenarios there remains a gap in understanding the full sce nario scope and complexity. Knowledge gained solely from remote opera tion centres, and from short-term trials may represents only a fragment of the broader picture. Remote operators often work within narrowly defined routes, time windows, and environmental conditions, limiting their exposure to rare or complex events. In contrast, domain experts, such as traffic manag ers, safety officers, and event marshals routinely handle critical and atypical traffic situations related to non-automated vehicles. Their extensive experi ence provides deeper insight into the variability and complexity of critical and edge-case scenarios that may challenge both automation and ROs.

This study integrates expert knowledge to develop a comprehensive understanding of RO intervention scenarios and their characteristics, with the aim of supporting safer and more effective workstation and HMI design. Specifically, the study seeks to identify and classify critical and edge-case scenarios and to analyse their complexity. Furthermore, the paper highlights how variability among intervention scenarios influences feasibility testing and human factors considerations in the design and evaluation of RO systems.

Although scenarios can be theoretically generated, those derived from real-world data represent a more reliable basis for accurate RO interaction modelling and offer a solid foundation to better understand the human fac tors associated with ROs (Kettwich et al., 2022). While scenarios derived from traffic accident data, naturalistic driving studies, and traffic monitoring provide valuable input, they represent only a subset of relevant situations. Many critical scenarios that are observed by drivers, traffic safety person nel, and other stakeholders can be missed with traditional data collection

methods. This limitation is particularly pronounced in restricted private areas, where incidents may not be documented by the authorities, and where public surveillance infrastructure is limited. By including knowledge elicita tion from domain experts who regularly operate in specific AVs ODDs, we can access undocumented experiential knowledge and identify edge cases potentially missed through purely data driven approaches to scenario iden tification methods. Thus, a knowledge-based approach complements data- driven methods and contributes towards a broader and more comprehensive foundation for RO implementation.

For this reason, we adopted a knowledge-based approach through a case study of the L4 shuttle bus deployment at the Birmingham National Exhibition Centre (NEC) in the UK, a private restricted area. This deploy ment is part of the Solihull and Coventry Automated Links Evolution Project (SCALE), which aims to connect the NEC with the Birmingham Business Park. To account for critical and edge case scenarios that may not be cap tured during the deployment by RO centres and safety drivers, the authors focus on interviews with domain experts and derive pertinent scenarios that can inform both RO requirements and HMI design. When domain experts identify ODD-specific risks and implement mitigations, the resulting crashes or events are avoided and therefore absent from the data, despite the risks remaining real and present. Consequently, domain expert knowledge is essen tial to capture scenarios that data-driven approaches may miss.

Scenario-Based Approaches Scenario-based approaches (SBA) have gained recognition as a reliable and widely employed methodology for validating the safety of automated driving systems (ADS) (Cai et al., 2022). SBA seeks to answer question of how to select and integrate all relevant scenarios efficiently. These approaches also extend to ROs, as both fields emphasize the identification of critical and edge case scenarios to effectively address safety challenges. In this context, critical scenarios are defined as those that could potentially lead to a collision or near collision, with severe consequences such as system failures, safety hazards, or user dissatisfaction. On the other hand, an edge case can be defined as a rare, challenging driving scenario that occurs at the boundaries of normal opera tion and could potentially expose limitations or failures in the ADS.

The identification and classification of critical scenarios serve as a crucial nexus between ADS evaluation and ROs. Both domains fundamentally rely on understanding how AVs perform in challenging situations. For ROs, the same scenarios represent potential intervention points where remote oper ators may need to assist or take control. However, the inability of the ADS to address a challenging scenario does not guarantee that a remote operator is able to manage it effectively. While some scenarios may be managed by the RO, others may only be partially addressed, and certain situations may exceed the capabilities of the remote operator altogether (Fig. 1).

By understanding the variety and complexity dimensions of these sce narios, we can provide valuable insights for both remote workstation and HMI design and contribute to scenario-based methodologies. To achieve

Human-Centred Classification of Remote Operation Intervention Scenarios 631 this the authors a) identified critical and edge case scenarios in the specified ODD, b) classified the characteristics of identified scenarios, c) proposed scenario complexity levels. This approach emphasizes understanding sce nario scope and complexity as a foundation for effective, human-centred RO design.

Figure 1: Interaction between the AV and RO scenarios.
Original PDF Figure 1:

As ROs evolve, their systems are expected to encounter an increasing num ber of edge-case scenarios. However, it is anticipated that certain edge-case scenarios will remain unpreventable or beyond current technological capa bilities. By systematically documenting these scenarios, we created a valuable reference library.

Scenario generation typically relies on two approaches: data-driven and knowledge-based (Riedmayer et al., 2020). Data-driven methods use crash data, naturalistic driving studies, or traffic monitoring can be analysed with new AI techniques, such as Conode AI used in the SCALE project. These methods underpin most contemporary SBA testing projects. In contrast, knowledge-based approaches draw on expert insight and domain knowledge to capture rare or underrepresented edge cases, improving test coverage and robustness (Nalic et al., 2020). Common sources include industry standards, guidelines, and expert assessments. Scenarios can be defined at three abstrac tion levels: functional, logical, and concrete (Menzel et al., 2019). Functional scenarios, expressed in human-readable language describe relationships among key entities. They form the foundation of this study, enabling experts to discuss, refine, and expand scenarios.

METHODOLOGY

We adopted a knowledge-based approach to scenario generation and con ducted semi-structured interviews with 13 local safety experts. These experts were grouped into semi-structured focus groups, composed from four NEC departments: traffic safety; bus drivers; event management; and safety and marshalling. The selection of these departments was made in consultation with the NEC general manager, and participants were chosen based on their experience at the NEC. Each participant had a minimum of 3 years and a maximum of 9 years of relevant experience. Notably, some participants held multiple positions or accumulated experience across different roles in the past.

Interviews were conducted on-site at the NEC, guided by a pre-developed interview structure. This structure focused on eliciting information about scenarios related to a) animals or objects, b) interactions with traffic par ticipants and their behaviour, c) weather and road conditions, and d) other observed events that could pose challenges for AVs. Interviews were divided into two sessions (seven and six participants respectively), each lasting approximately 60 minutes. Sessions were audio-recorded, transcribed, and analysed by the first author to identify critical scenarios, characterize sce nario types, and address AV challenges.

Data Acquisition and Processing To provide essential context, two presentations were delivered prior to interviews:

A) Introduction to the SCALE Project: Participants received an overview of the staged AV deployment. A B5-sized NEC map showing the intended route was distributed, ensuring shared understanding of the vehicle’s ODD and deployment context. B) Overview of AV Risks and Challenges: A presentation addressed

AV-related risks, challenges, and edge cases encountered regionally and internationally, followed by a Q&A session.

Participants received a B5-sized route map and post-it notes to a) verbally describe scenarios to the researcher; b) write brief descriptions on post-it notes; and c) place notes on the map indicating scenario locations.

The first author transcribed interviews. Data processing involved: 1) iden tifying and extracting scenario descriptions; 2) matching post-it descriptions to verbal accounts; 3) pinpointing scenario locations; and 4) extracting ODD information (e.g., road geometry, speed limits).

Data processing focused on extracting detailed information regarding both scenario descriptions and their locations. Scenario descriptions cap tured entity relationships, including sequences of events and the behaviour of both traffic participants and vehicles. Scenario locations provided critical insights into the ODD, including road geometry, speed limits, and lane mark ings. The data gathered from the interviews, post-it notes, and maps were integrated to create a detailed functional scenario (Fig. 2).

Assumed scenario location

Scenario description: - A stag dashes out from the woods onto the road.

“Stag on road”

Road geometry/ODD - Straight road, two-way street. - Speed limit 15 mph. - Inter urban area.

NEC MAP

Scenario description “A stag sometimes jumps over the fence and dashes onto the road.”

Post it note with scenario description and location.

Figure 2: Example of the FS and the mapping process.
Original PDF Figure 2:

Research validity was ensured through methodological triangulation com bining verbal descriptions, written notes, and spatial mapping along the NEC route. Structured data collection and collaborative cross-checking enhanced internal validity, while expert sampling from four NEC departments (traffic safety, driving, event management, and marshalling) with 3–9 years of expe rience ensured comprehensive domain coverage.

Data Analysis The extracted scenarios were organized into a Functional Scenario catalogue, incorporating essential information for further analysis, such as a) actor type, b) actor details, c) actor condition, d) scenario description, e) road geometry and ODD characteristics, f) challenges to the AV, and g) any additional rele vant information.

We classified the data further based on challenges faced by the AV, identi fying three categories: a) perception, b) manoeuvre, and c) decision-related challenges which align closely with components of the dynamic driving task (DDT) (SAE, 2021). Furthermore, it also conceptually aligns with Rasmussen’s Skill-Rule-Knowledge (SRK) framework (Rasmussen, 2012), where manoeuvre-related actions correspond to skill-based control, rou tine operational responses reflect rule-based behaviour, while complex deci sion-making under uncertainty engages knowledge-based reasoning.

Perception-related scenarios involve situations where the AV may struggle to accurately detect an object, while manoeuvre-related scenarios pertain to the AV’s movement or responses to surrounding conditions. Decision-related scenarios focus on strategic route planning and associated decision-making processes. Additionally, we varied scenario complexity, with some presenting with a singular challenge (e.g., a perception-related challenge), while others entailed multiple challenges. To account for this, we propose three complex ity levels: Level 1) a consisting of one challenge to the AV of either Perception (P), Manoeuvre (M), or Decision making (D), Level 2) a combination of two challenges, and Level 3) a combination of three challenges. L2 and 3 with two or more challenges are described as complex scenarios (Fig. 3).

Figure 3: Scenario complexity levels.
Original PDF Figure 3:

RESULTS AND DISCUSSIONS The data analysis resulted with the generation of 105 Functional Scenarios. The results focus on disturbance and AV challenge types, and scenario complexity.

Scenario Types and Sources of Disturbance Scenario types and disturbance sources inform HMI, remote workstation requirements, and operator training by indicating task types (e.g., assistance or driving). Four scenario types were identified: disturbances caused by ani mals, objects, other traffic participants, and “local control” (a specific cate gory encompassing scenarios from NEC management and safety personnel interactions with AVs).

Animal scenarios include dynamic interactions (SC 7: “A deer jumped over the fence and dashed onto the road, requiring an evasive manoeuvre”) and static (SC 6: “An unattended horse standing on the road”). Object sce narios involved dynamic objects (SC 12: “The barrier was dragged down the road due to strong winds”) and static (SC 27: “Unattended carpet rolls blocking the roadway”). Traffic participant scenarios included dynamic (SC 40: “Foreign drivers entering the roundabout on the wrong side”) and static scenarios (“A vehicle drove through the centre of the roundabout and got partly stuck, creating congestion”). Table 1 shows an overview of scenario types and actors involved. Disturbances were organized using existing inter national functional scenario catalogues. However, local control emerged as a novel disturbance source, reflecting unique NEC ODD characteristics.

Table 1: Scenario types according to sources of disturbance.

Table available in the original PDF paper.

Scenario Type Disturbance/Actor Actor Variety Scenario Example Animal Squirrel, Rabbit, Horse, Stag, etc...

8 An unattended horse in the middle of the road.

Object Wood, Road sign, barrier, boxes, etc...

22 Delivery boxes left in the middle of the road.

Traffic paiticipant

16 Pedestrian running over the street at non-crossing area.

Pedestrian, Car, Bus, Tractor, E-scooter etc...

Local control

Control centre and Control staff.

2 Control centre issuing a suspension of all service.

Human-Centred Classification of Remote Operation Intervention Scenarios 635 The local control scenarios are initiated by the NEC authority. Depending on the event and NEC needs, the control centre frequently modifies road geometry, alters traffic flow direction, opens or closes routes, and, in some cases, can suspend operations entirely as security measures during high-pro file events (e.g. SC 101 “The arrival of a royal family member at Birmingham Airport initiates a suspension of all transport service”). This exemplifies the challenge of direct communication between local control and AVs. The lack of direct communication between local control and AVs, forces AVs or ROs to independently interpret custom signage, temporary regulations, and rout ing changes.

These findings indicate that remote operator tasks should reflect the diversity of real-world ODD scenarios, enabling smooth transitions between monitoring, assistance, and intervention. Flexible task allocation, adap tive interfaces, and scenario-based training are essential to support vary ing engagement levels and prepare operators for diverse disturbances. Disturbance types also imply specific HMI needs (e.g. enhanced visual over lays for object and animal scenarios or dynamic prompts for local control events), directly informing RO system design.

AV Challenge Types and Scenario Complexity The study classified potential AV challenges into three categories: Perception (P); Manoeuvre (M); and Decision-making (D). Scenario complexity was assessed by how challenges combined, revealing distinct AV difficulty patterns.

Perception and Manoeuvre (PxM) are the most frequent combinations, involving objects obstructing the AV’s path (e.g., SC 27: “an unattended car pet roll on the roadway”). The AV must perceive the object and execute a safe manoeuvre to avoid collision. These combinations suggest interface enhancements like trajectory suggestions, proximity alerts, and semi-auto mated manoeuvring assistance.

Perception and Decision-Making (PxD) scenarios involve local control interactions requiring the AV to interpret traffic directives, temporary signs, or verbal instructions from NEC personnel. For example, SC 96: “When significant congestion occurs, the NEC directs vehicles through car parks via staff and traffic signs to create a longer queue.” The AV must perceive staff and adapted signs indicating unexpected traffic flow changes and adapt its routing strategy.

Perception, Manoeuvre, and Decision-Making (PxMxD) scenarios involve unpredictable traffic participant behaviour, such as SC 80: “a forklift moving against traffic flow on the wrong side of the road”. The AV must detect and manoeuvre around the vehicle while making informed decisions regarding lane usage and adaptation to surrounding traffic. PxMxD combinations sug gest workstations should integrate multimodal feedback (visual, audio, hap tic) to support rapid, multi-stream operator decision-making.

Three scenario complexity levels were defined. Level 1 (Single-Challenge, 14 cases) involved one challenge (perception, manoeuvre, or decision-making). Level 2 (Dual-Challenge, 59 cases) combined two challenges, most com monly perception and manoeuvre (PxM, 42 cases). Level 3 (Multi-Challenge,

28 cases) involved multi-actor interactions and dynamic NEC control scenar ios, requiring high situational awareness (SA) and adaptive decisions from AVs and ROs. Scenario complexity depended on challenge type and com bination. Different mixes of perception, manoeuvre, and decision-making demands shaped difficulty and corresponding operator tasks. Table 2 sum marizes challenge types, combinations, and prevalence.

Table 2: Challenge type and scenario complexity.

Table available in the original PDF paper.

Av Challenge Type

Number of Scenarios

Occurring Scenario Types

Scenario Complexity Level Level 1 (18)

Perception (P) 7 Mainly traffic participant and object.

Manoeuvre (M) 4 Traffic participant.

Decision (D) 7 NEC control.

Level 2 (59)

P × M 42 Mainly object and traffic participant.

P × D 15 Mainly NEC control and traffic participant.

M × D 2 Traffic participant and object.

P × M × D 28 Mainly traffic participant. Level 3 (28) Total 105

Challenge combinations directly impact operator workload and SA. Complex scenarios (Levels 2 and 3) likely impose higher cognitive and tem poral demands, requiring sustained attention, rapid information integration, and coordination with automation and the RO team. Elevated workload can reduce SA, causing delayed interventions or errors (Bogg and Birrell, 2025), particularly in dynamic contexts like local control. As workload fluctuates with scenario demands, task allocation, automation support, and inter face design should adapt in real time through predictive assistance, alerts, and adaptive visualization to maintain operator performance and prevent overload.

These findings highlight the importance of real-world, domain expert-de rived scenarios for assessing RO feasibility and limitations. Classifying 105 functional scenarios and their challenge types (perception, manoeuvre, deci sion-making) provides a structured basis for linking AV difficulty to oper ator task demands. The predominance of perception–manoeuvre (PxM) challenges underscore the need for precise environmental perception and adaptive motion planning, while local control as a distinct scenario type reveals how dynamic road geometry and temporary traffic rules add opera tional uncertainty. These insights reinforce the need for real-time communi cation between AVs, local control, and operators, and for RO system design grounded in real deployment contexts to ensure adaptive, context-aware, and resilient operation.

Future research should translate these findings into specific interface requirements, support tools, and operator training protocols to enhance RO efficiency. Further validation through on-road testing and simulation-based studies will be essential for refining intervention strategies and ensuring safe, effective HMI in ROs.

LIMITATIONS Limitations must be acknowledged. The temporal scope was restricted to single interview sessions, potentially omitting scenarios that vary season ally or across different event types. Formal saturation assessment was not conducted, limiting confidence in scenario comprehensiveness. Finally, the specific characteristics of the NEC event venue environment may constrain generalizability to other ODDs.

CONCLUSION

This paper highlights the value of real-world, knowledge-based scenario gen eration for defining HMI and RO workstation requirements and advanc ing understanding of RO feasibility. Through the creation of 105 functional scenarios, disturbance sources, and AV challenge classifications, the study highlights the central role of perception, manoeuvre, and decision-mak ing challenges in dynamic environments. The results point to the need for robust, adaptive RO interfaces that balance automation support with human oversight to manage workload and maintain SA. Future work should refine scenario-based evaluations to better assess operator performance and guide interface development, supporting safer and more reliable AV deployment in complex urban contexts.

ACKNOWLEDGMENT

The authors gratefully acknowledge James Connolly, Program Coordinator at Solihull City Council, for his instrumental role in organizing and facilitat ing the workshop that contributed to this research. His coordination efforts and ongoing support were essential to the successful execution of the study.

REFERENCES

Bogg, A., & Birrell, S. (2025). Overloaded, underloaded or in control: How many automated vehicles can one person supervise?. Computers in Human Behavior, 108690. BSI (2024). BSI Flex 1887 v1.0:2024-05, Human factors for remote operation of

vehicles. British Standards Institution. Cai, J., Deng, W., Guang, H., Wang, Y., Li, J., Ding, J. (2022). A survey on data-driven scenario generation for automated vehicle testing. Machines, vol. 10, no. 11, art. 1101. Golbabaei, F., Dwyer, J., Gomez, R., Peterson, A., Cocks, K., Bubke, A., Paz, A. (2024).

Enabling mobility and inclusion: Designing accessible autonomous vehicles for people with disabilities. Cities, vol. 154, art. no. 105333. Kamtam, S., Bhanu, Q., Lu, F., Bouali, O., Haas, C.L., Birrell, S. (2024). Network latency in teleoperation of connected and autonomous vehicles: A review of trends, challenges, and mitigation strategies. Sensors, vol. 24, no. 12, art. no. 3957. Kettwich, C., Schrank, A., Avsar, H., Oehl, M. (2021, September). What if the

automation fails?–A classification of scenarios in teleoperated driving. In 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications.

Kettwich, C., Schrank, A., Avsar, H., Oehl, M. (2022). A helping human hand: relevant scenarios for the remote operation of highly automated vehicles in public transport. Appl. Sci., vol. 12, no. 9, art. 4350. Koopman, P., Wagner, M. (2017). Autonomous vehicle safety: An interdisciplinary

challenge. IEEE Intell. Transp. Syst. Mag., vol. 9, no. 1, pp. 90–96. Menzel, T., Bagschik, G., Isensee, L., Schomburg, A., Maurer, M. (2019). From functional to logical scenarios: Detailing a keyword-based scenario description for execution in a simulation environment. In 2019 IEEE Intelligent Vehicles Symposium (IV), June 2019, pp. 2383–2390. Nalic, D., Mihalj, T., Bäumler, M., Lehmann, M., Eichberger, A., Bernsteiner, S.

(2020). Scenario based testing of automated driving systems: A literature survey. In Proc. FISITA Web Congress, vol. 10. Rasmussen, J. (2012). Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE transactions on systems, man, and cybernetics, (3), 257–266. Riedmaier, S., Ponn, T., Ludwig, D., Schick, B., Diermeyer, F. (2020). Survey on

scenario-based safety assessment of automated vehicles. IEEE Access, vol. 8. SAE International (2021). Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Tener, F., Lanir, J. (2024). Investigating intervention road scenarios for teleoperation of

autonomous vehicles. Multimedia Tools Appl., vol. 83, no. 21, pp. 61103–61119. Wolf, M. M., Schmidt, H., Christl, M., Fank, J., & Diermeyer, F. (2025). A usercentered teleoperation gui for automated vehicles: Identifying and evaluating information requirements for remote driving and assistance. Multimodal Technologies and Interaction, 9(8), 78.

Cover of the proceedings volume Intelligent Human Systems Integration (IHSI 2026): Disruptive and Innovative Technologies

Explore the complete digital volume

Intelligent Human Systems Integration (IHSI 2026): Disruptive and Innovative Technologies

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-76-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →