Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Control Centers for Maneuver-Based Teleoperation of Highly Automated Vehicles: System Model and Requirements

RWTH Aachen University, Institute of Industrial Engineering and Ergonomics (IAW), Eilfschornsteinstr. 18, 52062 Aachen, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1002483

Abstract

The drive towards highly automated vehicles is continuing to gain traction, however when bridging the gap from SAE level 2 automated vehicles to highly automated level 4 vehicles, there is a phase that will require temporary takeovers by a human driver. One approach is that a tele-operator is taking over the full control over the vehicle, which requires a sophisticated control center and a data connection enabling video with sufficient bandwidth and low latency. A slightly different approach can be chosen: When a driving automation reaches a system limit or border (e.g. a construction zone or an unclear traffic situation), it can request the support of an operator in a control center to support with the vehicle control. In this instance, the vehicle is not directly tele-operated but instead proposes a possible maneuver, which then can be selected, dismissed or approved by the operator, and will then be executed by the automated driving system. This is a likely scenario for vehicles with automation capabilities limited to specific use cases or in mixed traffic situations in which not all vehicles are capable of vehicle-to-vehicle communication. An additional role of the operator is responding to emergency calls by occupants by assessing the situation in the vehicle and its surroundings. Using a video and audio link, the operator can connect with an occupant in case of security or health concerns. The paper presents a human system analysis and sketches the requirements for a control center as an intermediate step towards the rollout of highly automated or autonomous vehicles.

INTRODUCTION

The concept of “driverless cars”was first presented in 1939, with GM exhibi- ting the “Futurama” concept, which showed a platooning automated vehicle (Kröger 2016). Until the early 1970s, the idea of a vehicle that drives itself remained fiction but then the PROMETHEUS project developed the VaMoRs (Versuchsfahrzeug für autonome Mobilität und Rechnersehen). The vehicle travelled along a closed-off section of the motorway at nearly 100 km/h in 1992 (Lossau 2017). Since there were some significant technological

© 2022. Published by AHFE Open Access. All rights reserved. 501

advancements and Google showcased a bespoke driverless car, without a ste- ering wheel and foot controls, navigating the roads of California in 2015 (Waymo 2016). In parallel to this technology-driven development, research and development in the field of human systems integration addressed the integration of such an automation with the human, the organizations and the environment. A central idea is that automation is not black and white, there is not only full automation or no automation, but that the distribution of control between humans and automation can be shared differently across multiple levels of automation. Initially exemplified for air- and ground veh- icles with the rider-horse metaphor (H-Metaphor, Flemisch et al. 2004ff), this led to concepts such as highly automated driving (Flemisch et al. 2006; Hoeger et al. 2011), the levels of driving automation (Gasser et al. 2012; SAE 2021), shared and cooperative guidance and control (Flemisch et al. 2014ff; Abbink et al. 2012), fluid distribution of control (Goodrich et al. 2006; Baltzer et al. 2015), and maneuver based driving (Flemisch et al. 2020a).

The advancement in vehicle automation technology has now reached a point that allows to consider vehicles that are capable to navigate normal driving situations fully automated, which would be defined as Level 4 ‘High Automation’ or Level 5 ‘Full Automation’ according to the SAE Levels of Automation (SAE 2021). Currently this level of automation is limited to vehicles that operate in confined areas, such as technology parks, where environmental hazards, traffic flows and general operational environments can be closely controlled. Those are typically driverless vehicles providing mobility on the first and last mile, whilst being monitored by an onboard supervisor. These operators typically have no influence over the vehicle other than being able to stop it in an emergency and in some applications to manu- ally manoeuver the vehicle using a joystick or a similar interface (Wasser et al. 2020). A potential solution to enable the technology to make the next step and to breach the gap to partake in normal on-road traffic, is the remote supervi- sion of these vehicles. In this scenario the vehicle automation is confronted to a number of additional challenges, such as unknown environments and situations as well as a mixed traffic situation where automated vehicles are using the same space as traditional, manually operated vehicles. This is likely to create situations in which the vehicle automation reaches the limits of its functionality, requiring a human intervention to resolve the situation. Remote connection is one potential solution, allowing a human operator to control a highly automated vehicle and manoeuver it through a situation the automa- tion is unable to resolve by itself. The operator can either directly control the vehicle, typically described as tele-operated or instead, as this paper proposes select an appropriate maneuver proposed by the vehicle automation, which it will then execute independently.

MANEUVER-BASED COOPERATIVE DRIVING Tele-operated driving describes a scenario in which a remotely located ope- rator is directly controlling a vehicle without physically being in the vehicle. This can be in the form of a small robot controlled by the operator whilst it is

Control Centers for Maneuver-Based Teleoperation 503

in the direct line of sight or a vehicle which the operator controls whilst seeing the surroundings via a video stream. The later may be used over long dista- nces, for example allowing an operator to safely control a vehicle in a hostile environment such as a remote reconnaissance unit such (Wasser et al., 2021), where an operator is able to directly control a small robotic platform, view- ing the surroundings on a monitor array and can, classify unknown objects highlighted in the viewing field.

However, tele-operated driving has significant disadvantages, especially in environments with poor network coverage due to the high amount of data required. In addition, there are difficulties due to operator and vehicle vari- ations, responsibility for multiple vehicles and other moving objects (Fong et al. 2001). Here, maneuver-based automation offers decisive advantages (Flemisch et al. 2020a): It describes a scenario in which an operator does not directly influence the vehicle movements, but instead supports the vehicle automation in the decision making process by selecting the appropriate next manoeuver, which is then executed independently by the automation. On the one hand maneuver-based driving is particularly useful with regard to coo- perative automated driving (e.g. Flemisch et al. 2014; Stiller et al. 2018; Usai et al. 2021). Here, one of the most obvious advantages is that it allows the driver to influence the automation (e.g., increase speed or initiate an over- taking maneuver) without having to completely take over the driving task (Flemisch et al. 2020b). On the other hand however, maneuver based control becomes elementary especially for vehicles that do not have a driver on board. In this scenario an operator in a control center does not have to continuously monitor the vehicles behavior but acts as a fallback option if the automation is unable to make a decision. This can be the case, for example, when the right-of-way rules are unclear (e.g. in a bottleneck) or when a violation of traffic rules must be actively cleared, such as when a stopped vehicle is blo- cking the lane and the automated vehicle has to cross a solid lane marking onto the oncoming lane to navigate around the obstacle.

Those possible maneuvers should be based on scenery, scene, situation and scenario, as proposed by Geyer et al (2014). The scenery here describes prede- fined possible road types (e.g., an intersection, or a rural road) in a necessary level of detail with any number of dimensions (e.g., lane width, traffic lights, or static obstacles). The scene extends the scenery with possible dynamic ele- ments (e.g. other vehicles or VRUs) as well as driving instructions (e.g. right of way rules). This does not require a complete description, but only a capture of the relevant elements. The situation describes the set of criteria that need to be true in order to perform the intended action of the ego vehicle. The end of a situation is defined by a change in one of the criteria describing the situation (e.g., passing through the intersection). The scenario is comparable to a storyline, e.g. the ego-vehicle has to reduce speed at the intersection, stop at the red light, accelerate again and then cross the intersection. Based on this definition, eleven basic maneuvers were proposed as part of the ‘Vorreiter’- project (Flemisch et al. 2020a): Start moving, move faster, stop moving, move slower, emergency brake, change lane right, turn right, change lane left, turn left, U-turn, overtake. This condensed set of maneuvers has several advan- tages over very detailed sets or direct tele-operation: the maneuvers are easy

to distinguish, they can be used to solve a wide variety of situations, and they can be selected and executed both directly and remotely either using a steering wheel or alternative controls (Kelsch et al. 2006).

POTENTIAL APPLICATION OF CONTROL CENTERS A number of potential applications for the remote operation of vehicles can be found in various domains, including military tasks such as remote recon- naissance. In such a scenario, the reconnaissance task can be conducted from a remote station where an operator controls a number of units transmitting video feeds to a single workstation. The task for the operator, supported by an intelligent software that highlights unknown objects, is to classify the obje- cts using a gaze based interaction method, whilst also monitoring the remote units (Wasser et al. 2021). In this concept the operator is monitoring highly automated vehicles, whilst mainly focusing on the classification task. In the future, when the reconnaissance units operate even more independently, the task may be split up again into classification and a separate operator who is monitoring a large number of units that only require infrequent support to either set the initial target or in the case a system limit is reached and an alternative manoeuver has to be selected. These roles could then be located in a control center, from where a fleet of automated units is monitored.

In civilian application a control center would support highly automated vehicles when they have reached a system limit and are unable to select the next manoeuver themselves, making a human intervention necessary. This can be caused by various reasons: situations with unclear traffic rules, limi- ted sensory foresight or to authorize driving maneuvers that are not permitted within the road traffic regulations, such as overtaking a broken-down vehicle in a no overtaking zone. In such cases, the vehicle requests assistance from the control center and provides information about the cause of the error and communicates possible maneuvers. For example, an obstacle in the lane could be the cause and the manual clearance of a lane change via the on-coming lane could be the requested maneuver which requires authorization by the operator. But also navigation outside the operational design domain (ODD), for example in depots of mobility hubs, or in the context of maintenance could make an external operation necessary. Another reason for an interven- tion by the control center is an unreachable destination. If the automated driving system is unable to reach a destination, for example due to parked transporter at the shuttle stop, a new destination must be defined manually by the control center.

However, the tasks of a control center may go beyond releasing maneuvers or resolving deadlock situations. In a large-scale survey regarding the wishes of potential users of autonomous shuttles in Germany, a large proportion of respondents explicitly requested safety measures (Herzberger et al. 2019). In particular, a contact person by telephone/video link via an emergency call system was mentioned as an important acceptance criterion. However, these emergency responses do not have to be limited to events inside the vehicle, such as assaults or medical emergencies, but can also include the surroundings (e.g., vandalism).

Control Centers for Maneuver-Based Teleoperation 505

Figure 1: System model showing the connections between an operator, a vehicle subsystem and potentially a passenger (adapted from Wasser et al. 2021).
Original PDF Figure 1:

Another use case is the topic of fleet management. This involves providing sufficient vehicle capacities at specific times at relevant locations for vehi- cles used in public transport. Daily peaks, such as commuter traffic, do not usually require intervention, but rather irregular major events such as confe- rences. However, the specific route selection of the individual vehicles is not made by the control center but is determined locally. Another aspect of fleet management is the tracking of vehicle status, such as the state of charge, or maintenance. Acute error messages, such as faulty sensors or those relating to the drive, can also be transmitted to the control center.

SYSTEM MODEL AND REQUIREMENTS FOR A CONTROL CENTER The main components of the system described in Figure 1, are the opera- tor located in the control center and the vehicle which may have passengers on board and is located in a different environment. Information from the environment is shown in purple, from the operator in turquoise, technical information in blue and passenger information in green. The passengers are connected via the interior interaction system onboard the vehicle subsystem, within which the external sensor data is processed to provide information to the automation software as well as a video stream to the remote opera- tor. The automation software or if required the operator can make strategic decisions, selecting the next appropriate manoeuver, which is then however always executed on the tactical and operational level by the automation sof- tware. In both, the vehicle subsystem and the control center environment, are interfaces connecting the operator, the passengers and the vehicle auto- mation. On the passenger side it enables a direct interaction, such as selecting a destination or contacting the operator in the case of an emergency as well as an indirect interaction, where sensors within the cabin monitor the pas- sengers in the case of a health emergency. The interface in the control center presents any relevant information about the situation the vehicle is in as well

as the proposed manoeuver. Overall the system model shows the different components and the required information flow between the sub-systems to enable a concept like a control center for highly-automated vehicles.

OUTLOOK The applications presented in this paper point out both the technical and safety-related need for control centers in the context of enabling the next development step of automated driving. It can be assumed that the need for maneuver clearances will initially be relatively high due to an increasingly high proportion of mixed traffic. However, as automated and connected veh- icles become more widespread, the need for driving task-related interventions will subsequently decrease. Nevertheless, the use-case of a remote emergency center will most likely remain for the acceptance of shared vehicles, such as first and last mile mobility vehicles or to resolve traffic situations in unknown areas.

A prototypical control center is to be set up as part of subsequent resea- rch. Subsequently, the stakeholder and system requirements will be identified within the framework of explorative workshops and user studies.

ACKNOWLEDGMENT

This publication was funded within the Priority Programme 1835 “Coopera- tive Interacting Automobiles (CoInCar)” of the German Science Foundation (DFG).

REFERENCES

Abbink, D. A., Mulder, M., & Boer, E. R. (2012). Haptic shared control: smoothly shifting control authority. Cognition, Technology & Work, 14(1), 19–28. Baltzer, M.; Lopez, D.; Kienle, M.; Flemisch, F. (2015): Dynamic distribution of con-

trol via grip force sensitive devices in cooperative guidance and control; Berliner Werkstatt Mensch-Maschine. Flemisch, F.; Kelsch, J.; Schieben, A.; Schindler, J. (2006): Stücke des Puzzles hochautomatisiertes Fahren: H-Metapher und H-Mode; 4. Workshop Fahrerassi- stenzsysteme; Löwenstein. Flemisch, F., Bengler, K., Bubb, H., Winner, H., & Bruder, R. (2014). Towards

cooperative guidance and control of highly automated vehicles: H-Mode and Conduct-by-Wire. Ergonomics, 57(3), 343–360. Flemisch, F., Diederichs, F., Meyer, R., Herzberger, N., Baier, R., Altendorf, E., ... & Kaiser, F. (2020a). Vorreiter: Manoeuvre-based steering gestures for partially and highly automated driving. In Smart Automotive Mobility (pp. 231–304). Sprin- ger, Cham.Fong, T., Thorpe, C., & Baur, C. (2001). Collaborative control: A robot-centric model for vehicle teleoperation (Vol. 1). Pittsburgh: Carnegie Mellon University, The Robotics Institute. Flemisch, F. O., Pacaux-Lemoine, M. P., Vanderhaegen, F., Itoh, M., Saito, Y.,

Herzberger, N., Wasser, J. Grislin, E. & Baltzer, M. (2020b). Conflicts in human-machine systems as an intersection of bio-and technosphere: Coopera- tion and interaction patterns for human and machine interference and conflict resolution. In 2020 IEEE International Conference on Human-Machine Systems (ICHMS) (pp. 1–6). IEEE.

Control Centers for Maneuver-Based Teleoperation 507

Gasser, T. M.; Arzt, C.; Ayoubi, M.; Bartels, A.; Bürkle, L.; Eier, J.; Flemisch, F.; … & Vogt, W. (2012) Rechtsfolgen zunehmender Fahrzeugautomatisierung - Gemein- samer Schlussbericht der Projektgruppe Bundesanstalt für Straßenwesen (bast), (F 83). Geyer, S., Baltzer, M., Franz, B., Hakuli, S., Kauer, M., Kienle, M., Meier, S., Wei-

ßgerber, T., Bengler, K., Bruder, R., Flemisch, F. & Winner, H. (2013). Concept and development of a unified ontology for generating test and use-case catalogues for assisted and automated vehicle guidance. IET Intelligent Transport Systems, 8(3), 183–189. Goodrich, K.; Flemisch, F.; Schutte, P.; Williams, R. (2006): A Design and Interaction Concept for Aircraft with Variable Autonomy: Application of the H-Mode; Digital Avionics Systems Conference; USA. Herzberger, N. D., Schwalm, M., Reske, M., Woopen, T., & Eckstein, L. (2019).

Mobilitätskonzepte der Zukunft-Ergebnisse einer Befragung von 619 Personen in Deutschland im Rahmen des Projekts UNICARagil. Universitätsbibliothek der RWTH Aachen. Hoeger, R.; Zeng, H.; Hoess, A.; Kranz, T.; Boverie, S.; Strauss, M. … & Nilsson, A.

(2011) HAVEit Deliverable D61.1, Final Report. Kelsch, J., Flemisch, F., Löper, C., Schieben, A., & Schindler, J. (2006). Links oder rechts, schneller oder langsamer? Grundlegende Fragestellungen beim Cognitive Systems Engineering von hochautomatisierter Fahrzeugführung. Kröger, F., (2016) “Automated Driving in its Social, Historical and Cultural Con-

texts” in Markus Maurer et al (Eds.) (2016). Autonomous Driving: Technical, Legal and Social Aspects. Springer. pp. 41–68 (p. 48). ISBN 978-3-662-48847-8. Lossau, N. (2017). Das erste autonome Auto kostete 200.000 D-Mark. Welt.

https://www.welt.de/wissenschaft/article169604489/Das-erste-autonome-Auto- kostete-200-000-D-Mark.html SAE International (2021). Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles (No. J3016_202104). 400 Commonwealth Drive, Warrendale, PA, United States. SAE International. Stiller, C., Burgard, W., Deml, B., Eckstein, L., & Flemisch, F. (2018). Kooperativ

interagierende Automobile. at-Automatisierungstechnik, 66(2), 81–99. Usai, M., Meyer, R., Baier, R., Herzberger, N., Lebold, K., & Flemisch, F.

(2021). System Architecture for Gesture Control of Maneuvers in Automated Driving. In International Conference on Intelligent Human Systems Integra- tion (pp. 65–71). Springer, Cham. Wasser, J., Parkes, A., Diels, C., Tovey, M., Baxendale, A. (2020). Human Centred Design of First and Last Mile Mobility Vehicles. 10.13140/RG.2.2.21586.58560. Wasser, J., Bloch, M., Bielecki, K., Vorst, D., López, D., Baltzer, M. & Flemisch,

F. (2021). Gaze Based Interaction for Object Classification in Reconnaissance Missions Using Highly Automated Platforms. 10.1007/978-3-030-68017-6_124. Waymo (2016) Hello to Waymo. https://medium.com/waymo/say-hello-to-waymowhats-next-for-google-s-self-driving-car-project-b854578b24ee

Cover of the proceedings volume Human Factors in Transportation

Explore the complete digital volume

Human Factors in Transportation

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-36-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →