Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Remote Monitoring of Autonomous Ships: Quickly Getting into the Loop Display (QGILD)

NTNU, Norwegian University of Science and Technology, Trondheim, NO-7491, Norway ↗
Publication year
Digital Object Identifier10.54941/ahfe1002506

Abstract

Operators in Remote Operation Centers (ROC) of future Maritime Autonomous Surface Ships (MASS) are to be faced with the challenge of quickly getting into the loop when ships they monitor, after long periods of perfectly working automation, suddenly needs remote assistance. The question of Human-In-The-Loop (HITL) or Human-Out-Of-The-Loop (HOOTL) will become crustal in future development of automation towards autonomy in unmanned ships. The goal of this study is to build Automation Transparency into the human-machine interface of remote operation centers. This work-in-process paper presents some early concepts on a Quickly Getting Into the Loop Display designed to do just that.

INTRODUCTION

In 2017, following a proposal by a number of Member States, IMO’s Mari- time Safety Committee agreed to include the issue of marine autonomous surface ships on its agenda. It did so by launching a scoping exercise to deter- mine how the safe, secure and environmentally sound operation of Maritime Autonomous Surface Ships (MASS) may be introduced in IMO instruments. Four degrees of autonomy was identified for the purpose of the scoping exercise (IMO, 2022):

1. Ship with automated processes and decision support: Seafarers are on board to operate and control shipboard systems and functions. Some operations may be automated and at times be unsupervised but with seafarers on board ready to take control. 2. Remotely controlled ship with seafarers on board: The ship is controlled and operated from another location. Seafarers are available on board to take control and to operate the shipboard systems and functions. 3. Remotely controlled ship without seafarers on board: The ship is con- trolled and operated from another location. There are no seafarers on board. 4. Fully autonomous ship: The operating system of the ship is able to make decisions and determine actions by itself.

© 2022. Published by AHFE Open Access. All rights reserved. 691

692 Porathe

Degree two and three involves a “remote control centre” of some kind and one may presume that the IMO also envision that degree four will invo- lve some kind of remote control as a fallback solution should the automatic system fail.

The MUNIN Project Research on design of remote-control centers for autonomous merchant ships started already in 2012 with the first large European autonomous ship project Maritime Unmanned Navigation through Intelligence in Networks, 2012– 2015 (MUNIN, 2015). This author was responsible for the work-package investigating and sketching a first design of such a center (Porathe, 2014). Findings from a cost-benefit analysis led us to conclude that the centre pro- bably would have to be quite large. Our final organizational design assumed some 100 ships, in 3 control rooms each with 6 operators and one supe- rvisor. Each operator would be responsible for some 6 ships in an open ocean situation with very little traffic. As traffic and traffic constraints increased operators would supervise less ships (Porathe et al., 2013). The 6-ships-concept came from the “active monitoring”procedure requiring each operator to spent 10 minutes virtually onboard each vessel monitoring some key parameters. Thus, each ship was visited once an hour, to keep the ope- rator in the loop. (Porathe et al., 2020). Today, as one gets older, I would say: 5 ships, and then 10 minutes for coffee or a convenience break. The last was jokingly said. The number of ships that can be supervised by a remote operator naturally depends on a lot of external factors like traffic intensity, weather situation, technical status etc. and individual factors like expertise and workload. Such a number may therefore not be fixed but must remain dynamic. I think, however, it is safe to say that large parts of shipping take place during uneventful days on the open sea in relatively calm weather and little traffic, where more than one ship can be safely monitored by a single operator. The big problem, I think, will instead be to keep the operator alert and in-the-loop.

HUMAN IN-THE-LOOP Mica Endsley has defined the concept of Situation Awareness (SA) as “the perceptions of elements in the environment in a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (Endsley, 1988). Basically, this means knowing what is going on. One of the dangers to a good SA pointed out by Endsley and Jones (2012) is what they call the “Out-of-the-loop-syndrome”. This might occur when operators monitor well-functioning automation starts to think about other things and slowly slides “out-of-the-loop”.

With well-functioning automation, being out-of-the-loop may not be a big problem, but when the automation fails, or encounters problems it was not designed to solve, the operator might be slow to detect the problem and inte- rvene in a timely manner (e.g. Ehprath and Young, 1981; Kessel and Wickens, 1982; Wickens and Kessel, 1979; Young, 1969). In addition, once the pro- blem is detected it can take considerable time to reorient and understand the

Remote Monitoring of Autonomous Ships: A Quickly Getting 693

nature of the problem and what should be done about it. In many cases this delay can be very problematic. This loss of SA occurs through three primary mechanisms (Endsley and Kiris, 1995):

1. Changes in vigilance and complacency associated with monitoring 2. Assumption of a passive role instead of an active role in processing information for controlling the system 3. Changes in the quality or form of feedback provided to the human operator Of the three factors connected to poor SA leading to out-of-the-loop syn- drome, poor feedback and information presentation are the most easily remedied through the application of good human factors design principles (Endsley and Jones, 2012).

However, there is another problem that might be built into the system in ways that make potential problems invisible for the operator.

Human-in-the-Loop and Human-out-of the-Loop System developers often wrestle with the architecture of automation in command-and-control systems (C2). The question might be: Is a Human- Out-of-the-Loop (HOOTL) capability, where the automation decides every- thing and acts autonomously, ignoring the human (cf. level 10 of Sheridan and Verplank, 1978), better at delivering Rapid Relevant Responses (R3) than a Human-In-the-Loop (HITL) approach, where the user has complete control to start or stop the automation?

It seems many developers are hesitant letting the human operator into the system. Some AI developers even take the perspective that you can never fully trust the human. Meaning it is better to go HOOTL than HITL (Eliot, 2019). This view of course has its problems as was illustrated by the recent Viking Sky accident on the Norwegian west coast.

Viking Sky was a huge cruse ship with more than 1300 passenger and crew. In stormy weather in March 2019 she left Tromso. A stop in Bodo had to be cancelled due to the storm. Instead, the ship continued to Stavanger the next port. During times the cruse ship passed through sheltered waters inside the archipelago but during some passages she had to go over open and expo- sed stretches causing heavy rolling. One such place was Hustavika, a stretch which would normally take 30-40 minutes to pass before again entering shel- tered water south of Molde. But out in the rolling seas all of Viking Skye’s engines suddenly stopped. They had been automatically shut down due to low lubricating oil pressure. It turned out that the lubrication oil levels in the sumps was only kept at 28–40% of full capacity, and the rolling of the ship caused the oil to slosh around in the sumps triggering the oil-level sen- sors. The accident commission concluded in its interim report that the engines where “shut down as a result of the loss of lubricating oil suction due to low sump tank levels, combined with pitching and rolling” (AIBN, 2019, p. 7). The final report is yet not published (February 2022). One cannot help reflecting on that automatic trip function, that evidently was HOOTL, but instead maybe should have been HITL. Even with the risk of permanently

694 Porathe

Figure 1: Example on what a QGILD possibly could look like. Explanations in the text.
Original PDF Figure 1:

damaging the engines, the alternative of keeping the engines running ano- ther 30-40 minutes, if possible, would have been better that risking the lives of 1300 passengers and crew. (The adventure ended well, the ship survived, dragging anchor but stopping one boat length before the deadly rocks. The Norwegian rescue services managed to airlift 479 passengers in the 18 hours between 1500 on the 23rd and 0900 on the 24th of March when the ship was tugged into safety with more than 800 people still remaining onboard.)

Design for Bringing Humans in the Loop The research question for this work has been: When an operator in a remote operation centre is summoned to his console by an alarm and he or she does not know what is going on, how can the interface facilitate that he or she is brought back in the loop as quickly as possible? How can we achieve automa- tion transparency? So that an operator quickly can see how the automation has analyzed the situation and what it is proposing to do about it. The ope- rator should be offered quick and salient ways of intervention. The result is sketched in the following.

QGILD The Quickly Getting into the Loop Display, QUGILD, is a standardized way of presenting information so that the operator can train and become familiar with where the information is to be found and what different symbols mean. In Figure 1 a first prototype sketch of what a QUGILD could look like is illustrated. Further development then needs to be done in a HCD process with subject matter experts, end users and designers. I will now go through some details of the design.

Classification Icon Top left is a “classification icon” that is designed to give an immediate under- standing of What is the problem. If it is an upcoming close quarters situation, as here, an icon resembling the type of ship, its size and its aspect (in which

Remote Monitoring of Autonomous Ships: A Quickly Getting 695

Figure 2: Examples of some different classification icons. from the left “close quarters” (showing icon of ship type, size and aspect), then “mechanical failure” and “navigation Hazard”.

Figure available in the original PDF paper.

Figure 3: The eye glyph is used to communicate some states of visibility. From the left: day and good visibility, night and good visibility, day and somewhat reduced visibility and “restricted” visibility, in the COLREG sense of the word meaning that rule 19 apply.

Figure available in the original PDF paper.

perspective you would see it were you onboard). Example symbols are shown in Figure 2.

Remaining Time The remaining time until the automation will execute its action (shown in the orange field) is shown both graphically and numerically. The number is counting down to 0. An important point here is that the automation will always act, going into fail-to-safe or other, so that the system will not fail if the operator does not respond or if the communication link is lost.

Environmental Conditions A Multi Data Glyph is here used to show a number of different weather conditions in a single symbol. Visibility is very important for interpretation of the collision regulations. The eye icons in Figure 3 denotes from left to right: daylight/good visibility, nighttime/good visibility, daylight somewhat reduced visibility, and finally, restricted visibility (in the formal sense, COLREGS rule 19 being used). The glyph is further amended with wind and current direction and force (shown in Figure 1: northwesterly wind, 15 knots, and south-going cur- rent, 2.5 knots). The glyph might need to be amended with some other data dimensions and augmented warnings for parameters passing set thresholds.

Textual Problem Statement It will be the automation system that detects that something has happened where there is a need to bring the operator into-the-loop. One might image a situation where the system is unaware of some developing situation. In such a case there will be no launching of the QGILD.

But if the QGILD is launched, then there is a deviation which the system wants to communicate to the human. This can be done though “Informa- tion”, “Warnings” and “Alarms”. In these three cases the screen will have a yellow, orange, or red border. In Figure 1 the screen has an orange border,

696 Porathe

and the system is here warning the operator about a give-way vessel appro- aching on the own ships port side. This is a COLREG situation where the approaching vessel should yield. However, in this case the other ship has not yet shown any sign of giving way. We shall, according to Rule 17 keep course and speed, but under (a) (ii) we may, take action to avoid collision by our maneuver alone, as soon as it becomes apparent that the other vessel is not taking appropriate action in compliance with COLREGS. The auto- mation has decided to take action according to (a) (ii) in 36 seconds. (Note that for clarity the map is not to scale in the example in Figure 1.) This is the automation’s interpretation of the situation in this example.

The top part of the textual information covers the problem statement and the parameters of the situation. The middle section refers to the rulebook/pro- cess/checklist, and the bottom part what action the automation is preparing and when. Here is also an option for the operator to execute this option right now (if relevant and possible).

Other types of problem statements could e.g. be “Grounding alert”, “Main engine failure”, “Low oil pressure”, “Propulsion problem”, “Communi- cation outage” etc. (If the operator sees the last message, the system has switched over to the digital twin back-up in the control centre.)

Graphical Problem Statement A Map or Camera view should also accompany the QGILD if possible at allow for immediate recognizing and Skill-based behavior in Scrambled situ- ations. In cases of system malfunction and troubleshooting one might imagine bringing up mimics of system parts. It is important that the graphics are use- ful for an operator possibly in a stressed situation, affected by information overload and cognitive tunneling. All means should be taken for cognitive off-loading, such as presenting maps and camera views in egocentric or exo- centric head-up view for time critical maneuvering actions is needed. These concepts have been elaborated by me in the past.

CONCLUSION

I have in this paper presented some early finding in a work-in-progress desi- gning a screen which should allow a remote operator to quickly acquire situation awareness when the ship he or she is monitoring asks for help. The display is part of the ubiquitous problem of “automation transparency” – how can we understand what high level automation is thinking and deci- ding? The suggested QGUILD is part of such a quest. In future studies the display will be tested and iterated.

ACKNOWLEDGMENT

This research is conducted within the LOAS (Land-based monitoring of auto- nomous ships) and the SFI AutoShip projects, which are both funded by the Norwegian Research Council, which is hereby gratefully acknowledged.

Remote Monitoring of Autonomous Ships: A Quickly Getting 697

REFERENCES

AIBN, Accident Investigation Board Norway. (2019). Interim report 12 November 2019 on the investigation into the loss of propulsion and near grounding of Viking Sky, 23 March 2019. Eliot, Lance (2019). Human In-The-Loop Vs. Out-of-The-Loop in AI Systems: The

Case of AI Self-Driving Cars. aitrends, The business and Technology of Enterprise AI. https://www.aitrends.com/ai-insider/human-in-the-loop-vs-out-of-the-loop-i n-ai-systems-the-case-of-ai-self-driving-cars/ [Acc. 2022-02-14] Endsley, Mica R. (1988) “Design and evaluation for Situation Awareness enhancement”, proceedings of the Human Factors Society 32nd Annual Meeting (pp 97.101). Santa Monica, CA: Human Factors Society. Endsley Mica R. and Jones, Debra G. (2012) Designing for Situation Awareness: An

Approach to User-Centered Design (2nd Edition). CRC Press: Boca Raton, FL. Endsley, Mica R. and Kiris, Esin O. (1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37(2), 381–394. Ephrath and Young, (1981). Monitoring vs. man-in-the-loop detection of aircraft

control failures, in: Human detection and diagnosis of system failures, Rasmussen, Jens and Rouse, William B. (eds.) New York: Plenum Press. IMO (2022). Autonomous shipping. https://www.imo.org/en/MediaCentre/ HotTopics/Pages/Autonomous-shipping.aspx [Acc. 2022-02-12] Kessel Colin and Wickens, Christopher D. (1982). The transfer of failure-detection skills between monitoring and controlling dynamic systems. Human Factors, 24(1), pp. 49–60. MUNIN. (2015) Maritime Unmanned Navigation through Intelligence in Networks.

https://cordis.europa.eu/project/id/314286 [Acc. 2022-02-12] Porathe, Thomas. (2014) “Remote Monitoring and Control of Unmanned Vessels: The MUNIN Shore Control Centre”, proceedings of the 13th International COMPIT 2014, 12–14 May 2014, Redworth, UK. Porathe, Thomas, Burmeister, Hans C. and Rodseth, Ornulf .J. (2013) “Maritime

Unmanned Navigation through Intelligence in Networks: The MUNIN project”, proceedings of the COMPIT 2013 conference, 15–17 April 2013, Cortona, Italy. Porathe, Thomas, Fjortoft, Kay and Bratbergsengen, Inger-Lill. (2020) “Human Factors, autonomous ships and constrained coastal navigation”, The 3rd Inter- national Conference on Maritime Autonomous Surface Ship (ICMASS 2020) IOP Conf. Series: Materials Science and Engineering 929 (2020) 012007, IOP Publishing. Sheridan, Thomas B. and Verplank, William L. (1978). Human and computer control

of undersea teleoperators. Technical Report, Man-Machine Systems Laboratory, Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge. Mass. Wickens, Christopher D. and Kessel, Colin (1979). The effect of participatory mode and task workload on the detection of dynamic system failures. SMC -9(1), pp. 24–34. Young, Laurence, R. (1969). On adaptive manual control. Ergonomics, 12(4),

pp. 635–657.

Cover of the proceedings volume Human Factors in Transportation

Explore the complete digital volume

Human Factors in Transportation

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-36-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →