Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

The use of metaverse in maritime sector – a combination of social communication, hands on experiencing and digital twins

Publication year
Digital Object Identifier10.54941/ahfe1001513

Abstract

COVID-19 has caused challenges in traveling, sales activities, and teleconferencing is at the moment the main communication tool in industry. Various technologies have been introduced during the last a couple of years to increase the immersion of remote presence. Technologies, that can be called as a virtual reality social platform (VRSP) such as Glue, MootUp, Breakroom, LearnBrite, Virtway Events, Engage, AltspaceVR, and VirBELA are robust solutions for a small group of people to participate in the same event. In July 2021, both Microsoft CEO and Facebook CEO published their visions for remote presence, teleconferencing, and much more. Microsoft plans to build an enterprise metaverse (combining technologies such as AltspaceVR, Mesh, Teams, and HoloLens). Facebook in turn intends to transit their business from social media platform to metaverse provider. In February 2021, much before the metaverse enhancement of the big tech giants, Turku University of Applied Sciences started the development of its own metaverse technology. According to Nevelsteen (2018) metaverse is a virtual universe shared amongst its users allowing them to interact with each other within the boundaries of the platform. For TUAS the metaverse concept is a combination of social communication, hands on experiencing and digital twin integration. None of the mentioned VRSP technologies include all these three features neither Microsoft or Facebook have this type of vision. Moreover existing VRSP technologies typically have challenges such as update management, number of simultaneous users, usability, user experience, license policies, customization but also limited user interaction and user data gathering. In this paper, we will introduce our own metaverse technology. Research conducted at TUAS tested existing VRSPs, compared Unity and Unreal game engines in metaverse development and based on industry trend analysis revealed that Epic Games intends to focus on metaverse software development tools. On the other hand the use of Unreal limits user groups for VR and PC users. This research approach presented in this paper provides metaverse experiences to be used with various ways from VR/AR/XR to PCs and mobile phones. This way, users who have limited resources available are able to participate in metaverse applications where the user experience will vary based the available device. This paper introduces the TUAS metaverse technology, the first experiences of developing it and how the first prototype version of this technology has been used in various maritime sector projects. To demonstrate the technology a harbor area has been selected as a test environment. In the first test, AltspaceVR technology was used to develop the environment itself with features needed in virtual exhibition called Match XR. This will be followed by an experiment where the harbor environment with same functionalities will be moved to the TUAS metaverse technology. Finally, the paper reports how multiplayer concepts such as MarISOT command bridge training scenarios can be developed utilizing the TUAS metaverse technology.

INTRODUCTION

Because of the current pandemic various domains have suffered such as avi- ation industry which has been forced to close flight simulator centers and which has been amongst the most vulnerable verticals during the first mon- ths of the pandemic (AviationPros, 2020). In maritime sector, during the most severe restrictions from March to June 2020 for example the global ship mobility was decreasing up to −13.77% for container ships, and up to −42.77% for passenger ships (Millefiori et al., 2021). International Maritime Organization (IMO, 2020) has studied the impact of COVID-19 on maritime education and training and as a conclusion defining distance learning delive- red online and/or remotely to be likely a new normal. This trend proposed by IMO is in line with EU which has seen VET (Vocational Education and Training) learners to be at disadvantaged compared to any other learners (EU Monitor, 2020).

© 2022. Published by AHFE Open Access. All rights reserved. 115

Metaverse approach presented later in this paper can be seen as a conti- nuum for Virtual Continuum (VC) presented by Milgram & Kishino (1994). In their visions, an ideal virtual space with reality essential for communica- tion will be needed within the same visual display environment. According to XR4All (20220) eXtended Reality (XR) can be seen as an umbrella term used for Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) – that is to say covering terms used in the VC taxonomy. Virtual reality is defined as a method of interacting with a computer-simulated environment (Das et al., 1994). Augmented reality, in turn, is a technology which allows computer generated virtual imagery to exactly overlay physical objects in real time (Zhou et al., 2008).

In our previous study (Markopoulos et al., 2021), we have focused on virtual reality social platforms (VRSPs) as a precursor to the metaverse allowing its users to host or join multiplayer instances in which users of this platform can socialize and interact with their environment and each other. Actually, the word “metaverse” comes from the prefix “meta”, meaning beyond, and “universe”. It describes a virtual universe shared amongst its users allowing them to interact with each other within the boundaries of the platform (Nevelsteen, 2017). Second Life launched alre- ady in 2003 has been one of the first and the most prominent example of virtual worlds (Kohler et al., 2009). Later VRSP technologies such as MootUp, Breakroom, LearnBrite, Virtway Events, Engage, Microsoft AltSpa- ceVR, Facebook Horizon, Glue, Mozilla Hub, VR Chat and VirBELA have been among the most commonly used virtual platforms (Jauhiainen, 2020; Österman, 2021).

During year 2021 the metaverse has become more frequently used term. Microsoft’s CEO Nadella (2021) has illustrated their visions of an enterprise metaverse as a platform layer bringing together IoT, digital twins, and mixed reality. So, it is evident that boundaries of the platform will be stretched in the near future – Microsoft for example has introduced Mesh for Micro- soft Teams to combine traditional teleconferencing and mixed reality features such as shared holographic experiences introduced already in their VRSP cal- led Microsoft AltspaceVR (Microsoft, 2021). Facebook in turn has described their organizations transition from a social media company to a metaverse company called Meta. This transition will require take time around 10-15 years before metaverse products will be fully realized (Bosworth and Clegg, 2021). Maritime among many other business sectors has struggled during the pandemic and challenges in maritime sector is naturally influencing in the global business growth significantly. According to IMO (2021) some mari- time universities have been forced to cancel entire academic years during the pandemic and is now concerned that the pandemic will may have negative consequences for the number of young people choosing a seafaring career. XR seems to have a potential to boost GDP globally by 2030 by up to $1.5 trillion (PwC, 2019). Global XR market size itself in turn is expected to grow from $12.4 billion (2019) to $181.9 billion by 2028 (Fortune Business Insi- ghts, 2020a&b). Before the pandemic Goldman Sachs (2017) has estimated global VR and AR markets to grow into a $95 billion market by 2025 in

The Use of Metaverse in Maritime Sector – A Combination of Social Communication 117

turn. In their forecasts education sector seems to be in margin with just 2.0% market share.

In this paper, three key features of metaverse designed for maritime sector and for technology industry will be introduced namely social communication, hands on experiencing and digital twin integration. In the next chapter, the preliminary study will be reported. The objectives were to find the most pro- mising combination of hardware and software tools. This will be followed by the prototyping study where development of the metaverse technology will be explained in test-generate cycles. Finally, the first pilots will be introduced shortly covering all three key features.

PRELIMINARY STUDY Turku University of Applied Sciences has started in February 2021 the deve- lopment of its own metaverse technology. This technology consists of features for social communication, hands on experiencing and digital twin inte- gration. Based on our understanding none of the above mentioned VRSP technologies include all these three features, neither Microsoft nor Facebook have introduced this type of vision. Moreover, existing VRSP technologies typically have challenges such as update management, number of simultane- ous users, usability, user experience, license policies, customization but also limited user interaction and user data gathering. In this paper, we will intro- duce our own metaverse technology. In the preliminary study phase, various existing VRSPs have been tested, a comparison study between Unity and Unreal game engines in the metaverse development has been conducted. Epic Games has plans to publish a metaverse software development tool with $1.0 billion funding round (Epic Games, 2021).

Based on our test results and intention to focus on the technology industry the use of Unreal limits user groups for VR and PC users. The resea- rch approach presented in this paper provides metaverse experiences to be used in various ways from VR/AR/XR to PCs and mobile phones. This way, users who have limited resources available are able to participate in metaverse applications where the user experience will vary based the available device. Through the prototyping and piloting studies, we intro- duce components of IoT integration and multi-user training scenarios which provided requirements for the technical prototyping of TUAS metaverse platform.

PROTOTYPING STUDY Remote Controlled Robot Application and Testing IoT Integration

First of the IoT prototyping use-cases was Tinkerkit Braccio robot arm (Figure 1), which was remote controlled using VR/XR hand-tracking over Internet public network tunnel. The prototyping provided information and documentation about the required protocol support and data flow requi- rements for integrating further embedded system solutions (Blanco Bataller, 2021).

Figure 1: Virtual reality headsets used to control robot arm remotely (Blanco Bataller, 2021).
Original PDF Figure 1:

Once the data flows and interfaces were documented and created, the solution would be scalable to TUAS metaverse usage though environment- triggered connectivity. This framework could be extended to factory environment simulations and digital twins: assembly line of machinery could be remotely operated by multiple users connected into common VR environment.

Multi-User Training of Fire Extinguisher Usage With Real-Time Instructions in VR Online hands-on-training use cases were tested in a prototype featuring fire extinguisher usage. In this prototype, instructor would be hosting an environ- ment with fire extinguishers and burning barrels. The students would pick up an extinguisher, prime it, target it towards the base of the fire and extinguish it. The scenario calculates the amount of fire extinguisher foam that hits the base of the fire, and fire animation then reacts accordingly.

The instructor can observe the actions of the students from ground view or bird view. They are also able to point a laser-pointer, to instruct students. The scenario also features voice communication between students and the instructor (Figure 2).

This prototype was created using Photon Bolt using Photon Cloud netw- ork connectivity which requires Internet connection also for local network multi-user setup. The maximum feasible number of concurrent users with the scenario’s complexity and physics was found to be between 6 and 8 users. More users would cause noticeable stuttering and break the immersion as well as cause difficulties from the training’s perspective. The stutter, latency and bandwidth issues were caused by the Photon Cloud connections, where each concurrent user would increase the data transfers exponentially, since every concurrent user is connected to every other user through the cloud.

PILOTING STUDY Porting Harbor Environment

This paper introduces the TUAS metaverse technology, the first experiences of developing it and how the first prototype version of this technology has been used in various maritime sector projects. To demonstrate the techno- logy a harbor area has been selected as a test environment. In the first test, AltspaceVR technology was used to develop the environment itself with fea- tures needed in virtual exhibition called Match XR (XR Center, 2021). This

The Use of Metaverse in Maritime Sector – A Combination of Social Communication 119

Figure 2: Multi-user training of fire extinguisher usage enabling the teacher student communication.
Original PDF Figure 2:
Figure 3: The 3rd person gameplay on desktop (left) and the 1st person gameplay on VR (right).
Original PDF Figure 3:

was followed by an experiment where the harbor environment with same functionalities was moved to the TUAS metaverse technology (Figure 3).

During the event week, a system update was initiated to the AltspaceVR technology backend, which rendered ready-made scenario dysfunctional. This downtime of the backend and system update with breaking changes would have caused significant disturbance of the event in case it would have happened during the environment event. AltspaceVR technology restrictions regarding scripting and dynamic activities was found to limit both the envi- ronment design and the activities that could be presented to the visitors. The environment supports static images, videos and text content, which dynamic updates of the content would require a new release of the environment.

Environment isolation is one of the core features in TUAS Metaverse tech- nology. The architecture could be maintained in its launch state indefinitely, without need of regular downtime from updates. The feature set is locked into the environment and future development can be integrated to an environment on-demand. The TUAS metaverse technology is built and structured to be modular from functionality and graphical representation of the environment perspective. Any graphical assets and functionalities created into the base platform feature set are available as core features for all environments, which provides significant improvement to development and prototyping cycles.

Unity environment conversion from AltspaceVR technology into TUAS metaverse technology is directly supported with minimal development effort requirements. Digital twins with specific functionalities can be integrated into the platform. With fully customizable API support of the platform, the digital

twins and their sensors could be coupled to real world machinery and equi- pment, providing access to executing remote operations. With data collection, this functionality could be further developed to user-assisted autonomous systems and to gather training data for neural network operated autonomous systems. Multi-user environment and activities provide high variance in the behavioral data, reinforcing the neural network training data set.

Harbour environment from AltspaceVR, specifically graphical assets, can be directly imported to TUAS metaverse technology. The required changes were:

  • Defining the possible teleportable areas, which determine where the users can teleport,
  • Defining the spawn points for the users,
  • Defining which objects in the environment would be synchronized over the network,
  • Defining the scaling of the environment to match the scaling of generic avatar The bandwidth consumption of the TUAS metaverse server scales with the number of concurrent users and the type of devices that users use to join the hosted environment. This includes desktop users (namely non-VR users) and VR users. Each type of user has different interaction input to the virtual world but visually represented the same in a 3D graphical model, namely avatar.
  • For desktop users, only the position and orientation of the avatar are synchronized over the network. This combination represents 1 transform package.
  • For VR users, in addition to the avatar transform, the position and orien- tation of the head, left hand and right hand are also synchronized over the network, as well as finger animations. Each of these individual compo- nents represents one extra transform package. Additionally, high-end VR headsets can produce eye-tracking data and this data could be optionally sent over the network.

Most of the interactions in TUAS metaverse are physics-based interactions. This makes interaction in the virtual world more intuitive and immersive because this mimics real world interaction. For the interaction synchroni- zation part, there are 2 most commonly used network models to consider: server-side authority and client-side authority.

  • Server-side authority: clients send inputs to server, the server processes and simulates the world based on that input then broadcasts the world states back to all clients. This introduces a lag between the user’s input action and its local visual representation of the world since the client has to wait for the world state to arrive and then update its local world based on it. How- ever, VR interaction requires instant input response to deliver the most immersive experience possible. Client-side prediction technique is usually applied to solve this problem where the clients apply the input immedia- tely and then send it to the server. However, client-side prediction requires state reconciliation technique to resolve conflicts between the server state

The Use of Metaverse in Maritime Sector – A Combination of Social Communication 121

and the client-predicted state, this, in turn requires a deterministic phy- sics engine to produce an acceptable result. Unfortunately, Unity’s physics engine is not deterministic, and even if it is, this approach would not scale well with substantial number of concurrent users due to high amount of physics state rollbacks performed per frame.

  • Client-side authority: clients process input locally and send its world state to the server which in turn broadcasts that state to all other clients. This approach makes each client a contributor to a collective world state. Each client claims its authority over the object it interacts with and does not allow other clients to interact with it to avoid state conflict between cli- ents. This approach provides instant input response to match the visual of the local world regardless of the internet delay and also scales well with numerous amounts of concurrent users because each client simulates its own world regardless of other clients.

The TUAS metaverse mainly adopts the client-side authority due to its lower complexity compared to server-side authority approach and its capa- bility to scale with high number of concurrent users. Under highly delay simulated connection condition of 500ms, the synchronization as well as input response were still well performed.

CONCLUSION

For massive number of concurrent users, peer-to-peer connections were found to be exponentially demanding, growing in relation to the number of users. Therefore, dedicated server solution was selected as the architecture for TUAS metaverse platform development. The dedicated server architecture moves the bandwidth and resource demand towards the server that hosts the environment. Especially calculations needed for server-side authority would require server hardware and performance. Unlike in peer-to-peer architecture, the hosting server does not need to run any graphics. The server itself would be hosted either locally in internal network, or in a datacenter connected to public Internet, depending on the intended clients. Both solutions may also exist in parallel: the same server could host environment for both deve- lopment (only internal connectivity) and production (publicly reachable or tunneled).

Future development efforts will be targeted towards hands-on-trainings, which require realistic simulations of physics and machinery operations. Especially physics synchronization over network to massive amount of users in a stutter-free environment remains a key development path. Maritime simulations require realistic physics and synchronization of ocean as well as forces between all participants.

In the near future, we will continue piloting research through command bridge training environment, where operations can be exercised by several users, maneuvering a ship together in various typical maritime scenarios. The environment would support data collection through eye-tracking and hand- tracking, providing logged information on user’s order of operations, focus and logical reasoning. We will also investigate how multilingual versions

could be developed for example enabling real-time translations between the trainer and the trainee.

Multi-user trainings with IoT integration and data collection offer attra- ctive possibilities for collaboration and co-operation trainings in observed and repeatable scenarios, as well as in digital twin remote control operati- ons. Data collection of multi-user environment actors provides options for autonomous systems research as well as automatic evaluation of the training activities.

AKNOWLEDGEMENT The authors would like to thank everyone who have participated in the development of the preliminary, prototyping, and pilot studies. This work was supported by Business Finland and Finnish Ministry of Education and Culture.

REFERENCES

AviationPros (2020) Flight Simulator Sales Ramp Up With AR and VR Integration (May 21, 2020). DOI = https://www.aviationpros.com/education- training/simulator-training/press-release/21139115/future-market-insights-flight- simulator-sales-ramp-up-with-ar-and-vr-integration-global-airline-closure-due- to-covid19-pandemic-temporarily-arrests-demand-growth Blanco Bataller, V. (2021) Using Virtual Reality to Control a Robot Remotely, BSc Thesis, Turku University of Applied Sciences. Bosworth, A., and Clegg, N. (2021) Building the Metaverse Responsibly, Meta News-

room (September 27, 2021), DOI = https://about.fb.com/news/2021/09/building- the-metaverse-responsibly/. Das, S.; Franguiadakis, T.; Papka, M.; Defanti, T.A.; Sandin, D.J. A genetic programming application in virtual reality. In Proceedings of the 1994 IEEE 3rd International Fuzzy Systems Conference, Orlando, FL, USA, 26–29 June 1994; pp. 480–484. Epic Games (2021) Announcing a $1 Billion Funding Round to Support Epic’s

Long-Term Vision for the Metaverse, DOI = https://www.epicgames.com/site/en- US/news/announcing-a-1-billion-funding-round-to-support-epics-long-term- vision-for-the-metaverse EU Monitor (2020) Explanatory Memorandum to COM (2020)275 - Vocational education and training (VET) for sustainable competitiveness, social fairness and resilience (July 1, DOI = https://www.eumonitor.eu/9353000/1/j4nvhdfdk3hydzq _j9vvik7m1c3gyxp/vla0ekyy79qj Fortune Business Insights (2020) Augmented Reality Market 2021-2028, Summary

of Report ID: FBI102553, DOI = https://www.fortunebusinessinsights.com/augm ented-reality-ar-market-102553. Fortune Business Insights (2020) Virtual Reality Market 2021-2028, Summary of Report ID: FBI101378, DOI = https://www.fortunebusinessinsights.com/industry- reports/virtual-reality-market-101378. Goldman Sachs (2016) Virtual & Augmented Reality Understanding the Race for

the Next Computing Platform, Equity Research, Excerpted, January 13, 30p. IMO (2020) The impact of COVID-19 on maritime education and training, Maritime Safety Committee, MSC 102/INF.25, October 14, 2020, 5p.

The Use of Metaverse in Maritime Sector – A Combination of Social Communication 123

Jauhiainen, J. (2020) Virtual 3D platforms in entrepreneurship and innovation events during the COVID-19 pandemic. The case of SHIFT in Finland in October 2020 on the VirBELA platform. Report number: BIIDEA reports. 2. University of Turku. Kohler, T., Matzler, K. and Füller, J. (2009). Avatar-based innovation: Using virtual

worlds for real-world innovation. Technovation 29: 6–7, 395–407. Markopoulos, P., Pyae, A., Khakurel, J., Markopoulos, E., Saarnio, R., and Luimula, M. (2021) Understanding How Users Engage in an Immersive Virtual Reality- Based Live Event, In: Proceedings of the 12th IEEE International Conference on Cognitive Infocommunications CogInfoCom 2021, pp. 881–888. Microsoft (2021a) CEO Satya Nadella’s keynote Speech, Microsoft Inspire 2021,

(July 15, 2021) DOI = https://news.microsoft.com/wp-content/uploads/prod /2021/07/Microsoft-Inspire-2021-Satya-Nadella.pdf. Microsoft (2021b) Mesh for Microsoft Teams Aims to Make Collabora- tion in The ‘Metaverse’ Personal and Fun, November 2, 2021, DOI = https://news.microsoft.com/innovation-stories/mesh-for-microsoft-teams/. Milgram, P., and Kishino, F. (1994), A Taxonomy of Mixer Reality Visual Displays, IEICE Transactions on Information Systems, Vol E77-D, No. 12, pp. 1321–1329. Millefiori, L.M., Braca, P., Zissis, D., Spiliopoulos, G., Marano, S., Willett, P.K.,

and Carniel, S. (2021) COVID-19 impact on global maritime mobility. Nature Scientific Reports 11, Art. 18039. Nevelsteen, KJ. (2017) Virtual world, defined from a technological perspective, and applied to video games, mixed reality and the metaverse. Computer Animation and Virtual Worlds. 29(1) / e1752, DOI = https://doi.org/10.1002/cav.1752. Österman, M. (2021) Development of a virtual reality conference application. Bach-

elor’s Thesis in Information and Communications Technology. Turku University of Applied Sciences. PwC (2019) Seeing Is Believing -How Virtual Reality and Augmented Reality Are Transforming Business and The Economy, PwC Reports, (November 19, 2019), DOI = https://www.pwc.com/gx/en/technology/publications/assets/how-virtual-r eality-and-augmented-reality.pdf. XR Center (2021) MatchXR Featured Partner Worlds, DOI = https://www.matchx

rhelsinki.com/partner-worlds/ XR4All (2022) Definition – What is XR?, XR4All Horizon 2020, DOI = https://xr 4all.eu/xr/ Zhou, F.; Duh, H.B.-L.; Billinghurst, M. Trends in augmented reality tracking,

interaction and display: A review of ten years of ISMAR. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, Cambridge, UK, 15–18 September 2008; pp. 193–202.

Cover of the proceedings volume Creativity, Innovation and Entrepreneurship

Explore the complete digital volume

Creativity, Innovation and Entrepreneurship

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-07-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →