Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

XR is more than the sum of AR, VR, and MR

Publication year
Digital Object Identifier10.54941/ahfe1003698

Abstract

Extended reality XR as the latest reality enhancement technology is regarded by many as the sum of augmented reality AR, virtual reality VR and mixed reality MR. However, this article argues otherwise. This article discusses the misconceptions of LED-based XR as the sum of AR, VR and MR. Although MR is still the combined form of AR and VR, their focuses are different from the LED-based XR. While the common features of AR, VR and MR mostly focus on the different treatments of environment as reality, their limitations lay on their inability to integrate real-time high resolution video of human figures. VR is also limited by computing power. More advanced motion and other sensory capture devices are not inaccessible to regular VR consumers. VR users can only see real-time characters or avatars in low polygon resolution. While MR is still the sum of AR and VR by combining virtual reality with the physical world, MR is not equipped with high resolution camera to achieve the real time integration of photo realistic image in the mixed reality. With some exceptional cases, most of these AR, VR and even MR are typically for the interactive experience of one single user at a time while others can only observe through the projection system. On the other hand, LED-based XR with its real time integration of high resolution camera is designed not for one single user. With motion capture devices integrated in the system, characters can interact among themselves and with the virtual environment. XR also has the power to integrate the immersive features of AR, VR and MR but more. The key feature of LED-based XR can place a high resolution moving human figure or avatar inside the virtual world real-time. However, LED-based XR is not without its limitations. This article focuses on discussing the strengths and weakness of LED-based XR and how it can achieve more than the sum of AR, VR, and MR.

INTRODUCTION

‘Reality ends here’ is the famous motto of the USC School of Cinematic Arts. For many decades, this motto encourages students to dream the dreams and to imagine beyond the reality. Another connotation of this motto can be interpreted as encouragement for their students to challenge or question the reality as we know it. Concept of reality is one of the popular topics in many sci-fi literature, in which our perception of reality is often being questioned. Human nature is fascinated by different interpretations of the reality as they are good sources of escapism. In a commencement speech at the Princeton

© 2023. Published by AHFE Open Access. All rights reserved. 82

XR Is More Than the Sum of AR, VR and MR 83

University, the famous film director Christopher Nolan urged the gradua- tes not to chase dream but to chase reality because our goal in life should be built upon our reality. Our precepted experience of the reality is subje- ctive and personal. Besides, what seems real to some can be experienced as unreal by others. Some people even believe that there are not just one rea- lity but multiple realities at work. In today vocabulary, the word reality itself often associates with many emerging media technologies that is experience- based and can simulate different realities. AR, VR, MR, and XR all deal with different creative and technical treatments of reality. These emerging techno- logies offer us new and different experiences in interacting with the reality or realities, which are separated by thin blurry line.

AR, VR, MR AS EXPERIENCE-BASED STORYTELLING MEDIA With interactive media such as AR, VR and MR, the sensual experiences of media content go beyond the usual sight and sound with more direct phy- sical experiences that can include touch, motion, dance and other forms of interactivity. Audience or user can participate and become co-creator of con- tent through interactivity. Interactive media can be described as lean-forward media, in which audience does not just lean back to receive the content pas- sively. It not only demands curiosity and attention from the audience or users; it also demands physical responses or participation that makes up the experience-based storytelling (Dubbelman 2016, Crawford, 2013, Koenitz, 2016). One primary objective of storytelling is to arouse curiosity and to simulate imagination or emotional feelings from the audience or users. For interactive media such as AR, VR or MR, pure emotional response is not enough. Response must also be in the form of physical experience and rea- ction. The designer of an interactive story must consider the participation of the audience or user as part of the storytelling experience (Miller, 2019, Yip, 2022, Vosmeer, Mirjam, and Ben Schouten, 2014). If more the better, experience-based interactive media can benefit from more immersive and more physical participation from the user or audience.

AR Storytelling AR first appeared as a device to overlay digital information or simple ani- mated graphics onto a real backdrop as an add-on feature. AR overlays the real world with digital information in real-time synchronization but limited interaction and occlusion. AR has become popular with the rising popularity of mobile smartphone and tablet. Unlike other experience-based media, AR requires no additional hardware other than a mobile smart device. Through simple in-camera scanning, wi-fi network and/ or sophisticated GPS conne- ction, digital magic can be captured and combined in any real location as backdrop. Several AR toolkits and many user friendly mobile applications have become available in open sources that facilitate AR creation.

In terms of using AR for storytelling, AR associates with location, which is one of the essential element of storytelling. Location can have unique cha- racteristics and can be narrative-rich. Azuma (2015) proposes taxonomy for location-based mixed and augmented reality storytelling. To re-skill or to

84 Yip

re-make reality by adding new meaning and experience to a location that can be “more powerful than the real location”. For example, Liestøl (2018) AR storytelling with mobile augmented reality on Omaha Beach provides a first-hand experience of the significant turning point of WWII that can- not be experienced from movie watching on the same topic. Nowadays, AR has been made easy and user-friendly with many pre-made assets with user-friendly click and drag function that add virtual content onto a loca- tion backdrop. AR storytelling is mobile, explorative, game-like, a personal media sharable in social media platforms. In addition to location, storytel- ling always involves character. The popular Pokémon GO game combines AR with gamification to reward players to catch pet creatures, train them for battle for level up gameplay. This game genre has inspired many similar story-based game of this location and character-based AR story game genre. The famous story franchise of Jurassic Park, The Ghostbuster stories combine education with entertainment or edutainment by building on this Pokémon GO collect-for-battle narrative model.

VR Storytelling VR environment is an open world not composed by shot or shot sequence. The strength of VR is its ability to immerse the users to a virtual world (Green- gard, 2019). Virtual reality VR provides a completed enclosed and immersive viewing environment in direct interactive control of the head, body and hands movement of a single VR user. Many VR research have focused on exten- ding senses into touch and smell and temperature in VR beyond sight and sound. VR creates a fully immersive experience in an enclosed virtual world separated from the real world. VR Story can be told from one or multiple POVs. Some VR projects can create empathy by letting users to immerse into the POV of other character in different roles and situations. VR can invite viewers to experience empathy through first person POV perspective with direct control of the head and body movement interacting with the environ- ment that conventional media cannot offer. This immersive quality of virtual storytelling fundamentally changes how a story can be told. It changes the relation between storyteller and story receivers at the other end. One misco- nception of VR is that it is 360-degree. We are limited by our own eyes’ field of view which is roughly less than 120 degrees. Aided our head movement, it is recommended that the best angle of view for VR is about 180–210 degree, which is not significant higher than viewing on a big or curved screen subject to the use of camera lens. Nevertheless, viewing on a big or curved screen is never immersive as compared to wearing a VR headset. Since VR can detect the position and head movement of the user and subsequently extend the content and the boundary of the virtual world accordingly, the field of view in VR is never 360 degree at any given time.

Visual aesthetics and the art of storytelling should be re-defined in the VR world. For example, shot composition aesthetics no longer need to consider composition confined by framing. The only frame is the boundary of our own field of view. For example, In VR, there is no need for close-up shot as the VR user can walk closer to the subject within reach. Although framing is

XR Is More Than the Sum of AR, VR and MR 85

limited by the boundary of our own field of view, the concept of foreground, middle-ground and background that separate the different zones of the image is still relevant. Nevertheless, some aspects of visual language of expression should be different from sequential imaging in conventional film language. Since actions are almost always continuous within a scene, shot-to-shot sequ- encing is no longer needed and continuous or discontinuous editing or jump cut effect is no longer relevant within a scene. Nevertheless, scene to scene transition as level transition is still relevant. The concept of on-screen or off- screen pace can be considered irrelevant as things that occur outside the field of view can be moved back to the field of view with a simple head turning gesture. Other forms and expressions of visual aesthetics such as color is still relevant and applicable in the VR environment. Color control such as con- trast and saturation etc. should be viewed and used as the same effect as in other form of visual media content.

The issue of authorship or the role of interaction in immersive and nar- rative experience in VR (Jenkins, 2004) presents an interesting challenge to VR storyteller and designer. What is considered the right balance between narrative and interaction to be embedded in the story? (Miller, 2019). How to evoke and embed narrative experience in an interactive and immersive media with cinematic style? (Yip, 2020) VR story audience and users can interact with the story artifacts that can alter the storytelling and the vie- wing experience. The early form of VR story game is described as walking simulator (Carbo-Mascarell, 2016) where user is free to explore the open story world and can discover and reveal plot information through voice-over and user’s interaction with the story artifacts in the virtual world. In terms of accessibility, unlike AR, VR requires special hardware that can be physi- cally demanding for some. The recommended time for VR content is typically about 30 min., shorter than other regular media platform. VR is known to make some people, especially elderly, feel dizzy or even motion sickness after wearing this heavy headset for a period. VR is therefore considered more popular among young generation.

MR as AR + VR and More Although AR and VR have evolved for decades, they can still be conside- red in infancy or experimental stage as storytelling media, particularly for MR and XR. Mixed or merged reality MR, which combines key features of AR and VR with holographic technology to mix and blend the physical world with digital objects together in real-time response to the tracking position and point of view of the MR user. MR is able to connect digital or virtual con- tent to the physical world with more precise interaction and occlusion. The technique of MR hologram works by recording 3D objects using a laser and then projecting and reflecting these laser lights from different angles through different surfaces in the process of light interference and diffraction in order to restore the recorded objects with 3D effect.

Other viewers need to wear a pair of 3D glasses to see the stereoscopic image displayed on the screen with tracking data being transmitted through the glasses. MR hololens can capture hologram using multiple optical and

86 Yip

light sensors on two sides to scan or sense the environment and blend holo- grams in the environment with a frontal depth camera to sense the hand motions when interacting with virtual holograms and real objects in the physical world where users can occlude and interchange with virtual world layers. Hololens glasses offer an unique mixed reality experience and seems to offer a solution to people who find VR headset repulsing for various reasons.

While VR can play pre-recorded 360 video overlayed with virtual images, it has no real-time high resolution image capture capability of the physi- cal environment. Although the latest VR headset already installs camera to scan the environment, their focus is not to blend virtual content to the envi- ronment. Unlike VR, MR turns a physical place into a semi-virtual reality overlayed with virtual content. The tracking function of AR adds the augmen- ted environment and location tracking feature to MR. MR can achieve both AR and VR effects with high precision and sophistication in the 360 virtual environment combining digital and virtual together into one environment with real-time interactivity and immersion. Advanced AR on wearable glas- ses with precision where the real world meets the virtual content. Due to its high cost, MR has not yet become popular. A typical price of a MR hololens is almost ten times more than the latest model of VR headset system.

Unlike AR and VR with their decades long development, MR is still in its infancy stage. Similar to the early development of VR applications, the applications of MR are more on training and education. MR is seen used in medical, construction and engineering trainings where precision is needed to illustrate and demonstrate the internal mechanism and system in 3D images and space with real-time interaction. It is anticipated in the foreseeable future that car will be equipped with some kind of MR technology can enable certain essential information, e.g. road map, direction etc., to be overlayed on the front windshield while not distracting the driver’s view and attention on the road.

Figure 1: Real-time integration and interaction of different video signals of different locations on one virtual stage.
Original PDF Figure 1:

XR and Metaverse Extended reality XR can be described as the combination of all the realities but more, where virtual content is extended beyond the ‘boundary’ with more real-time computing power (Kavakli & Cremona, 2022, Spielmann & Helzle,

XR Is More Than the Sum of AR, VR and MR 87

2018, Ye, 2022). LED-based Extended Reality (XR) is the latest cutting-edge technology that integrates systems of LED display, real-time render engines, studio camera tracking, lighting, and performance/ motion capture with ani- mated and/ or live action content. With camera tracking system, it can engage and immerse the audience with new experiences by extending the canvas infi- nitely in real-time. It blurs the lines between virtual and reality. This extended feature can add other systems, e.g. virtual camera and other interactive sen- sors to the extent that can have a role to play in the future development of metaverse for live and/ or recorded content.

Table 1. Comparison of AR, VR, MR and XR.

Table available in the original PDF paper.

Mobility Accessibility Character Location Interactivity Metaverse Yes No AR High High Virtual No occlu- sion

Yes No

VR Medium Medium Virtual Virtual 360 Yes Yes with avatar MR High Low Physical + vir- tual

Physical + vir- tual 360

XR Very Low

Very Low

Physical + vir- tual + high resolu- tion video

Physical + vir- tual through camera view

Limited Yes with avatar and human teleporta- tion

CONCLUSION

XR is sometimes confused with virtual production, in which the former tech- nology often associates with live performance and set extension and the latter often associates with filmmaking with real-time compositing. In a nutshell, XR extends the virtual set while in virtual production, what you see is what you get without the extension. XR is also often described as the sum of all different versions of AR, VR, MR because it can augment and mix realities altogether through extended virtual space. The popularity of LED walls have enabled XR to be more immersive. Depending on the size and resolution of the LED walls and other related hardware such as camera and real-time media server combined, LED-based XR is generally at least twenty to thirty times more than the cost of a pair of MR hololens. The high cost of LED-based MR and XR has made these two technologies less accessible as compared to other experience-based media. However, unlike AR, VR, and MR, XR can put a high resolution human images into the virtual world with real-time set extension and interaction, which might be merged into a new version of metaverse in the years to come.

REFERENCES

Azuma, R. (2015). 11 Location-Based mixed and augmented reality storytelling.

88 Yip

Carbo-Mascarell, R. (2016, August). Walking Simulators: The Digitisation of an Aesthetic Practice. In DiGRA/FDG. Crawford, Chris. (2013) Interactive storytelling. Routledge. Dubbelman, T. (2016, November). Narrative game mechanics. In International

Conference on Interactive Digital Storytelling (pp. 39–50). Springer, Cham. Greengard, S. (2019). Virtual reality. Mit Press. Kavakli, M., & Cremona, C. (2022, March). The Virtual Production Studio Concept–An Emerging Game Changer in Filmmaking. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 29–37). IEEE. Koenitz, H. (2016). Interactive storytelling paradigms and representations: a

humanities-based perspective. Handbook of digital games and entertainment technologies, 1–15. Liestøl, G. (2018). Story and storage–narrative theory as a tool for creativity in augmented reality storytelling. Virtual Creativity, 8(1), 75–89. Miller, C. H. (2019). Digital Storytelling 4e: A creator’s guide to interactive enter-

tainment. CRC Press. Spielmann, S., & Helzle, V. (2018). Augmented reality for virtual set extension.

In ACM SIGGRAPH 2018 Talks (pp. 1–2). Vosmeer, Mirjam, and Ben Schouten (2014). Interactive cinema: engagement and interaction. International conference on interactive digital storytelling. Springer, Cham. Ye, F. (2022, July). Image Art Innovation based on Extended Reality Technology.

In 2022 7th IEEE International Conference on Data Science in Cyberspace (DSC) (pp. 584–587). IEEE. Yip, D. K. M. (2020). Cinematic Surrealism of the Interactive Virtual Space. In Reconceptualizing the Digital Humanities in Asia (pp. 53–71). Springer, Singapore. Yip, D. K. M. (2022). Between Passive Viewing and Active Choosing in Storytel-

ling. Human Factors in Communication of Design, 49, 65.

Cover of the proceedings volume Human Factors in Communication of Design

Explore the complete digital volume

Human Factors in Communication of Design

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-66-7

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →