1 Institute of Computer Science, Foundation for Research and Technology—Hellas (FORTH), 70013 Heraklion, Crete, Greece ↗
2 Institute of Computer Science, Foundation for Research and Technology—Hellas (FORTH), 70013 Heraklion, Crete, Greece; University of Crete, Department of Computer Science Heraklion, University of Crete 70013 Heraklion, Crete, Greece ↗↗
This paper discusses the design and development of CreteAR, a system that integrates Extended Reality (XR) with a transformable physical model of the island of Crete to enhance learning experiences. CreteAR features an XR application for handheld devices that overlays information on a scale model, which is equipped with mechatronics to elevate or conceal showcases containing culturally significant items. Users can interact with both the digital content and the physical model, activating the showcases and accessing information through a companion display. The paper explores relevant literature, analyzes similar systems, and details the design process, including user requirements and expert feedback. CreteAR was initially conceived as a prototype, but it holds the potential to contribute to the development of broader applications across various domains beyond the scope of Crete's geography and heritage. It was designed to deliver an immersive and hands-on learning experience, emphasizing social interaction and cooperative exploration, engaging users in dynamic and meaningful interactions.
In recent years, the integration of Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) technologies has revolutionized interactive experiences, offering opportunities to enhance learning and information exploration across diverse fields (Stefanidi et al., 2020). These technologies, collectively referred to as Extended Reality (XR), transcend traditional boundaries by merging physical and digital worlds, thereby enriching user interactions and immersing them in dynamic environments (Leonidis, Korozi, and Stephanidis, 2024). Alongside XR, tangible interaction plays a vital
CreteAR: Enhancing Learning Experiences Through Tangible Transformable Artifacts 1079
role in providing intuitive, inclusive, and engaging user experiences (Claes and Vande Moere, 2015). Rooted in the theory of embodied cognition, tangible interaction capitalizes on users’ motor skills and sensory perceptions to bolster comprehension and memory retention, fostering emotional connections and meaningful interactions with digital content (Chettaoui, Atia and Bouhlel, 2022). By enabling users to physically manipulate digital content, such interfaces offer a more instinctive and accessible mode of interaction, reducing cognitive strain and amplifying user engagement, while facilitating deeper learning experiences by affording hands-on participation, benefiting users from diverse backgrounds and abilities.
Despite the fact that there are several approaches that explore similar themes of integrating physical artifacts with digital augmentations to enhance learning and interactive experiences (e.g., interactive museum exhibits), only a limited number allow users to interact with both the artifacts and XR applications concurrently. In many cases, physical artifacts often remain stationary or rotate slowly within a pedestal or display case, with only minimal interaction options accessible, such as pressing buttons to activate audio or lighting effects (Bellucci, Diaz and Aedo, 2015; Duranti, Spallazzo, and Petrelli, 2024). In more advanced approaches users have the opportunity to engage with digital content related to the artifact through XR technology (Kyriakou and Hermon, 2019).
Aiming to bridge this gap, we developed CreteAR, a system featuring an XR application for handheld devices capable of overlaying information on a scale resin model of the island of Crete. This model was crafted not only to showcase the island’s geographical features but also to incorporate mechatronics capable of concealing and elevating cases housing culturally significant items, thus transforming it into a versatile artifact. Through the XR application, users can select which item of interest to explore, triggering the rise of the respective showcase, while digital information is presented both on the island and in the surrounding environment. Additionally, users can interact with the physical model itself, tapping to select a showcase and explore its content through the companion display positioned right above the physical model.
Overall, CreteAR aims to offer a contemporary platform for spatial exploration, allowing users to engage with the geography and cultural heritage of Crete in a hands-on, immersive manner. This work is an initial step toward exploring whether integrating dynamic physical elements, such as electromechanical showcases, can enhance the user experience. It also demonstrates how a transformable (i.e., shape-shifting) artifact, combined with XR technologies, can improve interactivity, encourage social engagement, and promote collaborative exploration. The goal is to provide a dual pathway of digital and physical interaction to enrich the learning experience, making the exploration process more engaging and effective.
This paper begins by presenting the latest relevant literature and analyzing related systems; it then elaborates on the iterative design process undertaken, including conceptualization, user requirements gathering, and incorporation of feedback from experts. Next, it describes the functionality of the
prototype in detail and concludes with a discussion aimed at inspiring further advancements and innovations in XR-enhanced edutainment applications.
LITERATURE REVIEW
In recent years, numerous research efforts and commercial applications have explored the integration of AR, VR, and MR technologies to enhance interactive experiences across various domains, particularly in education, cultural heritage, and entertainment. This section reviews the latest literature and related systems, focusing on how XR technologies and tangible interaction have been utilized to create immersive and engaging learning experiences.
XR Applications for Cultural Heritage Many museums and cultural institutions have embraced XR technologies to create interactive exhibits, enhancing visitor experiences with greater immersion and engagement. XR applications have also facilitated virtual reconstructions of historical sites and artifacts, enabling users to explore and interact with them in ways that surpass physical settings. For example, the ARCHEOGUIDE project (Vlahakis et al., 2002) provided an AR-based guide for archaeological sites, allowing visitors to view reconstructions of ancient ruins overlaid onto the current landscape. This approach not only enriched the educational value of the sites but also deepened appreciation for their historical significance.
The integration of VR and AR has significantly enhanced accessibility and engagement in cultural heritage by seamlessly integrating contextual information and interactive elements (Economou and Pujol, 2011). Studies by (Koutsabasis, 2017) further underscore this point, demonstrating that AR applications in museums enrich visitor experiences through the dynamic overlay of digital information onto physical exhibits, thereby enhancing the narrative and educational value of museum displays.
Moreover, XR technologies have been employed in advanced implementations to develop interactive exhibits that respond to user input, offering personalized experiences. The Museum of the Future1 in Dubai, for instance, employs a variety of XR technologies to create dynamic exhibits that adapt to the interests and actions of visitors, offering a highly personalized and interactive exploration of futuristic concepts and technologies. In a related study (Hammady, Ma and Strathearn, 2019), authors introduce an innovative spatial user interface methodology designed specifically for creating mixed reality applications in museums using HoloLens.
Tangible Interaction and Embodied Cognition Tangible interaction leverages physical objects to interact with digital information, offering an intuitive and engaging mode of interaction that aligns with the principles of embodied cognition. According to (Antle, Corness and Droumeva, 2008), tangible interfaces can improve learning
1https://museumofthefuture.ae/en
CreteAR: Enhancing Learning Experiences Through Tangible Transformable Artifacts 1081
outcomes by allowing users to manipulate physical objects, thereby fostering a deeper understanding of abstract concepts through hands-on activities. The work in (Marshall, 2007) further supports this notion, arguing that tangible interaction can enhance collaboration and communication among users, making it particularly effective in educational and collaborative settings.
In museums, tangible interaction has been integrated into exhibits to enrich visitor experiences with cultural artifacts. For example, recent studies (Marshall et al., 2016) explore the integration of tangible smart replicas in museum exhibitions. These replicas augment visitor interactions by offering additional layers of interactive content beyond traditional factual information associated with museum objects. Another approach involves interactive plinths (Not et al., 2019) equipped with RFID technology and sensors within a flexible IoT infrastructure. These plinths feature designated areas where museum objects trigger multimedia content—audio narrations, animations, and textual descriptions—providing contextual insights into the objects’ history and usage when placed by visitors.
Research into Tangible Augmented Reality (TAR) interfaces, such as Postcard AR and CubeMuseum AR (Ningning Xu and Liang, 2024), has demonstrated their effectiveness in enhancing learning about cultural artifacts compared to traditional methods like printed leaflets. These interfaces leverage tangible interaction to improve user engagement and facilitate deeper learning experiences in cultural heritage contexts.
Furthermore, Augmented Reality in combination with Tangible interaction has been employed to enhance physical surfaces, such as tables with 2D printed maps, using plain paper as interactive screens (Grammenos et al., 2010).
Discussion
The literature demonstrates significant advancements in creating immersive and interactive learning experiences across various domains, especially in cultural heritage. Despite these advancements, a critical gap remains in the simultaneous, seamless interaction with both physical artifacts and their digital augmentations. Current systems often limit physical interaction to simple gestures, such as pressing buttons to trigger audio or lighting effects, while the digital content is primarily explored through separate AR or VR interfaces. More sophisticated setups allow users to engage with digital content through XR technologies, but they do not fully integrate this with tangible interactions involving the physical artifacts themselves.
This gap is where CreteAR aims to innovate. By integrating dynamic physical elements, such as electromechanical showcases that reveal and conceal culturally significant items, CreteAR merges tangible interaction with XR technologies in a novel way. The system allows users to physically interact with a scale model triggering mechanical movements and revealing hidden artifacts, while simultaneously receiving augmented digital information on both the model and the surrounding environment. This dual interaction pathway aims to enrich the user experience and also promote a deeper, more engaged form of learning through hands-on manipulation and immersive digital augmentation.
Furthermore, CreteAR aims to enhance social interaction and cooperative exploration by allowing multiple users to engage with the physical model and its digital augmentations simultaneously. This approach fosters collaborative learning and discussion, leveraging the principles of embodied cognition to support more meaningful and memorable educational experiences.
SYSTEM FUNCTIONALITY AND FEATURES The CreteAR system consists of two main components: a Shape-Shifting Scale Model of Crete and an AR Application. The model serves as the physical centerpiece, representing the geography and cultural landmarks of the island. Meanwhile, the AR application enables users to explore information in a highly engaging and interactive manner.
Shape-Shifting Scale Model of Crete The Shape-Shifting Scale Model of Crete (Figure 1) is an abstract yet engaging representation crafted to facilitate exploration of the island’s geography and cultural heritage. It was designed as a 1.70m x 0.70m x 0.72m table so as to invite multiple users to gather around and engage with it simultaneously. The tabletop is made of colored resin, featuring varying shades of blue to symbolize different depths of the surrounding sea. Resting on this resin surface is a finely cut wooden silhouette of Crete, which is designed to mimic the general shape and contours of the island. The wooden base serves as a canvas for additional artistic elements, where smaller pieces of wood are stacked to create an abstract interpretation of Crete’s mountainous terrain. This layered approach aims to evoke a sense of the island’s topography rather than replicate it realistically, providing a unique and artistic perspective. LED lighting encircles the perimeter of the wooden base, aiming to highlight key features and create visual interest. These lights can serve both practical and aesthetic purposes, guiding users’ attention and enhancing the overall presentation of the model.
Figure 1: The scale model of Crete (left), a 3D representation of the elevating showcase mechanism (middle), a photo of the actual construction of the elevating showcase mechanism (right).
CreteAR: Enhancing Learning Experiences Through Tangible Transformable Artifacts 1083
The construction includes four openings along its length, each housing a plexiglass showcase on top of sophisticated mechatronics. These showcases include culturally significant items specific to their respective regions. For instance, the showcase representing Chania Prefecture features a model of the Kri-Kri goat, endemic to Crete. In Rethymno Prefecture, a bronze olive branch symbolizes the area’s ancient olive trees. Heraklion Prefecture houses a replica of the famous Phaistos Disc, and Lasithi Prefecture includes a model windmill, typical of the region’s landscape.
The showcases are also equipped with touch-sensitive technology, allowing visitors to trigger them to open by pressing on their top. When a showcase elevates, it reveals the exhibit inside, and simultaneously, the companion display above the physical model and the AR application present detailed information about it.
AR Application The AR application is currently designed for handheld devices, such as smartphones and tablets, with plans to expand support to head-mounted AR devices. It is built using Unity2 and incorporates Vuforia’s3 AR packages to gain essential AR functionalities. We used LiDAR technology to scan the model and its surrounding environment, allowing the application to detect the model and accurately indicate interactive areas in AR. This scanning process also enabled the precise placement of virtual models, both in small scale on the model’s surface and life-size around it. By integrating this data within Unity using Vuforia packages, we managed AR tracking, interactions, and developed the application’s AR features. Unity’s tools and components were essential for creating a responsive user interface optimized for touch-based interaction on handheld devices.
When users launch the AR application on their device, the experience begins with the camera scanning the surroundings to locate the physical scale model of Crete. A pulsing indicator on the screen informs users that the application is actively scanning. Once the model is successfully detected, its digital representation overlays onto the physical model in real-time, accompanied by audio feedback and hidden addressable LED strips lighting up to highlight regions of interest.
Digital indicators also appear at specific locations on the model, indicating interactive elements or showcases containing culturally significant items. As users approach these indicators, UI elements appear on the screen, allowing them to activate the mechanical showcases. Upon activation, each showcase unveils a various information such as textual descriptions, images, animations, 3D models, and interactive elements. This digital content can either float within the AR environment around the scale model or be precisely placed on the model itself or nearby surfaces (as depicted in Figure 2). For example, selecting the showcase featuring the Cretan wild goat triggers 3D animals to roam areas where encounters are likely. At the same time, a life- sized virtual 3D goat appears next to the user and begins walking around, allowing for close interaction and detailed exploration.
Throughout the interaction, the AR application continuously monitors the environment and the physical scale model of Crete. This ensures that the digital content remains synchronized and responsive to the user’s movements and interactions.
DESIGN PROCESS CreteAR was designed for the “Intelligent Museum”in vitro simulation space at the AmI Facility (http://ami.ics.forth.gr/) within the Institute of Computer Science of the Foundation for Research and Technology – Hellas (FORTH- ICS). The design followed the AmI-Design Process (Stephanidis et al., 2024) – an iterative methodology for designing for Intelligent Environments, which is rooted in Design Thinking principles (Brown and others, 2008). This process encompasses the design of Intelligent Spaces (e.g., Intelligent Homes), their associated intelligent artifacts (e.g., smart coffee tables), and related applications. It involves several stages: Understand, Define, Ideate, Filter & Plan, Design, Implement, and Test. When designing an artifact such as CreteAR, once these stages are complete, it can be integrated into the intended IE and become a functional component of the broader intelligent ecosystem.
In a series of meetings with the design team (including analysts, designers, engineers, and artists) and several potential end users (both male and female, aged 20 to 45, with no disabilities), scenarios and personas were developed for the “Understand” and “Define” steps of the process. Subsequently, multiple brainstorming sessions were organized for the “Ideate” phase. Numerous ideas were produced and then filtered through interviews with domain experts (e.g., experts in automation and robotics, industrial designers). This process led to the exclusion of unfeasible ideas, like those involving extremely sophisticated robotics or impractical designs and shapes or inappropriate materials for creating the desired scale model. Experienced
CreteAR: Enhancing Learning Experiences Through Tangible Transformable Artifacts 1085
interaction designers and developers reviewed the remaining ideas, offering valuable insights and preferences for the most promising concepts in terms of innovation, research interest, and user appeal.
The design phase of CreteAR was a comprehensive and detailed process, starting with the creation of the scale model. Key aspects considered included:
Level of Detail: Determining how much detail to incorporate in the scale model was crucial. This involved deciding on the precision of the island’s contours, the representation of its topographical features, and the inclusion of cultural and historical landmarks. A minimal level of detail was intentionally chosen to ensure that the physical model complemented rather than overshadowed the AR experience.
Selection of Materials: The choice of materials was essential for both aesthetic and functional purposes. Colored resin was selected for the tabletop to depict varying sea depths, while wood was chosen for the island’s silhouette and mountainous terrain, balancing durability and visual appeal.
Size of the Model: The dimensions of the model (1.70m x 0.70m x 0.72m) were chosen to ensure it was large enough to be detailed and engaging, yet small enough to fit comfortably within the designated exhibition space and allow multiple users to interact with it simultaneously.
Positioning of the Showcases: The placement of the four showcases along the length of the table was strategically planned to ensure easy access and visibility for users. The positioning also needed to balance the overall aesthetic and functional design of the model.
Lifting Mechanisms: The design also included the engineering of the sophisticated mechatronics for the touch-sensitive, elevating showcases, ensuring smooth operation and durability.
Shape and Size of the Showcases: The showcases feature a transparent, curved design for visibility from all angles and a lightweight construction to ease the demands on their lifting mechanism. Within each case, a top- mounted light source ensures optimal artifact illumination and a hidden touch-sensitive mechanism permits interactive engagement. To achieve a sleek and futuristic design, the showcase makes the artifact appear to float when raised, while its upper part seamlessly integrates with the surrounding topography when closed.
The Design phase continued with the prototyping of interfaces for the AR application that would accompany the physical model. Prior to implementing the AR application, User Experience (UX) experts assessed these prototypes, offering valuable feedback and identifying usability issues early in the design process. The revised prototypes were subsequently implemented by the development team, incorporating the feedback and refinements suggested during the evaluation phase. The next step involves conducting user-based evaluations to assess the effectiveness and UX of the final implementation, ensuring that the artifact meets its design goals and provides a seamless, engaging interaction for end users.
CONCLUSION AND FUTURE WORK Initially conceived as a demo prototype, CreteAR shows promise for enhancing other domains that could benefit from its concept. Beyond its focus on Crete’s geography and heritage, the principles and technologies demonstrated by CreteAR could be adapted to other educational, cultural, and interactive contexts. Initial presentations at national exhibitions have highlighted the application’s capacity to captivate and engage diverse audiences. These early interactions suggest that CreteAR could be an innovative tool that not only bridges the gap between physical and digital realms but also stimulates curiosity and learning, and promotes active engagement.
The next phase involves conducting comprehensive user-based evaluations to assess the effectiveness and user experience of the final implementation. Insights gained from these evaluations will be crucial in refining the application and exploring its potential for wider application, ultimately advancing interactive learning and digital engagement in diverse fields.
In addition to mobile platforms, we plan to extend CreteAR’s compatibility to include AR-capable HMD (e.g., Microsoft HoloLens 2, Oculus Quest 3). This development will enable users to experience CreteAR in a mixed reality environment, allowing for hands-free interaction and a more immersive exploration of the digital and physical elements. The HoloLens integration will take advantage of its advanced spatial mapping and gesture recognition features, further enhancing the interactive and educational potential of CreteAR.
ACKNOWLEDGMENT
This work has been supported by the FORTH-ICS internal RTD Programme ‘Ambient Intelligence and Smart Environments’.
REFERENCES
Antle, A. N., Corness, G. and Droumeva, M. (2008) ‘What the body knows: Exploring the benefits of embodied metaphors in hybrid physical digital environments’, Interacting with Computers, 21(1–2), pp. 66–75. Available at: https://doi.org/10.1016/j.intcom.2008.10.005. Bellucci, A., Diaz, P. and Aedo, I. (2015) ‘A See-Through Display for Interactive
Museum Showcases’, in Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces. New York, NY, USA: Association for Computing Machinery (ITS ‘15), pp. 301–306. Available at: https://doi.org/10. 1145/2817721.2823497. Brown, T. and others (2008) ‘Design thinking’, Harvard business review, 86(6), p. 84. Chettaoui, N., Atia, A. and Bouhlel, Med. S. (2022) ‘Examining the Effects of Embodied Interaction Modalities on Students’ Retention Skills in a Real Classroom Context’, Journal of Computers in Education, 9(4), pp. 549–569. Available at: https://doi.org/10.1007/s40692-021-00213-9. Claes, S. and Vande Moere, A. (2015) ‘The role of tangible interaction in
exploring information on public visualization displays’, in Proceedings of the 4th International Symposium on Pervasive Displays, pp. 201–207.
CreteAR: Enhancing Learning Experiences Through Tangible Transformable Artifacts 1087
Duranti, D., Spallazzo, D. and Petrelli, D. (2024) ‘Smart Objects and Replicas: A Survey of Tangible and Embodied Interactions in Museums and Cultural Heritage Sites’, J. Comput. Cult. Herit., 17(1), p. 7:1-7:32. Available at: https://doi.org/10. 1145/3631132. Economou, M. and Pujol, L. (2011) ‘Evaluating the use of virtual reality and
multimedia applications for presenting the past’, in Handbook of Research on Technologies and Cultural Heritage: Applications and Environments, pp. 223–239. Available at: https://doi.org/10.4018/978-1-60960-044-0.ch011. Grammenos, D. et al. (2010) ‘PaperView: augmenting physical surfaces with location-aware digital information’, in Proceedings of the Fifth International Conference on Tangible, Embedded, and Embodied Interaction. New York, NY, USA: Association for Computing Machinery (TEI ‘11), pp. 57–60. Available at: https://doi.org/10.1145/1935701.1935713. Hammady, R., Ma, M. and Strathearn, C. (2019) ‘User experience design for mixed
reality: a case study of HoloLens in museum’, International Journal of Technology Marketing, 13(3–4), pp. 354–375. Available at: https://doi.org/10.1504/IJTMKT .2019.104600. Koutsabasis, P. (2017) ‘Empirical Evaluations of Interactive Systems in Cultural Heritage: A Review’, International Journal of Computational Methods in Heritage Science (IJCMHS), 1, pp. 100–122. Available at: https://doi.org/10. 4018/IJCMHS.2017010107. Kyriakou, P. and Hermon, S. (2019) ‘Can I touch this? Using Natural Interaction in
a Museum Augmented Reality System’, Digital Applications in Archaeology and Cultural Heritage, 12, p. e00088. Available at: https://doi.org/10.1016/j.daach. 2018.e00088. Leonidis, A., Korozi, M. and Stephanidis, C. (2024) ‘Enhancing User Experience with Tangible Interactive Systems through Extended Reality.’, ERCIM News, 2024(137). Marshall, M. T. et al. (2016) ‘Using Tangible Smart Replicas as Controls for
an Interactive Museum Exhibition’, in Proceedings of the TEI ‘16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction. New York, NY, USA: Association for Computing Machinery (TEI ‘16), pp. 159–167. Available at: https://doi.org/10.1145/2839462.2839493. Marshall, P. (2007) ‘Do tangible interfaces enhance learning?’, in Proceedings of the 1st International Conference on Tangible and Embedded Interaction. New York, NY, USA: Association for Computing Machinery (TEI ‘07), pp. 163–170. Available at: https://doi.org/10.1145/1226969.1227004. Ningning Xu, L. Y., Yue Li, Xingbo Wei, Letian Xie and Liang, H.-N. (2024) ‘CubeMuseum AR: A Tangible Augmented Reality Interface for Cultural Heritage Learning and Museum Gifting’, International Journal of Human– Computer Interaction, 40(6), pp. 1409–1437. Available at: https://doi.org/10. 1080/10447318.2023.2171350. Not, E. et al. (2019) ‘Digital Augmentation of Historical Objects Through Tangible
Interaction’, J. Comput. Cult. Herit., 12(3). Available at: https://doi.org/10.1145/ 3297764. Stefanidi, E. et al. (2020) ‘MagiPlay: An Augmented Reality Serious Game Allowing Children to Program Intelligent Environments’, in M. L. Gavrilova et al. (eds) Transactions on Computational Science XXXVII: Special Issue on Computer Graphics. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 144–169. Available at: https://doi.org/10.1007/978-3-662-61983-4_9.
Stephanidis, C. et al. (2024) ‘Design for Intelligent Environments’, in Human- Computer Interaction in Intelligent Environments. CRC Press. Vlahakis, V. et al. (2002) ‘Archeoguide: An augmented reality guide for archaeolog
sites’, Computer Graphics and Applications, IEEE, 22, pp. 52–60. Available at: https://doi.org/10.1109/MCG.2002.1028726.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Design, Engineering, and Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-35-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional