Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Holograms for Minimally Invasive Surgery Training and Planning

University of California San Diego, La Jolla, CA, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1004848

Abstract

The medical field has rapidly adopted Minimally Invasive Surgery (MIS) as a powerful alternative to traditional surgery as it reduces infection, pain, hospital stays, and costs. Despite these benefits, the in-depth knowledge of procedures coupled with the variety of tools used requires extensive training and technical planning for successful surgery. Prevailing Virtual Reality (VR) and Augmented Reality (AR) applications enable a broad spectrum of MIS training and pre-surgery planning. However, these currently existing systems are bulky, complex, and expensive and have had poor adoption rates in the field attributed to a lack of knowledge and financial barriers2. In this paper, we present a novel low-cost approach to develop a mobile hologram platform for MIS surgery training and potentially pre-surgery planning. Our developmental process consists of converting a patient's CT DICOM data to a 3D model using region of interest identification and volumetric or surface rendering. We prioritize realistic model texturing as the texture has a significant impact on visual perception and potential for use in medical diagnosis. The realistic 3D model can then be displayed on a variety of accessible devices such as mobile phones, 2D tablets, 3D tablets, and phone-based Google Cardboards or HUD headsets, all with their tradeoffs. The integration of models with devices can rely on different software for development, in our instances exploring the capabilities of Unity, Unreal, React, and XCode. Using this software, we then developed an overlay functionality that tracks aspects of the physical world and places the virtual 3D model on physical models of organs, mannequins, and/or live humans, allowing for real-world replications of medically relevant augmented scenes. An integral part of surgical practices is incisions that can cause blood flow. This is a feature we are mimicking with fluid dynamics and implementing using particle systems to yield a high-quality real-world medical training platform. To use these realistic models for MIS training, we developed specific controls based on ergonomics research that were suited for each platform. We experimented with approaches such as hand-tracking and touch-based screen gesture controls to tailor usability to device specifications. Finally, to improve perceptual adaptation, we focus on the disparate distance as it plays a critical role in user experience. Further development in user interactions and surgical design is required to allow for pre-surgical planning functionality on our applications.

INTRODUCTION

The medical field has rapidly adopted Minimally Invasive Surgery (MIS) as a powerful alternative to traditional surgery as it reduces infection, pain, hospital stays, and costs (Mayo Clinic, 2024). Despite these benefits, the in-depth knowledge of procedures coupled with the variety of tools used requires extensive training and technical planning for successful surgery. Prevailing Virtual Reality (VR) and Augmented Reality (AR) applications enable a broad spectrum of MIS training and pre-surgery planning. How- ever, these currently existing systems are bulky, complex, and expensive and have had poor adoption rates in the field attributed to a lack of knowledge and financial barriers (Balla, 2023). Shao-Hua et al. developed a PC-based laparoscopic surgery with 3D models of complex vascular structures, but the amount of memory and RAM required to achieve this reduces the usabil- ity. Another team developed an effective AR-based surgical planning tool for neurosurgery, but the tool was highly specific and thus would not transfer to a laparoscopic surgery setting (Coelho, 2020).

© 2024. Published by AHFE Open Access. All rights reserved. 151

This paper presents a novel low-cost approach to developing a mobile hologram platform for MIS surgery training and potentially pre-surgery plan- ning. The proposed system can be used alone with just a mobile phone or paired with advanced devices such as AR goggles with live data feed. Our developmental process consists of converting a patient’s CT data to a 3D model using region of interest identification and volumetric or sur- face rendering. We prioritize realistic model texturing as the texture has a significant impact on visual perception and potential for use in medical diagnosis. The realistic 3D model can then be displayed on a variety of accessible devices such as mobile phones, 2D tablets, 3D tablets, and phone- based Google Cardboards or HUD headsets, all with their tradeoffs. We then developed an overlay functionality that tracks aspects of the physical world and places the virtual 3D model on physical models of organs, mannequins, and/or live humans, allowing for real-world replications of medically relevant augmented scenes.

An integral part of surgical practices is incisions that can cause blood flow. This is a feature we are mimicking with fluid dynamics and implementing using particle systems to yield a high-quality real-world medical training plat- form. To use these realistic models for MIS training, we developed specific controls based on ergonomics research that were suited for each platform. We experimented with approaches such as hand-tracking and touch-based screen gesture controls to tailor usability to device specifications. Finally, to improve perceptual adaptation, we focus on the disparate distance as it plays a critical role in user experience.

FROM DATA TO MODEL The process of creating 3D models varies. To ensure accurate, realistic, and more importantly, personalizable models, we use software to extract 3D mod- els from DICOM data. DICOM data consists of ultrasounds, CT scans, and MRIs. Our primary focus is on CT scans and MRIs to capture the model from the most relatable MIS areas, using samples provided by 3D Slicer (Slice, 2024) and our patient data. The conversion itself requires analyzing the data sets and finding the poten- tial for an ideal model. An ideal model would be categorized as a crisp clear scan, available for viewing from three axes (upper left: axial, bottom left: coronal, bottom right: sagittal). The three axes are critical, as the modeling must be done from all angles. Building something three-dimensional with no information other than a two-dimensional view either misses length, height, or depth. Although these unideal scans would cause critically large differences between an individual and training model, and therefore not be considered, even the smallest differences are disadvantageous to a proper medical tool where precision is invaluable.

Narrowing down the volumes of interest can be challenging with the shad- ing of different components. Using tools Threshold and Grow from Seeds in 3D Slicer, we rely on the differences of tones to differentiate organs, bones, vessels, etc. When tones are similar, we might pick up extras unintentionally in the volume segmentation. Figure 1 shows the 3D volume rendering of a cavity and a 3D model export of an aorta with an aneurysm.

Holograms for Minimally Invasive Surgery Training and Planning 153

A large tradeoff for creating such precise modeling is file size. The exporta- tion of 3D models relies on vertices to replicate points and edges. A common and intuitive understanding of the relationship between precision and file size is that the more vertices and points of data required to store an entire model cause larger files. This is a hindrance to application programming and system performance. An equilibrium must be struck to maintain the integrity of the patient’s organs as shown by the model, while enabling fast load times for enhanced user experience.

Figure 1: The cavity (left) and aorta (right) extracted from the CT DICOM data.
Original PDF Figure 1:

TEXTURING It is normal to assign colors to individual organs. It would be helpful to sim- ulate the surface texture of an organ per symptoms, from healthy, inflamed, to cancerous. We first study internal organs to match their tones within the environment they would be found in for an MIS. We can reference a video of screenshots taken of a healthy liver to mimic the color and texture.

Considering different kinds of organ textures, e.g. sick, healthy, cancerous, etc., we can evolve the texture over time. Figure 2 shows the spectrum of kidney textures that can be rendered in three dimensions, extracted from Adobe Substance (Adobe, 2024). Figure 3 shows liver stages as healthy and cancerous.

Figure 2: The kidney textures extracted from adobe substance (Adobe, 2024).
Original PDF Figure 2:
Figure 3: The healthy liver (left) and the cancerous liver (right).
Original PDF Figure 3:

ANIMATED SIMULATION Animations can help the user understand the dynamic nature of surgery and improve surgical training and planning for disaster scenarios. We mimic the dynamics of surgery through a custom realistic bleeding simulation. Com- mon bleeding animations are geared toward graphic combat video games that exhibit exaggerated bleeding with a pixelated surface appearance. To mimic bleeding observed in surgery, we developed a bleeding animation that is controlled and interacts with 3D organ models.

The bleeding animation is based on a Unity engine particle simulation. This system is set to a 4,000 emission rate over zero distance for a burst-like effect, which emits particles in a box shape on the pixel scale of 1 by 1 by 1 with the parent object. The model contains physical parameters such as gravity that are used to modify the speed and direction of the particle flow. We tested this parameter with users until a realistic flow was achieved with gravity set to 0.09. The 3D virtual vascular structures are enabled with a mesh collider and a custom incision component, which enable the virtual surgical knife to penetrate the organ to cause bleeding. When an interaction with the surgical knife occurs on the structure, raycasting is used to locate the point in screen space. This point is set as the origin of the particle system and it is activated to start the bleeding.

SSF Pro Shader is used to render the fluid simulation. This method is ideal for our application because it is relatively lightweight compared to other fluid modeling approaches. It does not require rendering a mesh but instead uses depth and thickness information to create a smooth fluid-like appearance. Briefly, particles are rendered as spheres in the depth buffer, storing the dis- tance of each particle from the camera. Particles are also rendered as spheres in the thickness buffer to accumulate the thickness along the view direction. A bilateral filter is then used to smooth the depth buffer to maintain edges and remove noise. A curvature flow filter is also applied to reduce the blobby appearance. Next, finite difference is used to compute the normals of the sur- face from the depth buffer. The overall surface is shaded with the normals, thickness buffer, and blood particle color. This shader produces the dynamic fluid appearance of realistic blood flow over the transparent particle system.

When the flowing particles in the system collide with organs that fit with mesh colliders, they move around the objects according to particle interac- tions in fluid dynamics. This motion creates a realistic flow of blood into

Holograms for Minimally Invasive Surgery Training and Planning 155

the surgical cavity during MIS. The opaque nature of the fluid surface causes realistic occlusion from bleeding for enhanced disaster scenario training and surgical planning. Our combination of the specific 3D model of patient data with this realistic bleeding animation in a versatile application creates a highly effective and accessible tool for MIS training and planning.

Figure 4: Screenshots of the bleeding from the artery over time.
Original PDF Figure 4:

HOLOGRAPHIC OVERLAY Holographic overlay is the key feature of Augmented Reality. We can overlay 3D training objects with mobile devices such as phones, tablets, and goggles. The first step is to find flat planes within a scene. Then place objects onto those planes. We design the manual control interface allowing for movement, scale, and rotation of overlay. The challenge is always to match the lighting of the environment for objects because of the diversity of the environment. In general, the app prefers an environment with a large space and dimmed ambient lights but not a dark space.

We can simply overlay the 3D objects to the ambient environment on the phone screen. Figure 5 shows the holographic overlay on the 2D screen of a mobile phone. The user can move the phone around to see the details around the object, even inside.

Figure 5: Holographic overlay on 2D screen on the phone.
Original PDF Figure 5:

We can also overlay the 3D objects in stereo on a phone with a split screen. We then can have a stereo vision with a foldable stereoscopic lens clipped to the screen or insert the phone into an inexpensive Google Cardboard head- set. Figure 6 shows an example of the laparoscopic surgery table overlaid in stereo to a space in a Starbucks.

Perhaps, the most interesting application would be the 3D virtual objects that are overlaid with the physical objects in the real world. Figure 7 shows the virtual stomach is overlaid to the physically 3D printed digestive cavity with digitally rendered texture.

Figure 6: The stereo holographic overlay on the phone screen.
Original PDF Figure 6:
Figure 7: The holographic stomach overlaid on the physical simulated digestive system.
Original PDF Figure 7:

Holograms for Minimally Invasive Surgery Training and Planning 157

MANIPULATION It is desirable to manipulate the 3D objects by hand gestures. We designed several hand control methods: open-air gesture, touchscreen gesture, and Leap Motion gesture control. Hand tracking can be programmed to recog- nize hand gestures such as thumbs up or a pinch to trigger related commands. Figure 8 shows the hand tracking to flip the body from face down to face up.

Hand tracking is necessary for manipulating the virtual objects on the headset which further allows user interaction with virtual models. For 3D objects, the manipulation of the objects’ selection, location, orientation, and size is helpful to surgical training and preoperative planning.

For non-head-mounted holographic interfaces, for example, phones or tablets, the gestures for touch screens can be easily implemented. For exam- ple, the “IKEA movement”for moving the object closer or distant by dragging the object on the screen up or down.

Figure 8: Hand tracking using Apple Vision kit to control the avatar face up or down.
Original PDF Figure 8:

USABILITY The community of Extended Reality technology is rapidly growing. We can have apps on the phone, VR goggles, AR goggles, 3D tablets, 3D monitors, 3D projectors, Google Cardboards, etc. We are constantly surrounded by these devices. If we recall, 2017 was the year of 3D televisions. There were many 3D TV products at the Consumer Electronics Show (CES) that year. But now, nobody talks about 3D TV anymore. Why? There is very little 3D content on television programs. Therefore, the usability is very low and the concept of 3D TV faded away quickly.

Should we look into the usability of the holographic overlay interfaces? We can examine the fidelity of the display, the cost of the device, manipu- lation capacity, mobility, and interaction between the user and surrounding people. Table 1 summarizes the comparison of types of interfaces versus the assessment criteria.

The assessment is based on common sense because the facts are observable and obvious. We conclude that the holographic overlay apps on the phone are low in fidelity for 3D contents, however, it is affordable in cost with

moderate hand tracking gesture control, very high mobility, and interaction capacity. On the other side of the spectrum, the prevailing AR goggles are the most expensive devices with moderate fidelity due to the overlay interference. They have highly accurate hand, head, and even eye-tracking functions for object manipulation. However, their mobility is rather low, and interaction between users and the people is weak.

We found that 3D tablets could be a new trend for holographic overlay interfaces. They are moderate in fidelity, cost, and manipulation, but high in mobility, and interaction between users and people around. This is important to MIS training and presurgical planning because surgery is teamwork.

Table 1. Usability comparison for types of interfaces versus assessment criteria.

Table available in the original PDF paper.

Interfaces Fidelity Cost Manipulation Mobility Interaction Phone App Low Low Moderate High High VR Goggle High Moderate High Low Low AR Goggle Moderate High High Low Moderate 3D Tablet Moderate Moderate Moderate High High 3D Monitor High High Moderate Low Moderate 3D Projector Moderate Low Moderate Low Low Google Cardboard Moderate Low Low Moderate Low

CONCLUSION

This paper presents a novel low-cost approach to developing a mobile holo- gram platform for MIS surgery training and potentially pre-surgery planning. Our developmental process consists of converting a patient’s CT DICOM data to a 3D model using region of interest identification and volumetric or surface rendering.

We prioritize realistic model texturing as the texture has a significant impact on visual perception and potential for use in medical diagnosis. The realistic 3D model can then be displayed on a variety of accessible devices such as mobile phones, 2D tablets, 3D tablets, and phone-based Google Cardboards or HUD headsets, all with their tradeoffs.

We then developed animation for dynamic events such as bleeding, hand- tracking gesture-based object manipulation, and an overlay functionality that tracks aspects of the physical world, and places the virtual 3D model on physical models of organs, mannequins, and/or live humans, allowing for real-world replications of medically relevant augmented scenes.

Finally, we conducted a usability study for multiple holographic overlay interfaces. We found that the holographic overlay apps on the phone are low in fidelity for 3D contents, however, it is affordable in cost with moderate hand tracking gesture control, very high mobility, and interaction capacity. We also found that 3D tablets could be a new trend for holographic overlay interfaces. They are moderate in fidelity, cost, and manipulation, but high in mobility, and interaction between users and people around.

Holograms for Minimally Invasive Surgery Training and Planning 159

ACKNOWLEDGMENT

The authors would like to thank the support of the Wellcome Leap SAVE Program and the NIST PSCR Program.

REFERENCES

Adobe (2024). Substance 3D Substance 3D Community Assets (adobe.com). Alameddine, M. B. Claflin J, Scally CP, Noble DM, Reames BN, Englesbe MJ, et al.

(2015) Resident surgeons underrate their laparoscopic skills and comfort level compared to attending surgeons. J Surg Educ. 2015;72:1240–6. Angelov, V. Petkov, V. Shipkovenski, G. and Kalushkov, T. (2020) Modern Virtual Reality Headsets, 2020 International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA), Ankara, Turkey, 2020, pp. 1–5, doi: 10.1109/HORA49412.2020.9152604. Apple (2024). https://developer.apple.com/videos/play/wwdc2020/10653/ Balla, A. Sartori A, Botteri E, Podda M, Ortenzi M, Silecchia G, Guerrieri M,

Agresta F. (2023). ARMIS (Augmented Reality in Minimally Invasive Surgery) Collaborative Group (2023). Augmented reality (AR) in minimally invasive surgery (MIS) training: where are we now in Italy? The Italian Society of Endo- scopic Surgery (SICE) ARMIS survey. Updates Surg. 2023 Jan; 75(1): 85–93. doi: 10.1007/s13304-022-01383-6. Epub 2022 Sep 21. PMID: 36131182. Chen, H. Hesser, J. and Männer, R. (2003). Ray casting free-form deformed-volume objects. The Journal of Visualization and Computer Animation 14.2 (2003): 61–72. Coelho G, Rabelo NN, Vieira E, Mendes K, Zagatto G, Santos de Oliveira R, Raposo-

Amaral CE, Yoshida M, de Souza MR, Fagundes CF, Teixeira MJ, Figueiredo EG. Augmented reality and physical hybrid model simulation for preoperative plan- ning of metopic craniosynostosis surgery. Neurosurg Focus. 2020 Mar 1;48(3): E19. doi: 10.3171/2019.12. FOCUS19854. PMID: 32114555. Donatelli G, Mutter D, Dhumane P, Callari C, Marescaux J. (2012). Transhepatic metallic stenting for hepaticojejunostomy stricture following laparoscopic chole- cystectomy biliary injury: A case of successful 20 years follow-up. J Min Access Surg 2012;8: 99–101. Hamad A, Jia B. (2022). How Virtual Reality Technology Has Changed Our Lives: An Overview of the Current and Potential Applications and Limitations. Int J Environ Res Public Health. 2022 Sep 8;19(18):11278. doi: 10.3390/ijerph191811278. PMID: 36141551; PMCID: PMC9517547. Horos (2024). https://horosproject.org/ M, I. I. Butaslac, Y. Fujimoto, T. Sawabe, M. Kanbara and H. Kato (2023). Systematic

Review of Augmented Reality Training Systems, in IEEE Transactions on Visual- ization and Computer Graphics, vol. 29, no. 12, pp. 5062–5082, Dec. 2023, doi: 10.1109/TVCG.2022.3201120. Mayo Clinic (2024). Minimally Invasive Surgery. Minimally invasive surgery - Mayo Clinic. Mi SH, Hou ZG, Yang F, Xie XL, Bian GB. A 3D virtual reality simulator for training

of minimally invasive surgery. Annu Int Conf IEEE Eng Med Biol Soc. 2014;2014: 349–52. doi: 10.1109/EMBC.2014.6943601. PMID: 25569969. Müller, Matthias, David Charypar, and Markus Gross (2003). Particle-based fluid simulation for interactive applications. Proceedings of the 2003 ACM SIG- GRAPH/Eurographics symposium on Computer animation. 2003.

Nagendran M, Gurusamy KS, Aggarwal R, Loizidou M, Davidson BR. (2013).

Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev. 2013;2013: CD006575. Scott, D. J. Young, W. N. Tesfay ST, Frawley WH, Rege RV, Jones DB. (2001).

Laparoscopic skills training. Am J Surg. 2001;182:137–42. Shi, W. Gao, X. Lv, L. Pan, Z. Shao, J. (2021). A new geometric combination of cutting and bleeding modules for surgical simulation systems. Comput Meth- ods Programs Biomed. 2021 Jul;206:106109. doi: 10.1016/j.cmpb.2021.106109. Epub 2021 Apr 18. PMID: 33975182. Slicer (2024) https://www.slicer.org/ SSF (2024) Pro Unity Asset (https://assetstore.unity.com/ssfPro).

Cover of the proceedings volume Healthcare and Medical Devices

Explore the complete digital volume

Healthcare and Medical Devices

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-06-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →