Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Visualization of Additive Manufactured Orthosis via Augmented Reality

Publication year
Digital Object Identifier10.54941/ahfe1004037

Abstract

Virtual try-on applications came to market not too long ago to be a supportive part of online shopping, allowing consumers to try clothes or accessories on before making the purchase. This kind of applications proved to be highly useful in decreasing return rates of the products. More and more online markets are using AR technology to improve the shopping experience of consumers in fashion field. However, AR technology should be able to do more than making products more attractive, especially in the medical field. Finding an orthosis product that both satisfies the medical requirement and has an appealing design has always been hard for patients with physical inconvenience, especially during corona time. Traditionally, consumers first need to make a mould in a physical orthosis store, then they have to make multiple trips to the store to keep giving suggestions to the staff of how the orthosis needs to be adjusted, until they are satisfied with the final product. This process takes a long time, since orthosis are usually hand-made, and the adjustment process could be frustrating for the orthosis consumers. To go to the physical store, orthosis consumers have to tolerate the time spent on the way, limited choices of design to try on in each store, and the risk of corona infection due to close contact with people over there. The producers also have to cover for the wear-out samples, the time it takes for multiple discussion on improvements, as well as taking the risk of the patients not liking the design after production. A virtual orthosis try-on application is the best solution to satisfy both sides, it bridges the gap between orthosis consumers and the producers. This application would make individualization of orthosis products easier, saving the time of communications and possible miscommunications. Moreover, this app make the digitalized production line to be more appealing and trustworthy to orthosis consumers. The orthosis model is automatically generated with the 3D scan of the consumers' leg, then the consumers use this application to try on different designs and colours, and the final product can be additive manufactured after they made their choice. In this way, consumers do not need to go to the physical store anymore, saving much time and effort. The try-on application was first designed and assigned functions that it is supposed to achieve using design thinking methods, then designed and build with game engine Unity, which has many ready-to-use packages, together with the tracking technology from the Vuforia engine. Both Marker-less and Marker-based tracking were tried out during the development. Marker-based tracking were chosen in the end for a more satisfying end-product. Finally, a preference test was made for a better version of this application with a controlled group.A marker-based Android application was successfully made after many trails. Consumers only need to place two QR codes on both sides of their thighs, then they can view the augmented orthosis from different angles on the camera of a personal Android device, choose the design patterns and colours of the orthosis model by simple clicks on the user interface, and select the model that attracts them the most.

INTRODUCTION

The concepts of Augmented Reality (AR), Virtual Reality (VR), and meta- verse has been mentioned several years ago, but it has not been widely accepted by the public. Things have changed wildly since the breaking out of the coronavirus, people are forced to find remote solutions for their daily needs. These solutions revolutionized the way people view all these technolo- gies, thus more and more applications came to the market and most of them make our life more flexible and more efficient. However, this technology has rarely been used to digitalize the production line of medical device. This paper presents the development steps of an orthosis try-on application, which bridges the communication gap between manufacturers and the consumers, making both the production and the purchase much easier. A design thinking framework from Stanford is implemented in this study, which consists of five steps: Emphasize, Define, Ideate, Prototype, Test (Ashtari et al., 2020).

Fundamentals of AR Technology Most people would get confused by the term AR, VR, and Augmented Virtu- ality (AV). VR builds a completely immersive environment for the users, and

© 2023. Published by AHFE Open Access. All rights reserved. 305

the real world cannot be seen, while AR brings virtual information or objects into the user’s real surroundings, thus enhancing the user’s ability to gather information from the environment. AV is something between AR and VR, but is closer to VR, while AR is closer to reality (Carmigniani et al., 2011).

There are three kinds of tracking for an AR system: marker-based tracking (MB), markerless tracking (ML), and hybrid tracking. MB tracking uses the camera to detects the feature of a fiducial marker, and overlay the augmented object on top of it, whereas ML tracking use one or multiple objects them- selves as tracking targets and overlay augmented objects on them. Hybrid tracking it the method that combines both MB and ML tracking in a creative way and would not be discussed in detail here.

AR technology has proven itself to be able to increase purchase behaviour, mainly because of three features. First of all, it brings the virtual world to the real world, making the shopping experience more distinguishable, it bridges the Gap between the virtual product and the real user. Second of all, the con- sumers can interact with the virtual objects in real-time, making the products more attractive and interesting, and the features of the product would also seem more convincing. Third of all, the virtual objects could be attached to a certain position in 3D, allowing the consumers to have the illusion that they are actually trying the product on or interacting with it in other ways, making the consumers feel more present with the product, and making the product seems more personalized, thus more attractive (Smink et al., 2020).

STATE OF THE ART The first step of design thinking is to emphasize with the users, which means that through observation, researching and questioning, developers learn to look at things from the user’s perspective so that the user’s needs could be more easily found out (Ashtari et al., 2020).

Traditionally, orthosis making is a long process of try-on and adjustments, which means a lot of travels to the physical store. Therefore, it would be a great relief if the orthosis model could be automatically generated with a 3D scan of the patients’ leg, and if the patients can select the design of the orthosis remotely.

The second step of design thinking is to define the problem, so that the developers can know which features of the application they should be focus- ing on to solve problems from users (Ashtari et al., 2020). In this scenario, the problem of orthosis selection and the long time that it takes for orthosis consumers to get the final product need to be solved.

After the problem is defined, it is time to brainstorm for ideas. Some AR applications in the fashion industry were reviewed for this purpose.

The application developed by Kjærside et al. is based on marker-based tracking technology. After the consumers have selected the clothes, they can place them on the bar in a closet to the left of the mirror and put their phone in the box on the right side. The computer would detect the RFID tag hanging on the clothes and receive data of the consumer’s body measurements from the mobile application. The image on the virtual mirror then combined this information together with the real-time data from the camera and gives the

Visualization of Additive Manufactured Orthosis via Augmented Reality 307

illusion to the consumer that he or she is having the augmented clothes. Three fiducial markers are placed on the consumers’ upper body for better tracking (Kjærside et al., 2005).

Parth et al. developed an Android AR watch try-on application. The tablet camera can detect the marker placed on the consumer’s wrist, and the consumer can go through different models on the user interface, switch to different models, or change the colors of the watches according to their preferences. The augmented watch will be placed on their wrist covering the marker. ML tracking is more challenging to implement than marker-based tracking since more features are required to make the estimation of where the target object is (Yang et al., 2014). Developers such as Philipp Presle uses the Kinect, a depth-recognizing camera to develop a virtual mirror (Presle, 2012). Yang et al. proposed a hybrid tracking also with Kinect, but with three different modes: only MB tracking, only ML tracking, or both of them combined (Yang et al., 2014). The JD online shopping platform has adopted a virtual shoe try-on system for online shoe-selling called ARshoe, which is a complex system that contains an encoder, a decoder, and a processing phase (An et al., 2021).

METHODOLOGY

Analysis and Selection of SDK After the Ideate phase, a prototype needs to be sketched out. The prototypes should then be tested either through experiment or through interviews and discussions, before an end-product could be generated. To test the orthosis try-on application, meetings with several fellow students were arranged, and valuable suggestions for building this application were gathered from them (see Table 1), such as building platforms or useful software development kits (SDKs).

Table 1. Comparison of popular SDKs (modified) abbreviations: Comm-Commercial, F-Free, P-Paid, O-Open-source, A-Android (Dima Titov 2020; Amin and Govilkar 2015; Sanket Prabhu 2017).

Table available in the original PDF paper.

License type Supported platform

HMD support

Unity support

Cloud recognition

Edge-based tracking Vuforia Comm, F/P A, iOS Yes Yes Yes Yes ARKit F iOS Yes Yes No No ARCore F A Yes Yes No No ARToolkit O A, iOS Yes Yes No No Kudan Comm, F/P A, iOS No Yes Yes Yes Wikitude Comm, F/P A, iOS Yes Yes Yes Yes Maxst Comm, F/P A, iOS Yes Yes Yes Yes Metaio Comm, F/P A, iOS Yes Yes No Yes

Based on the criteria in the table above, Vuforia was selected as the best SDK to use in this case, with Unity as the editor.

Unity Development Engine Unity 3D as a multi-platform development engine is usually used for game design. It is free to use most of the basic functions, and it is also possible to make upgrades with payments. To save time for the game developers, it has a clean and structured interface, a useful and multi-functional toolset, a play scene to test the game within the development process, and most important of all, Unity 3D provides a library of native plug-ins called Asset Store, which allows the developer to import functionality, 3D models or properties from third-party companies, making the interactions in games more diverse and interesting, at the same time easy to implement. Unity also provides a large community for users to discuss new ideas or problems that come up dur- ing the development phase, which helps developers with similar problems (Tetiana A. Vakaliuk and Svitlana I. Pochtoviuk, 2021).

Most of its plug-ins and functions could also be applied for general application development, and it is especially suitable for AR application development because of its on-the-fly play testing and the feature-rich edi- tor which generates the play scenes (Günes et al., 2015). Within the editor, Developers can generate or import objects of various formats, set properties to the objects, change the materials or add sounds to the objects. The orthosis models generated by Ren were imported and modified with these functions of Unity. To make the objects move or interact with the users as the developers want, the developers could use C#, JavaScript, or Boo as the programming language. The orthosis try-on application in this study used C# scripts to create the interactions.

Vuforia Platform The Vuforia platform is currently one of the most popular Tools for AR appli- cation development. It supports all the major platforms like Android and iOS and proposed a powerful plug-in to use its functions in Unity. Most of its fea- tures are free to use, and an update of plug-ins costs 99$ per month (Tetiana A. Vakaliuk and Svitlana I. Pochtoviuk, 2021). The platform offers efficient, up-to-date, and reliable algorithms for computer vision problems such as image recognition or 3D object tracking, which saves AR application devel- opers from a lot of technical constraints. It also provides a clearly structured User Interface, which allows the developer to know which tool to use in no time. The SDK also supports 3D tracking, multi-target tracking, cylinder tar- get tracking which tracks objects of cylindric shape, frame marker tracking, etc. (Amin and Govilkar, 2015).

From the design point of view, on the one hand, MB tracking is easier to implement, but the patients might find the markers troublesome. On the other hand, ML tracking requires more effort to develop, but requires fewer actions from the consumers’ side. The development of the orthosis try-on application takes therefore both MB tracking and ML tracking into consideration, in the hope that a better end-product could come out.

The development of MB version is planned to have the following steps: Firstly, fiducial markers will be generated and set up in Unity. Secondly, the orthosis models are adjusted and placed on the marker accordingly. Thirdly,

Visualization of Additive Manufactured Orthosis via Augmented Reality 309

a user interface will be designed in Unity, Fourthly, the application will be built through Unity, and last but not least, an interview will be conducted for the validation of this application. Because the consumers of orthosis usually cannot move around as freely as the consumers of regular shoes or clothes, it is important to make the try-on procedure as simple as possible. During the development phase, two different ways of using the marker were designed and were compared during the interview for a version selection.

Vuforia supports model target tracking, which was used for the ML version of this application. This way of tracking imports the 3D model of the object to be tracked into the model target generator from Vuforia, and the 3D model works the same way as the fiducial marker for the camera to keep track of. After the 3D model of the consumer’s leg is detected, the Orthosis model would be augmented on the outside of the leg. In this way, no fiducial markers would be used, and it would provide more comfort for orthosis consumers. The development procedure is planned to be the same as the MB version of this application, except that the 3D model of the leg must be generated and used to train the Tracker of Vuforia Engine.

RESULTS

In the design thinking process, the prototype sketching step and the testing step should be repeat in turn multiple times for a better result. After the general development steps were planned and validated by fellow students, the application prototype was developed, and a preference test interview was made with 7 orthosis manufacturers at the Düsseldorf medical trading fair.

Design After the orthosis model is generated by a parametric design tool with the medical scan of the patient’s leg (Pan Ren, 2022), the application helps the user to individualize the model, so that it could be additive manufactured and delivered to the patients. The application program flow is divided into 3 steps. First, the patients can select the model that the doctors recommended, then the design pattern of the lattice that they like, and lastly the colour of the orthosis model that looks the best on their leg.

Development After the literature review, the MB version was first developed. Since the camera view is limited, it is hard to give a 360

  • view of the orthosis model for the patients with only one marker. Thus, two markers were used at first, one on the outer side of the patient’s leg. And the other marker is placed on the inner side. This version is named the “2 Code Marker-Based” version (2CMB) for this study. Later, it was modified to require only one marker, and this version is referred to as the 1CMB version in this study. After many trials of scanning a human leg and generation of a model target, the end-product of ML tracking was not so promising, thus this version was discarded in the end.

First of all, the marker needs to be generated and imported. An online Marker generator was used for this purpose. For feature tracking with higher

accuracy, two 5*5cm black-and-white QR codes were generated and were used as the fiducial markers to be tracked.

After Unity was successfully downloaded and installed, the Vuforia plug- in could be found in its Asset Store and could be added it to Unity. To make this plug-in work, it has to be installed in this project first, and the building settings of Unity need to be changed accordingly. For example, the normal camera of the default Unity setting needs to be replaced with the AR camera. All 12 models with different designs and patterns are then aligned up in the scene.

Until this stage, occlusion of the orthosis models has not been taken care of. The patient would realize that the entire augmented orthosis model is floating on the leg instead of wrapping it around. To fix this, a transparent material was created and attached to a 3D human leg model, which then put inside all 12 models. This step would hide the part of the orthosis model which is not supposed to be seen at certain angles when these parts are covered by the foot or part of the leg (see Figure 1).

Figure 1: Occlusion adjustment.
Original PDF Figure 1:

After the first orthosis model is placed correctly and adjusted accordingly, when one marker is hidden from the camera and the other is in view of the camera, the orthosis would show up at the same position. For this purpose, the whole setting is then mirrored on the other marker, so that one marker allows the consumer to view the orthosis model from the inside of the lower leg, and the other marker lets them look from the outside when the previous marker is beyond the reach of the camera.

The UI windows are then designed, and each assigned an animation, so that it can slide into and outside the screen, when certain buttons are clicked. Afterwards, a C# script called “OrthosisSelect”was created, so that each time an orthosis is selected on the screen, the corresponding 3D model is chosen to be shown. A colour picker Unity add-in was also added in the project, and this, together with the script called “ColorSelect”, enables the colour change of the orthosis models.

Visualization of Additive Manufactured Orthosis via Augmented Reality 311

1CMB is designed to let the consumer move the marker as well as the tablet, and also interact with the augmented orthosis model on the screen. The most important change for this version is adding the rotation function. To try the orthosis on, the patient first places the marker on the outer side of his or her leg, showing how the orthosis looks like on the outside of their leg, then the orthosis can be rotated 180 degrees by swiping on the screen, so that the patient can move the QR code to the view it on the inner side of his or her leg. The Unity add-in called Lean Touch enables it, as well as scaling the orthosis and moving it around on the screen.

Validation The last test was taken in the form of an interview to validate the most suit- able version of this application. This interview was carried off with seven orthosis manufacturers in a medical device trading fair called medica at Düs- seldorf. The seven orthosis manufacturers were asked about their opinions on the 1CMB and 2CMB version of the orthosis try-on application. All of them showed strong interest in this application and the additive-manufactured orthosis models, and they were willing to apply this digitalized production line if the end-product is more reliable and more socially accepted in the future. The 2CMB version was preferred by five out of seven manufactur- ers, since it is more intuitive and requires less physical movement from the patient.

CONCLUSION

In the end, an AR orthosis try-on application was developed. The 2CMB version needs two markers to be printed out yet is more intuitive and requires less moving of the leg from the consumers. The 1CMB version requires only one marker, but it is less easy to understand, and moving the camera might be burdensome for orthosis consumers. One way to build an ML version of this application was also proposed, but the ML version was not good enough for a final product. Still, it gives some thoughts on future AR application creation.

Some producers from the survey already gave some good suggestions for improving this application. For example, some chatting functions could be integrated into the application, so that doctors could directly talk to con- sumers and give suggestions on which model to purchase. Also, an elastic band could be used as a cylindric target for Vuforia to track, so that it is easier for consumers to put the marker in the right position.

Some more improvements could be made to this MB version of the appli- cation. Because the building platform is Android, and the MB version of the application does not take much storage place, the same design principle could be used for a Hololens application, so that consumers can use an HMD for orthosis try-on in the future, thus they do not have to hold a tablet for visualization anymore.

The Kinect set could be applied in the future for the ML tracking version to be realized, for it provides additional depth data for the developers to get features of the leg more easily. As mentioned in previous chapters, there

already exist some shoe try-on applications, which used the Kinect camera for ML tracking of the feet.

Some of the consumers might find downloading a new application and printing out the markers too complicated, to solve this problem, a later ver- sion of the AR orthosis try-on application could be integrated onto a website, and the Markers could be made to send to the consumer’s smartphone, so that they could use their smartphones to have the augmented orthosis sticking on it, saving more time an effort.

If orthosis production and delivery could be personalized in this way, the same principle of digitalized production line could be applied to other addi- tive manufacturable products, such as necklaces, bracelets, or even costumes. The consumer can try the personalized version of the product within the application and order them for additive manufacturing without any waste of materials or time searching in physical stores.

In digitalizing the production of ankle-foot orthosis models, a more environment-friendly and personalized production line can hopefully be built. AR is still a growing technology. We are hoping that it could bring us a more convenient and simple way of purchasing.

REFERENCES

Amin, Dhiraj., Govilkar, Sharvari. (2015). Comparative Study of Augmented Reality Sdk’s. International Journal on Computational Science & Applications 5 (1), 11–26. https://doi.org/10.5121/ijcsa.2015.5102. An, Shan., Che, Guangfu., Guo, Jinghao., Zhu, Haogang., Ye, Junjie., Zhou, Fangru.,

Zhaoqi, Zhu., Wei, Dong., Liu, Aishan and Zhang, Wei (2021). ARShoe: Real- Time Augmented Reality Shoe Try-on System on Smartphones. Ashtari, Narges., Bunt, Andrea., McGrenere, Joanna., Nebeling, Michael., Chilana, Parmit K. (2020). Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities. In: Proceedings of the 2020 CHI Confer- ence on Human Factors in Computing Systems. New York, NY, USA, Association for Computing Machinery, 1–13. Carmigniani, Julie., Furht, Borko., Anisetti, Marco., Ceravolo, Paolo., Damiani,

Ernesto., Ivkovic, Misa (2011). Augmented reality technologies, systems and applications. Multimedia Tools and Applications 51 (1), 341–377. https://doi.or g/10.1007/s11042-010-0660-6. Dima Titov (2020). 11 Best Augmented Reality SDKs to Start AR Development in 2021. Available online at https://invisible.toys/best-augmented-reality-sdk/# (accessed 1/23/2023). Günes, San., Sanli, Okan and Ergün, Övgü Öztürk (2015). Augmented Reality

Tool for Markerless Virtual Try-on around Human Arm. In: 2015 IEEE Interna- tional Symposium on Mixed and Augmented Reality - Media, Art, Social Science, Humanities and Design, 59–60. Kjærside, Krista., Kortbek, Karen Johanne., Hedegaard, Henrik., Grønbæk, Kaj (2005). ARDressCode: augmented dressing room with tag-based motion tracking and real-time clothes simulation. Proceedings of the Central European Multimedia and Virtual Reality Conference. Pan, Ren (2022). Design einer additiv gefertigten patientenindividuellen Unter-

schenkelorthese zur Behandlung von Cerebralparese. Masterarbeit. Aachen, RWTH Aachen.

Visualization of Additive Manufactured Orthosis via Augmented Reality 313

Presle, Philipp (2012). A Virtual Dressing Room based on Depth Data. Sanket, Prabhu (2017). Augmented Reality SDK Comparison: Parameters to select the right AR SDK. Available online at https://www.arreverie.com/blogs/augmente d-reality-sdk-comparison-parameters/ (accessed 01.2023). Smink, Anne R., van Reijmersdal, Eva A., van Noort, Guda and Neijens, Peter

C. (2020). Shopping in augmented reality: The effects of spatial presence, per- sonalization and intrusiveness on app and brand responses. Journal of Business Research 118, 474–485. https://doi.org/10.1016/j.jbusres.2020.07.018. Tetiana A. Vakaliuk and Svitlana I. Pochtoviuk (2021). Analysis of tools for the development of augmented reality technologies. In: International Workshop on Augmented Reality in Education. Yang, Yu-I., Yang, Chih-Kai., Chu, Chih-Hsing (2014). A virtual try-on system in

augmented reality using RGB-D cameras for footwear personalization. Journal of Manufacturing Systems 33 (4), 690–698. https://doi.org/10.1016/j.jmsy.2014.05. 006.

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET 2023): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET 2023): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-87-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →