Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

CoSculpt: An AI-Embedded Human-Robot Collaboration System for Sculptural Creation

1 School of Design, The Hong Kong Polytechnic University, Hong Kong SAR, China ↗
2 Laboratory for Artificial Intelligence in Design, Hong Kong SAR, China
Publication year
Digital Object Identifier10.54941/ahfe1004115

Abstract

Artificial intelligence (AI) and machine learning (ML) have witnessed significant advancements, leading to the emergence of AI-aided systems that assist in artwork creation. However, the current AI-powered creation support tools predominantly focus on generating 2D images, with limited assistance for 3D object creation, particularly in the field of sculpture. Additionally, as the automation capabilities of machines continue to grow, artists often struggle to maintain control over the creative process. Therefore, the collaboration between artists and AI in sculpturing remains unclear, along with the allocation of responsibilities to ensure the artist's controllability over the sculpture creation process.This study aims to address these gaps by developing CoSculpt, an AI-powered human-robot collaboration sculpturing system with three distinct modes. We explore how humans and AI-embedded robots can interact, delegate tasks, and assume control during various stages of the sculpture creation workflow. By utilizing this system, artists can effectively collaborate with AI technology, leveraging its capabilities while maintaining a significant degree of control over the sculpture creation process.The findings from this study shed light on the potential of human-AI collaboration in sculpturing and provide insights into how artists can effectively navigate the intersection of art and technology. The proposed AI-powered human-robot collaboration system offers a new paradigm for sculptural creation, fostering a harmonious synergy between human creativity and the technical capabilities of AI. Ultimately, this research contributes to the ongoing dialogue on the role of AI in the artistic domain and paves the way for future advancements in the field of sculpture.

INTRODUCTION

Artistic creation has long been regarded as a quintessential human endeavor. However, with the rapid advancements in artificial intelligence (AI) and machine learning (ML), the boundaries between human and machine creativ- ity are becoming increasingly blurred (Guljajeva and Sola, 2023). Machines, armed with algorithms, are now capable of generating artworks with min- imal human involvement or little need for human input (Gangadharbatla, 2022). This AI-aided approach to artwork creation has sparked a new era

© 2023. Published by AHFE Open Access. All rights reserved. 73

of exploration, where artists are seeking to collaborate with AI systems to enhance their creative processes.

While AI has made significant progress in generating 2D images, there remains a dearth of AI-powered tools that assist in 3D object creation, particularly in the realm of sculpture (Guljajeva and Sola, 2023). The absence of such tools restricts artists from fully leveraging the potential of AI technology in their sculptural works. Furthermore, as AI and machine autonomy continue to grow, artists face the challenge of maintaining control and agency over the creative process (Hwang, 2022). Ensuring that artists can actively participate and maintain controllability over the collaborative creation process becomes imperative in this context.

The objective of this study is to bridge these gaps by proposing a human- AI collaboration system specifically designed for sculpting. Our research aims to explore how artists and AI-embedded robots can effectively collabo- rate and interact, while allocating responsibilities and sharing control during various stages of the creative workflow. By integrating AI technology into the sculptural creation process, we strive to empower artists with enhanced capabilities and controllability, ultimately enabling them to unleash their cre- ative potential. In order to attain a comprehensive insight into the sculptural creative process, we conduct interviews with professional sculptors. Our findings reveal that sculptural creation involves a complex workflow and requires a considerable workload, making it a time-consuming process. We have focused our study on the crucial phase of creating a sculpture proto- type, which entails setting design objectives, conceptualization, design and planning, sculpting, and refinement.

To achieve this, we present CoSculpt, an comprehensive AI-powered human-robot collaboration system with three distinct modes to support sculpture creation. These modes encompass different stages of the sculptural process, allowing for seamless interaction and information exchange between the artist and the AI-embedded robot. The system not only facilitates the realization of the artist’s vision but also enables AI to contribute its com- putational prowess and assist in overcoming the limitations of traditional sculptural techniques.

We also evaluate the availability of CoSculpt by testing it with sculptors. The user studies suggest that CoSculpt could enhance the controllability of sculptors and optimize the sculptural creation process.

This position paper presents the following contributions: (1) CoSculpt, a novel human-AI-robot collaboration system in sculpture creation that could enhance artists’ controllability over the process, (2) a detailed technical approach to embed the AI generative model (DALL-E) and computer vision in robotic arm systems, (3) highlighting future possibilities for enhancing support to sculptors in order to better cater to their needs in the field.

RELATED WORK

Most existing research on robotic arm-assisted sculpture focuses on model scanning and analysis, as well as the planning and precision of tool paths. Baglioni et al. developed a multi-body modeling and dynamic analysis system

CoSculpt: An AI-Embedded Human-Robot Collaboration System 75

for milling manufacturing, as well as a position error evaluation system (Baglioni et al., 2016).

In actual applications, there are already many companies and studios that use robotic arms in the sculpture production process. Robotor Filippo Tin- colini and Giacomo Massari (Amoruso, 2019) developed a system that can fully automatically plan the entire sculpture process. The system uses differ- ent tool types at different carving stages, from carving materials to polishing, and can even automatically spray water during precision carving. However, these applications are all replicas of existing sculptures scanned and copied, rather than the process of creating a sculpture. Moreover, almost no human participation is required throughout the entire sculpture process, and human creativity and ideas are not fully expressed.

Some researchers have explored more artistic directions. Ma et al. (Ma et al., 2021) designed an interactive design system that generates robot arm tool path planning algorithms based on sculpture style, hoping to incorporate more artistic expression and style. Zoran and Paradiso (Zoran and Paradiso, 2013) developed a handheld digital milling device that combines computer assistance and monitoring with the sculptor’s production process and expe- rience. Nikola Damjanov generates complex 3D floral sculpture forms using AI, removes noise, and then models and 3D prints based on the form (Dam- janov, N., 2022). Although they increase the degree of human participation to varying degrees in the sculpture creation process, they still work on a pre- determined model. This can lead to the loss of sculptors’ temporary ideas or the expandability and inclusiveness of manual carving processes (Rosner and Taylor, 2011).

Currently, there is still no relevant research on the role relationship between mechanized equipment, computers, and sculptors in the sculpture creation process. The automation brought about by digital sculpting gradu- ally excludes sculptors from the workflow, which is not conducive to their control over the creative process and results.

UNDERSTANDING SCULPTURE CREATION PROCESS We conducted interviews with three professional sculpture artists to gain insights into their experiences with the sculpting process and the pain points they encounter during their work. During the interviews, the artists expressed the challenges they encountered during the sculpture prototype phase. This phase serves as a foundation for the subsequent stages of sculpture creation and significantly impacts the final outcome.

The creative process of sculpture prototype commences with the establish- ment of design objectives, whereby artists articulate clear goals and intentions for their artistic endeavors. Subsequently, artists embark on the conceptu- alization phase, a stage characterized by the pursuit of inspiration and the development of a concept that embodies their unique artistic vision. Design and planning ensue, with artists meticulously strategizing the composition, proportions, and overall structural framework of the sculpture. This phase involves the artist engaging in thoughtful sketching and visualization tech- niques to bring their ideas to life. The sculpting phase then ensues, involving

the tangible realization of the artwork through the skillful application of tech- niques such as carving, modeling, casting, and welding. Lastly, refinement assumes a pivotal role in the creative journey, as artists meticulously fine- tune the sculpture to ensure a harmonious alignment of every detail with their original vision.

One of the prominent challenges mentioned by the artists was the physical demands of sculpting. They highlighted the time and energy required to shape and form the sculpture, emphasizing the physically intensive nature of the process. The artists expressed the need for stamina and endurance to sustain long hours of work and to achieve the desired results.

Another key aspect discussed in the interviews was the importance of hav- ing a clear idea in mind before embarking on the sculpting process. The artists emphasized the significance of understanding the materials, techniques, and tools involved in sculpting to effectively translate their vision into a tangible form. They expressed the need for precision and attention to detail in order to achieve the desired outcome.

These insights inform our understanding of the sculptural creative process and lay the groundwork for further research and development of tools and technologies to better support sculptors in their artistic endeavors.

COSCULPT SYSTEM DESIGN This study offers three distinct modes of human-AI co-creation interaction to the users to freely choose between during the sculpting process. Upon initiating the system, users are presented with the option to select a mode to commence their sculpting endeavor. The freedom to choose any mode according to their specific needs is conferred upon the users.

In the Execution mode, users are afforded the opportunity to convey their artistic vision by etching lines onto the foam, which can be verified on the interface. Subsequently, upon selecting the “execute” option, the robotic arm will proceed to materialize the sculpture prototype by precisely cutting the foam following the user-drawn lines. This mode can be engaged by users who possess a distinct concept and meticulously devised plans concerning their approach towards achieving the desired results.

In the Collaboration mode, users are encouraged to sketch a line on the foam and verify it on the interface. Should the user find any aspect of the drawn line dissatisfied, a functionality is provided to erase the correspond- ing part of the line. The user then enters keywords to signify their intended modifications. The AI algorithm processes the drawn line and the keywords to create a line that aligns with the user’s requirements. Once a satisfactory result is attained, users can proceed to cut the foam by selecting the “execute” option. This mode is especially well-suited for users who have a general idea of how they want to sculpt the foam but are seeking inspiration or suggestions from the AI to refine or enhance their concept.

In the Automation mode, users have the option to enter keywords that reflect their desired outcome, allowing the system to create corresponding prototypes. Users can then choose a satisfactory design to proceed with, and by clicking confirm, the robotic arm will autonomously execute the foam

CoSculpt: An AI-Embedded Human-Robot Collaboration System 77

cutting process. This mode is particularly beneficial for users who may have uncertainties or seek AI-generated designs that are ready to use.

Throughout the sculpture prototype creative process, users have the free- dom to select and seamlessly transition between the three modes as they see fit, offering them flexibility and control over their sculpting experience.

Figure 1: Snippets of code to build the system vision with visual feedback: A) Image process, B) Skeletonization algorithm, C) K-Nearest Neighbors algorithms (KNN).
Original PDF Figure 1:

METHODS

OpenCV As the computer vision of the system, it captures the sketchy line on the foam in the format of an image as initial data input to feed the algorithm. How- ever, the presence of a bubble-like texture on the foam’s surface, coupled with the ever-changing lighting conditions resulting from varying sunlight and environmental factors throughout the day, poses a challenge to main- taining consistent image quality. This necessitates the use of supplementary lighting devices (see Figure 2) and algorithmic compensations. Therefore, we use the OpenCV library to process the picture and an optimal planning path for the subsequent robotic arm application.

The snippet A mainly uses Gaussian Blur to enhance the smoothness of the sketchy line (See Figure 1), and brightness control is executed with cv2.addWeighted(). Following step is to increase the contrast of an RGB image and turn the image to a binary picture. In the snippet B, we apply a thinning function cv2.ximgproc.thinning(), that is used to perform skele- tonization on a binary image using the Zhang-Suen algorithm. The resulting skeleton image is then displayed, in which the shape of the drawn line is turned into a series of coordinates in two dimensions of an image area. In order to arrange all random coordinates to an ordered sequence, we use K- Nearest Neighbors algorithms (Figure 1. C) to analyze the inclination of the line and identify the end point as an initial value. System finds the nearest point to the start value and assigns it as the next value in the coordinate dataset. By iteratively calculating, non-linear coordinates are distributed to linear status. Incorporated with the visual functions and effective algorithms,

a sketchy line is radically altered to a sequence of ordered coordinates as the planning path data for the robotic arm to do the cutting task on the foam.

Figure 2: GUI for generative shape in different modes. i) collaborative creation, ii) full automated mode.
Original PDF Figure 2:

DALL·E As a prominent tool integrated into the system, DALL·E offers the capability to generate different shapes derived from the original drawn line as the new input. we invoked its API and developed a dedicated function to encapsu- late its operations. Later, we design a canvas for the user to engage with and inpaint with, thereby creating an image mask area in 0% opacity for the sys- tem to further generate novel forms. In Figure 2. i), which corresponds to the collaborative creation mode, an eraser with a round brush is provided. Based on the input text prompt, a new line is created, facilitating an iterative and collaborative creative process. (See Figure 2. ii)) In the full automated cre- ation mode, the system creates multiple choices to stimulate user inspiration. A preferred one would be selected and further built by the robotic arm.

SETUP DETAILS We established a dedicated experimental workspace to conduct our exper- iments, as depicted in Figure 3. In order to capture the drawn line on the foam accurately, we employed a Logitech Camera as our visual perception. Given the camera’s sensitivity to light and potential defects, maintaining a consistent environmental illumination level was crucial. To address this, we supplemented the lighting using an annular light bar. To position the foam material, we designed and fabricated a laser-cut base. At the end of the robotic arm, we integrated a heating device capable of material subtraction, allowing

CoSculpt: An AI-Embedded Human-Robot Collaboration System 79

for the creation of intricate shapes and structures in the artwork. The robotic arm employed in our system is the xArm7 UFACTORY.

Figure 3: System setup with camera, robotic arm and foam base.
Original PDF Figure 3:

EVALUATION To assess the effectiveness of our CoSculpt system in providing creativity sup- port to sculptors, we conducted a comparative user study between using our system and using a traditional system.

PROCEDURES We selected a sample of 10 experienced sculptors via online platforms. Par- ticipants were tasked with creating the sculpture prototype using two distinct methods. One was to use traditional tools and the other is to use CoSculpt. Participants were provided with the necessary materials, resources, and instructions to execute their artistic vision using both methods.

OBSERVATIONS It was observed that when using the first method, some participants heavily relied on their own skills and expertise, without utilizing AI creativity sup- porting tools. This approach resulted in significantly longer creation times, and participants expressed feelings of fatigue and exhaustion. Although a few participants did make use of certain AI creativity tools, the overall impact and effectiveness were limited. They frequently had to switch between different tools, which disrupted their workflow and caused fatigue. Additionally, they

often struggled to find suitable resources, resulting in moments of feeling stuck and uncertain about their creative direction. The available tools either lacked the desired features or did not seamlessly integrate into their work- flow. As a result, participants felt that their reliance on such tools did not significantly alleviate the difficulties they encountered.

By contrast, CoSculpt empowered participants to leverage AI tools and features at any point during the creative process, providing them with on-demand support and enhancing their efficiency. Whether it was gener- ating design ideas, refining concepts, or executing intricate sculpting tasks, CoSculpt offered a comprehensive suite of AI-driven functionalities to assist users.

The integration of a robotic arm allowed for precise and automated sculpt- ing, relieving users from physically demanding and time-consuming manual work. This not only reduced the workload for participants but also enabled them to focus more on the artistic aspects of their creations.

LIMITATIONS AND FUTURE WORK Limited Focus on Sculpture Prototyping. Our study primarily focused on the creation of sculpture prototypes for demonstration purposes. Further research is needed to develop iterative and interactive approaches that enable sculptors to refine and optimize their algorithms through multiple iterations. This would enhance the efficiency and effectiveness of the sculptural creative process.

Integration of Complex Manipulation. Our current work primarily addressed basic manipulation techniques in sculptural creation. Future inves- tigations should delve into more complex manipulation methods that closely align with the intricate workflow of sculpture creation. This would enable artists to achieve greater artistic expression and explore innovative tech- niques.

Quantitative Validation Methods. While our research explored various aspects of sculptural creation, the lack of quantitative validation methods limits our ability to objectively measure the effectiveness and quality of the generated sculptures. Future work should focus on developing quantitative metrics and validation frameworks to assess and compare the artistic merit and fidelity of sculptural outputs.

CONCLUSION

In conclusion, the CoSculpt system has significantly contributed to the field of sculpture creation by providing enhanced controllability and facilitating effective human-AI collaboration. Through its innovative tools and technolo- gies, CoSculpt empowers sculptors to express their creativity while preserving their artistic agency. By offering flexible modes such as Execution, Assistance, and Automation, CoSculpt combines the power of artificial intelligence and robotic arm technology to provide sculptors with greater control and adapt- ability throughout the entire sculpting process. This integration of AI and

CoSculpt: An AI-Embedded Human-Robot Collaboration System 81

robotics not only streamlines the workflow but also expands the possibil- ities for artistic exploration and realization. CoSculpt’s ability to facilitate collaboration between humans and AI systems opens new avenues for cre- ativity and paves the way for further advancements in the realm of sculpture creation.

ACKNOWLEDGMENT

This project was funded by the Intelligent Systems Design Programme, School of Design, the Hong Kong Polytechnic University. The work was also substan- tially supported by the Projects of Strategic Importance of The Hong Kong Polytechnic University (Project ID: P0036851). The authors would like to acknowledge the technical and ideation supports from Zhengtao Ma, and Cong Fang at the Intelligent Systems Design Programme, School of Design, the Hong Kong Polytechnic University.

REFERENCES

Amoruso, G. (2019) Digitalization and Cultural Heritage in Italy: Innovative and Cutting-edge Practices. Baglioni, S. et al. (2016) ‘Multibody modelling of N DOF robot arm assigned to

milling manufacturing. Dynamic analysis and position errors evaluation’, Journal of Mechanical Science and Technology, 30(1), pp. 405–420. Damjanov, N. (2022). Artstation - protist florist A01 - Serbia art biennale 2022.

https://www.artstation.com/artwork/zDGLzm. Gangadharbatla, H. (2022) ‘The role of AI attribution knowledge in the evaluation of artwork’, Empirical studies of the arts, 40(2), pp. 125–142. Guljajeva, V. and Canet Sola, M. (2023) ‘AI-Aided Ceramic Sculptures: Bridging

Deep Learning with Materiality’, in Artificial Intelligence in Music, Sound, Art and Design. Springer Nature Switzerland, pp. 357–371. Hwang, A. H.-C. (2022) ‘Too Late to be Creative? AI-Empowered Tools in Creative Processes’, in Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems. New York, NY, USA: Association for Computing Machinery (CHI EA ‘22, 38), pp. 1–9. Ma, Z. et al. (2021) ‘Stylized robotic clay sculpting’, Computers & graphics, 98,

pp. 150–164. Rosner, D. K. and Taylor, A. S. (2011) ‘Antiquarian answers: book restoration as a resource for design’, in Proceedings of the SIGCHI Conference on Human Fac- tors in Computing Systems. New York, NY, USA: Association for Computing Machinery (CHI ‘11), pp. 2665–2668. Zoran, A. and Paradiso, J. A. (2013) ‘FreeD: a freehand digital sculpting tool’, in

Proceedings of the SIGCHI Conference on Human Factors in Computing Sys- tems. New York, NY, USA: Association for Computing Machinery (CHI ‘13), pp. 2613–2616.

Cover of the proceedings volume Human Systems Engineering and Design (IHSED 2023): Future Trends and Applications

Explore the complete digital volume

Human Systems Engineering and Design (IHSED 2023): Future Trends and Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-88-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →