Zhiyu Zhao, Siqi Sheng (2026). Multisensory Interactive Conceptual Design for Nanyin Art Based Cultural Healing via Human-AI Collaboration. In: Affective and Pleasurable Design. AHFE Open Access, vol. 204, pp. 143–153. https://doi.org/10.54941/ahfe1007340
This research proposes an innovative method to transmit Nanyin culture by combining human-AI collaboration with multisensory interactive technology. Applying a combination of literature reviews, a formative pilot survey, and case analyses, this study establishes a theoretical approach for integrating traditional culture with modern art-based healing. The pilot survey examined public emotional needs and analyzed successful cases to identify the key elements of multisensory interactive design. The specific conceptual design process proceeded as follows: it begins with data collection and the extraction of cultural symbols, combining human insight with AI to recognize traditional Nanyin elements. ChatGPT and other generative tools were subsequently used to generate prompts for storyboard development. Using extracted audio features and traditional visual symbols as core inputs, AI technologies including Generative Adversarial Networks (GANs) were applied to create the image and audio materials for the exhibition. Human designers further refined the AI-generated materials to maintain stylistic consistency and ensure smooth animation transitions. This study establishes a comprehensive conceptual design path and a corresponding virtual exhibition prototype, offering new models for sharing intangible cultural heritage and using it therapeutically today.
Research Aim and Background Rapid urbanization and globalization are making it harder to preserve traditional local cultures like Nanyin. Nanyin is a regional art known for its melodic tunes and profound emotional expression. To address these transmission gaps, this research aims to employ human-AI collaboration to design a conceptual multisensory interaction exhibition hall. By integrating visual, auditory, and tactile experiences, multisensory interaction effectively enhances users’ immersion and emotional resonance with cultural content. This study focuses on utilizing the unique audio-visual elements of Nanyin culture through a fully AI-assisted design process, spanning venue design, script storyboarding, and visual and voiceover production to foster art-based cultural healing. The findings thus help to explore new ways of local culture dissemination in modern society.
In recent years, the world has entered the AI era, characterized by the rapid emergence of large language models such as generative pre-trained transformers. The most advanced AI has enabled various applications, especially in generative design innovation (Liu, 2023). For example, tools like DALL-E 3, Dream Fusion, and ChatGPT excel at tasks ranging from complex image creation to 3D modeling and information summarization. These highly advanced technologies can assist people in completing most of the design and generation needed for an exhibition hall. The human-AI collaborative design approach focuses on integrating human creativity with the efficiency that AI brings to complex tasks, producing outcomes that go beyond what traditional design methods can achieve. Especially in the creative field, AI can help designers explore more diverse and innovative solutions quickly by generating various design prototypes or cultural elements (Zeng & Sun, 2022; Shi et al., 2023). Human-AI collaboration can inspire designers’ creativity by having AI suggest design prototypes for humans to select and refine. This approach has broad application prospects in art creation and cultural promotion.
In cultural communication specifically, this collaborative design effectively improves the practical efficiency of cultural creation. At the same time, it reflects the dynamic transformation of traditional culture in an intelligent environment, thus supporting the renewal and innovative development of cultural heritage in a modern context. For example, in the Human-AI Co-Dancing project at MIT Media Lab, the design method of collaborative creation between human dancers and virtual dance partners was explored, using the dance vocabulary of Thai traditional dance and algorithm-generated virtual characters to achieve artistic innovation through human-AI collaboration (Pataranutaporn et al., 2024). The intelligent dance creation based on cultural heritage not only showcases the application potential of AI in the arts and culture but also attracts many audience participants through multisensory interaction, further confirming the effectiveness of human-AI collaborative creation and multisensory experience in local cultural dissemination.
Emotion-driven interaction design, by capturing users’ emotions and adjusting the interactive content in real time, can enhance the appeal and immersion of cultural experiences. Don Norman pointed out in Emotional Design: Why We Love (or Hate) Everyday Things that by capturing users’ emotions and adjusting the content presentation accordingly, the resonance experience of users with cultural content can be effectively enhanced (Norman, 2004). Emotional design can stimulate users’ positive emotions, increasing the pleasure and depth of memory during the experience. In disseminating local culture, emotion recognition technology is particularly applicable. The system can customize and deliver targeted cultural content based on the user’s real-time emotional state, such as playing soothing music or adjusting lighting to create a warm, ambient atmosphere. This design approach effectively encourages emotional connection between users and the cultural content, thereby enhancing users’ understanding of and connection to local culture.
Multisensory Interactive Conceptual Design for Nanyin Art Based Cultural 145 The application of emotion recognition technology in cultural dissemination is gradually expanding. By using AI to identify changes in users’ emotions and dynamically adjusting the presentation form of cultural content based on feedback, local cultural dissemination can achieve high personalization. For instance, through biofeedback technology, physiological parameters such as heart rate and breathing are converted into emotional responses. Then through multisensory feedback such as light and sound, users are gradually immersed in specific emotional scenarios during the experience. This dynamic content presentation based on users’ emotions not only enhances the effect of cultural dissemination but also helps users achieve deep emotional satisfaction during the cultural experience. For example, the series of art experimental projects Emotion Theatre by artist Fei Jun and psychologist Liu Zhengkui, through artificial intelligence and emotion computing technology, detect and present the emotions of the audience, and generate personalised and therapeutic audio-visual theatre experiences for each viewer, creating a positive psychological effect from emotion recognition, transformation to acceptance for the experiencers.
In recent years, many multisensory technologies have been applied in cultural experience design, providing users with more immersive interactive experiences (Luo, 2023). In China, the project Pattern as a Carrier of the Way. The Palace Museum and Tencent Immersive Digital Experience Exhibition have been a well-known example. This project combines visual, auditory, and tactile multisensory technologies to create a comprehensive cultural involvement space for visitors. It uses high-definition projections, dynamic images, and interactive screens to vividly present the pattern art of traditional culture in a modern technological way. Visitors can closely examine all the intricate pattern details, and with the touch devices, they can also feel the texture and the history behind them. Moreover, with the spatial audio effects combined with the images, they feel as if they are immersed in the cultural atmosphere conveyed by the patterns. This all-round interactive experience not only enhances the visitors’ sense of participation and immersion but also rejuvenates traditional pattern art in the digital context.
Research Methodology This study integrated three research methods, namely literature research, questionnaire surveys, and case analyses, to establish a multi-dimensional design path. Through literature research, a comprehensive review of relevant theoretical findings was conducted. Based on this, a design path for human-AI collaboration was proposed. The study also used case analyses to identify the key traits of this design approach and to examine its practical applications, especially in cultural communication and emotional expression. Specifically, a pilot study was conducted using an online questionnaire to delve deep into the public’s genuine requirements regarding cultural experiences and emotional healing. The pilot study recruited a preliminary sample of 78 respondents to provide empirical evidence for the design orientation. By applying multisensory conceptual interaction design to Nanyin culture, the authors translated its artistic qualities and emotional roles into tangible
146 Zhao and Sheng spatial experiences, which effectively supported the research objectives from multiple angles.
Nanyin was chosen as the core research subject for its profound historical heritage, outstanding artistic expression, and distinctive emotional value. With its melodious vocal and instrumental lines and finely nuanced emotional expression, the genre encapsulates the core essence of Minnan ritual music and cultural sentiments, and holds an innate capacity to soothe the mind and regulate emotional states. The dongxiao’s timbre, the pipa’s shape, and the ceremonial side of Nanyin performance all lend themselves well to multisensory interaction tech, helping this art form find new life in the digital age. In the context of modern society, the fast-paced lifestyle has exacerbated psychological stress, thus emphasizing the urgency of emotional healing. Incorporating Nanyin into the design of healing-themed exhibition halls not only safeguards and transmits this intangible cultural heritage but also facilitates a two-way interaction between culture and emotion in the user experience. This allows visitors to not only appreciate the aesthetic attraction of traditional art but also attain profound emotional equilibrium and psychological solace. This approach not only addresses the social need for spiritual well-being but also paves a new way for the creative transformation of cultural heritage in the era of human-AI collaboration.
FEASIBILITY ANALYSIS OF MULTISENSORY INTERACTIVE ART THERAPY DESIGN OF NANYIN CULTURE FROM THE PERSPECTIVE OF HUMAN-AI COLLABORATION Cultural Symbols of Nanyin
Originating in Quanzhou, Nanyin has a history of over a thousand years, blending elements of the Central Plains’ refined music and folk music from southern Fujian. It serves as a comprehensive carrier of traditional ritual and music culture, relying on distinctive visual and auditory symbols (Wang, 2008). These include Gong Yi ancient music scores, which possess deep visual aesthetic value, and unique instruments such as the curved-neck Nanpa (horizontal lute), the Dongxiao (bamboo flute) engraved with auspicious patterns, and the Sanxian (Zeng & Sun, 2022). Furthermore, performers wear traditional Minnan costumes, such as the Phoenix Crown and Red Robe, which symbolize historical inheritance and provide rich visual assets for dynamic digital transformation.
The Healing Potential of Multisensory Interaction Design of Nanyin Culture From the Perspective of Human-AI Collaboration Nanyin centers on auditory perception, which is complemented by visual elements such as costumes, props, and stage settings, as well as the layered sounds of its instruments. Multisensory interaction technology can transform it into dynamic expressions. For example, designers can digitally present the movements of the pipa playing. Designers can also combine the sound of the Dongxiao with lighting to render the atmosphere of ritual music. Dynamically visualizing the Gong Yi notation symbols integrates hearing and seeing,
Multisensory Interactive Conceptual Design for Nanyin Art Based Cultural 147 thereby enhancing immersion. The slow, gentle melodies of Nanyin adeptly depict profound emotions and have a soothing effect on the soul.
To empirically validate this healing potential, the aforementioned pilot study (78 respondents) identified a widespread psychological need for cultural healing. Currently, 50% of the respondents reported experiencing emotional stress on a weekly basis, and 21.8% were troubled by it daily, with 80.8% explicitly stating they lack adequate sources of emotional support. Integrating Nanyin into an interactive venue presents clear market potential, as 34.6% of respondents were very interested and 41.0% were interested in such immersive emotional experiences. Crucially, 59.0% of respondents believe that existing emotional therapy devices lack personalized settings, and 56.4% think that the therapeutic effect is not obvious. Human-AI collaborative design addresses this homogenization problem: AI dynamically adjusts the presentation rhythm and light and shadow tones based on Nanyin symbols, while human designers avoid the fragmented defects of AI-generated content. This dual mechanism, which aligns technology with user needs, anchors emotional experience in culture, lies at the core of improving therapeutic outcomes.
AIGC technology is now widely used in cultural design, reshaping how local cultures are shared. In the multisensory interaction design of Nanyin culture, AIGC technology can enable an intelligent upgrade of the entire process, from extracting cultural symbols to content creation, injecting more creativity and vitality into the design. Using deep learning algorithms on cultural heritage data, AIGC can identify Nanyin’s symbol system, including Gongche notation and instrument features. This allows designers to extract and adapt traditional elements more efficiently, offering intelligent support for the revitalization of cultural heritage (Lu et al., 2023). For example, synchronized light and shadow effects with Nanyin melodies and dynamic cultural elements in virtual environments allow audiences to immerse themselves in the artistic and cultural experience through multisensory engagement. In creating Nanyin culture’s graphic design, storyboard development, animation production, and the design of multisensory exhibition halls, each phase involves steps to ensure seamless collaboration between AI and human designers: data collection and cultural symbol extraction, AI image generation, and designer iteration and optimization, ultimately producing high-quality results (Zhao, 2025) (See Figure 1).
Figure 1: Design path of human-AI collaboration.
Figure available in the original PDF paper.
Data Collection and Cultural Symbol Extraction By combining manual efforts with AI technologies, the authors thoroughly explored and extracted traditional elements of Nanyin culture, including musical scores, instruments, and performance techniques. They compiled a
148 Zhao and Sheng comprehensive collection that includes classic Nanyin repertoire, performance segments, historical performance videos, and materials related to Nanyin musical instruments, ancient scores, costumes, and props. This aimed to gain a complete understanding of the emotional and cultural meanings embedded within. Additionally, the authors extensively reviewed and analyzed a large amount of literature and research reports. This provided a thorough understanding of the development path and unique features of Nanyin culture, helping to identify its distinct cultural elements. The collected data was then carefully classified and organized before being imported into AI tools. These tools used neural networks for data learning and processing.
AI Image Generation Using the extracted audio features (melody, rhythm, emotional qualities), traditional visual symbols, and the keywords we generated, the authors brought in advanced AI tools like GANs to handle the design and content creation. That gave the authors the images and audio needed for the exhibition.
Designer Iteration and Optimization Despite AIGC’s progress, the technology is still developing and has clear limitations. The content generated by these tools often exhibits fragmentation and lacks cohesive artistic expression. At this stage, the subjective integration ability of human designers becomes particularly critical. Designers can make up for the inadequacies of AI-generated content through subjective integration and creative polishing. They optimize video details that deviate from natural principles. Relying on their aesthetic sensibilities, human designers ensure the stylistic consistency of the entire exhibition design, guarantee the smooth flow of animations, minimize elements that disrupt the viewing experience, and play a pivotal role in integrating various components. This serves to compensate for the logical and design shortcomings inherent in the generated content. Only by continuously iterating and optimizing through human-AI collaboration can the final work truly achieve the seamless integration of technology and creativity, culminating in high-quality design work.
MULTISENSORY INTERACTION DESIGN APPROACHES FOR NANYIN CULTURE BASED ON HUMAN-AI COLLABORATION Data Collection of Nanyin Culture and Cultural Symbol Extraction
This study employed Ji Meng AI, a homegrown Chinese generative AI tool, for multimedia content development. The platform is built with targeted optimizations for Chinese cultural elements, enabling the precise generation of multimedia content including images and videos. Its integrated storyboard function additionally supports shot synthesis, streamlining video creation workflows and improving overall production efficiency. Critically, as a natively developed Chinese AI generation platform, its specialized tuning for Chinese cultural elements ensures consistently high precision when creating content tied to these cultural features. Initially, the authors imported the pre-collected
Multisensory Interactive Conceptual Design for Nanyin Art Based Cultural 149 traditional Nanyin elements, such as musical scores and instrument forms, into Ji Meng AI. Subsequently, the authors specify clear requirements for composition, style, color, and other aspects to refine the generated images. Taking the pipa symbol as an example, the authors first imported a pipa image as a reference and then entered relevant prompt words. If the aim was to obtain an abstract planar pipa graphic element with Minnan regional characteristics and traditional Chinese color schemes, the following prompt words were entered: Symmetrical composition, pipa, abstract, traditional color palette, warm colors, elegance, Minnan culture, planar composition, symbolization, tenderness, Chinese elements, ink-wash style, and high saturation. This was used to generate the images. Subsequently, the authors made local modifications to the generated images or added keywords for multiple iterative generations. Finally, with reference to traditional elements, the author adjusted the design draft details. Eventually, the design drawing of the pipa-element symbol was created (See Figure 2).
Figure 2: Design of Pipa symbol by Ji Meng AI.
Storyboard Script and Animation Generation Based on Human-AI Collaboration After extracting cultural symbols, the authors used Ji Meng AI’s storyboard script feature to transform static symbols into dynamic videos. Its multi- scene continuous generation function can be combined with prompt words and parameter adjustments to efficiently achieve the emotional dynamic presentation of Nanyin culture. This function is particularly suitable for the design of Nanyin cultural content that requires emotional expression and dynamic representation. The following outlines the specific process for creating storyboard scripts and animations.
Creating the storyboard script is the first step in animation production. It defines the animation’s rhythm and content through scene-by-scene descriptions, allowing Nanyin cultural symbols to convey rich emotions and cultural imagery visually. The authors generated a storyboard script through ChatGPT as a reference. The authors precisely described the script requirements, including timing, shot sizes, and scene details, and emphasized that it would be used for AI image generation. Based on the generated script, the authors made subjective modifications and added detailed descriptions of the image scenes to facilitate storyboard generation.
150 Zhao and Sheng The second step is scene creation. In Ji Meng AI, the authors described the content and visual effects of each generated shot through its storyboard creation function. After that, the authors ensured the smooth connection of the entire storyline and generated each storyboard image. After completing the organization, the authors used the tool’s image-to-video function to combine the images and describe the generated scenes and actions to complete each fragment video. Take one of the storyboard scripts as an example: Shot size: Wide shot. Scene: Night sky projected on the dome, with stars twinkling, and the dawn light slowly rising, accompanied by the clear sound of the Nanyin flute. Effect: The light gradually brightens, and the silhouettes of birds fly across the scene, symbolizing the beginning of the prologue. The author added a description of the scene: Traditional Chinese warm color tones, abstract graphic style, Chinese elements. These prompts were input into the tool together, and the first storyboard image was finally generated. By analogy, the AI tool assisted the author in creating more storyboards (See Figure 3).
Figure 3: Storyboard images by Ji Meng AI.
After generating the initial version of the animation from the script, the authors evaluated whether its movement rhythms matched the emotional expression of Nanyin culture. The authors then checked whether the movement of the symbols matched the rhythm of the Nanyin melody, ensuring that the dynamic display of the ink-wash-style background was coordinated with the emergence of the Nanyin symbols. Only after that did they add music to the complete video and proceed to optimize the dynamic content based on the actual effect. Subsequently, the authors iteratively optimized the unsatisfactory parts by adjusting the prompts and parameters. For the unnatural parts of the dynamic trajectory, the authors reset the path. For instance, the authors input ‘smoother movement trajectory, gradual deconstruction of symbol elements’ to enhance the scene transition effect. Another example is ‘background gradient, soft light, and shadow transition’,
Multisensory Interactive Conceptual Design for Nanyin Art Based Cultural 151 which was input to intensify the emotional atmosphere. Finally, the authors made detailed adjustments. The authors needed to manually correct the generated animation, such as optimizing the detailed special effects using post-production tools like After Effects. After a series of processes, a complete video component of an immersive exhibition hall has been obtained.
Design of a Multisensory Interactive Exhibition Hall for Nanyin Culture Based on Human-AI Collaboration Following the same steps as above, the authors generated numerous exhibition hall renderings using AI and selected those that met the requirements for 3D modelling. The authors imported the images into Unreal Engine 5 (UE5) to demonstrate the interactive exhibition hall effect. UE5 can significantly enhance visual effects and interactive experiences while optimizing overall performance, enabling a contemporary presentation of traditional culture. UE5’s Lumen global illumination system can simulate real-time light reflections and dynamic changes, such as light penetrating bamboo shadows in a Minnan courtyard and subtle reflections on bird feathers in the morning light. These details endow the exhibition hall with a high degree of immersion. Figure 4 shows the rendering effect of the AI-generated video displayed in the UE venue.
Figure 4: Conceptual exhibition hall for Nanyin culture.
The spatial design of the conceptual exhibition hall centers on enveloping immersion, using 360-degree surround screens to create a closed narrative field. From the floor to the walls and then to the dome, the screens seamlessly connect like flowing water to form an unbounded visual carrier. The overall environment is maintained in a low-illumination state through precise light control: only the light sources of the screen content are retained, while the rest of the area fades into a soft dimness. This design is like building a barrier for the senses, not only blocking out the hustle and bustle of the outside world and the flickering of electronic devices, but also naturally focusing the viewer’s visual attention on the flowing Nanyin symbols on the screen. The
152 Zhao and Sheng curves of the Gong Yi score gradually unfold in the darkness, the outline of the Nanyin Pipa brightens and dims with the melody, and the silhouette of the Dongxiao trembles with the airflow, as if the light and shadow of the ancient opera garden a thousand years ago have been brought into the present space. This distraction-free, immersive design ultimately points to the unique emotional core of Nanyin: when the senses are precisely guided and attention forms a deep dialogue with traditional cultural symbols, the viewer will naturally resonate emotionally with the thousand-year-old cultural heritage. Anxiety gradually subsides in the slow melody, and restlessness slowly settles in the gentle light and shadow. Ultimately, in the encounter of culture and soul, one can touch the unique, gentle healing power of Nanyin, like the sea breeze of southern Fujian.
CONCLUSION
This research explores a new development path for Nanyin culture through human-AI collaboration and conceptual multisensory interaction. By integrating technological means with traditional cultural values, the study effectively enhances the appeal of Nanyin in modern society and injects new vitality into its inheritance and dissemination. It is important to note that the primary contribution of this research lies in establishing a conceptual design path for human-AI collaborative cultural healing. Within this scope, the questionnaire was deliberately designed as a formative pilot study. Although the study included only 78 respondents, the sample was sufficiently focused to meet its exploratory objectives. It effectively captured the public’s initial emotional needs and offered reliable empirical support for the later conceptual design process.
Furthermore, although the current UE5-based virtual exhibition effectively demonstrates the technical feasibility and aesthetic value of this approach, it remains a conceptual prototype. Future research will move from this conceptual stage to empirical evaluation, implementing a physical multisensory exhibition and conducting systematic human-computer interaction tests with larger participant samples. This will allow collection of quantifiable physiological and psychological data to assess the actual therapeutic effects of the Nanyin cultural experience.
REFERENCES
Liu, L. (2023). Challenges and Opportunities of Packaging Design Industry Driven by AIGC Intelligence, Packaging Engineering, 44(Sup.2):236–240. Available at: https:// kns.cnki.net/kcms2/article/abstract?v=y_SiIdm5mqsVjfGWfqHT03TtQlJAN RMIf4JDzt4hC2vJpXVrHZTbXtRCzowRo44eGiDv02mOBIax-LYMU2A- JDno3g9ojqVE0dv-eNiiZ5Um4O96LWunxv1phbzBVLW5zhZf4A8kXr opaG8zvlBlWLPjEFT_wK_UakTfUNJNRAKV0bURYDcu5Q==&uniplat form=NZKPT&language=CHS Lu, Z. et al. (2023). State of Arts and Development of Intelligent Design Methods
under the AIGC Trend, Packaging Engineering, 44(24):18-33+13. Available at: https://link.cnki.net/doi/10.19554/j.cnki.1001-3563.2023.24.003
Multisensory Interactive Conceptual Design for Nanyin Art Based Cultural 153 Luo, C. (2023). Research on the design of interactive intelligent toy art experience
based on multimodal sensory, Applied Mathematics and Nonlinear Sciences, 8(2):351–360. Available at: https://doi.org/10.2478/amns.2023.2.00351 Norman, D. (2004). Emotional Design: Why We Love (or Hate) Everyday Things.
New York: Basic Books. Pataranutaporn, P. et al. (2024). Human-AI co-dancing: Evolving cultural heritage through collaborative choreography with generative virtual characters, In Proceedings of the 9th International Conference on Movement and Computing, Article 14, 1–10. Available at: https://doi.org/10.1145/3658852.3661317 Shi, Y. et al. (2023). Understanding Design Collaboration Between Designers and
Artificial Intelligence: A Systematic Literature Review, In Proceedings of Proc. ACM Hum.-Comput. Interact. 7, CSCW2, Article 368, 1–35. Available at: https://doi.org/10.1145/3610217 Wang, Y. (2008). The Protection and Inheritance of Nanyin in China, Fu Jiang Arts, 01: 5-11. Available at: https://kns.cnki.net/kcms2/article/ abstract?v=y_SiIdm5mqv7Z580vjR_UUyVS_T c-UWKjwqfMUI1_qwmb OezHW16HJZZXSFVvLuzdHCAjdukOcwvQ3lXOKoMk6mWt21GWvz IASd94TksrpGm2QkRjYBJI2APDm1FgAw57BOPtahPELHL1akDjG Z9y2fFKofnQrFMlhuYxFEjbDfDBSopa6NlkQ==&uniplatform= NZKPT&language =CHS Zeng, Z. & Sun, X. (2022). Human-machine Collaborative Design Exploration
Based on the Concept of Augmented Intelligence, Packaging Engineering, 43(20):154–161. Available at: https://link.cnki.net/doi/10.19554/j.cnki.1001- 3563.2022.20.016 Zhao, Z. (2025). Multimedia Interaction Design Model of Local Culture in Human-AI Collaborative Environment, In Proceedings of International Conference on Human-Computer Interaction, 229–245. Available at: https://link.springer.com/ chapter/10.1007/978-3-031-93160-4_15
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Affective and Pleasurable Design
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-80-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional