Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

AI-Driven Personalized Multisensory Design of Cultural Heritage: A Case Study of Kunqu Opera

East China University of Science and Technology, Meilong Road 130, Shanghai, China ↗
Publication year
Digital Object Identifier10.54941/ahfe1006060

Abstract

With the evolution of global cultural consumption habits, the influence of traditional cultural heritage in contemporary society has gradually diminished. While Kunqu opera is highly regarded for its profound cultural value, its international influence remains limited due to changes in modern cultural consumption habits, language barriers, and the constraints of traditional dissemination methods. To enhance the appeal and impact of traditional culture in modern society, this study takes Kunqu opera, a Chinese intangible cultural heritage, as a case study and explores AI-driven personalized multisensory design strategies. By integrating artificial intelligence technologies, the study aims to create multisensory interactive experiences that engage users’ visual and auditory senses, enabling them to perceive the charm of Kunqu opera, stimulate emotional resonance, and enhance the effectiveness of personalized cultural transmission.This paper proposes a multi-layered design framework that encompasses user data analysis, personalized customization, and multisensory content generation. The study employs AI to deeply analyze and generate the artistic elements and cultural connotations of Kunqu opera. In an innovative approach, it presents Kunqu stories in the form of an animated audio comic, using popular music to express the opera's melodious and delicate vocal style and elegant poetry, while employing a fresh comic style to depict the intricate designs of Kunqu costumes, such as flowing sleeves, fans, and detailed makeup. This transformation of Kunqu's classic performances into dynamic audio-visual experiences allows users to engage interactively and appreciate the opera’s unique charm. On the visual front, generative adversarial networks (GANs) are used to reconstruct Kunqu costumes, stage settings, and performance actions; on the auditory front, deep learning algorithms are applied to perform style transfer and synthesis of traditional singing and music.This research aims to provide new insights for the protection and dissemination of cultural heritage through the AI-driven recreation of Kunqu opera, ultimately enhancing its international recognition and acceptance.

INTRODUCTION

As one of the most important representatives of traditional Chinese opera, Kunqu has deep cultural connotations. It originated in the Ming Dynasty, from the Suzhou area, and initially evolved from local miscellaneous operas, and gradually developed into a comprehensive art form combining singing,

© 2025. Published by AHFE Open Access. All rights reserved. 53

54 Zeng and Zhou

reciting, acting and playing.In folklore, Kunqu has long been transmitted by word of mouth. Kunqu is not only a treasure of Chinese theatre art, but also part of the world cultural heritage. Against the background of globalisation, Kunqu, as an important carrier of Chinese culture “going out”, has been gradually increasing its dissemination and acceptance in the international arena, but it still faces obstacles such as language and cultural differences, which have led to its relatively limited global influence and recognition.

Using modern technology to innovate Kunqu’s dissemination is key to enhancing its international status. AI can reconstruct Kunqu’s artistic elements through deep learning and generative adversarial networks, aligning it with global aesthetics and cultural backgrounds. This study aims to blend traditional Kunqu singing with modern music through style transfer, while digitally recreating costumes and stage designs to enhance expressiveness. AI technology combines Kunqu’s traditional style with modern music, preserving its heritage while appealing to diverse audiences. Additionally, AI can accurately render and innovate Kunqu’s costumes, props, and sets using computer graphics, optimizing them for modern tastes.

PERSONALIZED CUSTOMIZATION AND USER PROFILE CONSTRUCTION BASED ON USER DATA ANALYSIS Through in-depth analysis of the questionnaire data, users of different age groups showed significant differences in their interest in and acceptance of the Kunqu opera The Peony Pavilion. This study first defines several key assessment dimensions: understanding of the play, cultural identity, emotional engagement, and acceptance of modernised adaptations. Each dimension was rated on a quantitative scale of 1 to 5, where 1 indicates very low and 5 indicates very high. The age groups were categorised as: 10–18 years old, 19–30 years old, 31–45 years old, 46–60 years old and 60 years old and above. The quantitative assessment of these dimensions can then be used to effectively depict the different interest performance and acceptance level of the audience of each age group when facing the Kunqu Opera “Peony Pavilion”. Specific data analyses are shown in Table 1.

Table 1: Analysis of user preferences across different age groups for kunqu opera.

Table available in the original PDF paper.

Age Group Understanding of the Play (1–5 Points)

Cultural Identification (1–5 Points)

Emotiona1 Engagement (1-5 Points)

Acceptance of Modern Adaptation (1–5 Points) 10–18 years 3 2 3 4 19–30 years 4 3 4 3 31–45 years 4 4 5 2 46–60 years 5 5 5 3 60+ years 5 5 5 1

Specific data shows that younger users (18–30 years old) prefer the combination of Kunqu with modern technologies. In contrast, users over the age of 30 preferred traditional stage performances (62.39%) and focused on

AI-Driven Personalized Multisensory Design of Cultural Heritage 55

music (60.68%) and stage design (45.3%), reflecting their strong attachment to traditional aesthetics.

In addition, 23.93% of the respondents said they were “generally interested” and 18.8% said they were “not interested”, due to cognitive barriers to the complex plot and traditional forms of Kunqu. The most valued element when watching kunqu was the music and accompaniment (60.68%), followed by the actors’ performance (43.59%) and the stage set (45.3%). In terms of themes, ancient love stories, especially The Peony Pavilion, were the most popular, with a high support rate of 72.65%, providing a clear direction for personalised design. Based on this user profile, the study focused on redesigning the youth version of The Peony Pavilion. The adapted version of The Peony Pavilion retains the classic plot of the play while incorporating innovative visual elements and modern theatre techniques to cater for contemporary tastes.

The survey shows that 59.83% of respondents find Kunqu plots complex and 57.26% of the audience supports the adapted Kunqu, so this study will dynamically adjust the cartoon content to meet diverse needs.

MULTI-SENSORY DESIGN ELEMENT EXTRACTION (EMOTIONAL DESIGN) Visual Elements

Traditional Patterns The traditional patterns of Kunqu opera costumes are inspired by auspicious motifs and classical modelling arts, and common elements include flowers, animals and mythological symbols (Docin. n.d.). The stage costumes of The Peony Pavilion serve for the shaping of the characters, with different costumes for different characters and different costumes for the same character on different occasions, which also satisfy the audience’s pursuit of visual richness. In the design of this study, relevant key elements were extracted and recreated. Chart 2 shows the visual elements extracted from the female and male protagonists in the audio comic and the related intentions.

Table 2: Visual elements extraction in the costume design of the main characters in ‘The Audio Comic of The Peony Pavilion’.

Table available in the original PDF paper.

Character Costume Color

Main Motif Motif Symbolism

Costume Changes and Scene

Color and Expression of Motif

Costume Functions and Expression

Du Liniang White,

Butterfly Lightness, Fine details in

“Dream” scene: Through

(The heroine)

Light- colored fabrics

love, youthful romance

embroidery, soft colors contrasting with the fabric, rich in layers

Costume adorned with butterfly motifs, reflecting Du Liniang’s youthful romantic atmosphere

butterfly motifs, the costume reflects Du Liniang’s romantic, youthful emotional expression Continued

56 Zeng and Zhou

Table 2: Continued Character Costume Color Main Motif Motif Symbolism

Table available in the original PDF paper.

Costume Changes and Scene

Color and Expression of Motif

Costume Functions and Expression Through motifs

High, firm, Subtle colors, clear

“Dream” scene:

Liu White,

Plum Yellow, Green elegant, passionate love Mengmei (The hero)

Plum blossom motif, representing hope and love

outlines of the motif, showing clarity and strength of character

Blossom, Bamboo, Peach Blossom, Orchid, Chrysanth emum

like plum blossoms, the costume expresses Liu Mengmei’s steadfast love and emotional depth Use of Colours The Peony Pavilion, as a classic of kunqu, has been interpreted many times and has gradually taken on a new look. In Mr Bai Xianyong’s youthful version of The Peony Pavilion, the choreography combines traditional and modern elements, and incorporates set changes, lighting changes, the realistic rendering of the curtains, and the freshness and lightness of the actors’ costumes with their voluptuousness and beauty (Baidu, n.d.).

Therefore, in the audio comic redesign of this study, the colour elements in the youth version of The Peony Pavilion will be extracted and used. Take the use of colours in the play “A Dream” as an example: Du Liniang wears a goose-yellow coat and a pink cape, showing the delicacy and vitality of youth, while Chunxiang’s emerald-green costume complements Du Liniang’s attire.

Sense of Hearing The auditory design consists of sound effects and voice acting. Kunqu’s singing style is characterized by its delicate, slow, and elegant tones, with clear, soft timbre and emotional subtlety (Jiang, 2015). The questionnaire results indicate that the respondents are most concerned with music and accompaniment (60.68%). Therefore, it is necessary to appropriately balance the original tune and the proportion of adaptation. The music in the audio comics must preserve the traditional essence of Kunqu, featuring instruments like pipa, guzheng, and erhu.

One key feature of Kunqu singing is the technique of aligning the melody with the lyrics. This voice technique helps performers navigate the use of both true and false tones, ensuring both emotional depth and clarity. To enhance Kunqu’s appeal to a broader audience, while maintaining its emotional depth, the voice acting will be adapted to vernacular Chinese and incorporate modern vocal techniques, adding a sense of lightness and contemporary flair. The image below shows an excerpt from the Dream Awakening scene adaptation.

Tactile In scene design, through the appropriate introduction of interactive touch elements, can significantly enhance the audience’s emotional experience and sense of immersion. Take the elements in A Dream in the Garden as an example. In the garden scene, when the audience touches the blooming

AI-Driven Personalized Multisensory Design of Cultural Heritage 57

flowers, it can trigger the character Du Liniang’s exclamation or plot twist, which expresses her perception of the beauty of spring; while clicking on the flowing water or the bridge can further enhance the shaping of the natural atmosphere through scene switching or sound generation. In addition, touching Du Liniang’s sleeve can trigger her dynamic reactions, such as looking back or dancing, to show her emotional fluctuations; clicking on a book or a folding fan can help reveal the character’s psychological activities and deepen the emotional connection between Du Liniang and Liu Mengmei. The interactive elements are not only tools for plot advancement, but also deepen the audience’s emotional resonance through the dual role of vision and hearing.

AIGC AUDIOBOOK COMIC DESIGN PRACTICE – TAKING “PEONY PAVILION” AS AN EXAMPLE Application and Characteristics of AIGC Technology

With continued breakthroughs in text, voice, image, video and 3D content generation applications, generative AI technology will simplify the creative process and improve creative efficiency in the field of culture and art. Based on deep learning technology, generative AI can simulate human thinking, consciousness and creativity based on user prompts and instructions, automatically generating creative digital content in various forms, and can be applied to screenplay creation, scene rendering, special effects design, audio processing, face reconstruction, intelligent images.

Script Adaptation Based on the original plot of the ancient text, the author collects the scripts and then transforms the dialogue into vernacular based on the ancient text, which is conducive to the understanding of the plot by the majority of kunqu audience.

Sub-Scene Design The storyboard design is primarily based on the adaptation of the Script for the Youth Edition of Peony Pavilionwhich 72.65% of respondents show a preference for ancient love stories such as Peony Pavilion in Kunqu according to the questionnaire data, enriched with content from the Ming Dynasty novel The Peony Pavilion, and incorporates reference images from the Kunqu stage production. Using the “Dreaming in the Garden” segment as an example, the original text is from The Peony Pavilion (Act 10: “Dream Awakening”). Table 3 Storyboard showing the visualisation of the section ‘A Dream in the Garden’.

Table 3: Storyboard design for ‘A Dream in the Garden’.

Table available in the original PDF paper.

Scene Character Setting Lines Camera Action Split-screen sketch Scene 1 Du Liniang Inside her

“Liniang, I’m Full shot,

Camera moves

leaving.”

(The main female character)

into the room, stopping in front of Du Liniang.

room, warm spring light filters through the window.

camera zooms in, showing her peaceful state.

Continued

58 Zeng and Zhou

Table 3: Continued Scene Character Setting Lines Camera Action Split-screen sketch

Table available in the original PDF paper.

Scholar walks in,

“Xiu Cai, are Mid-shot, Liu Outside Du

Scene 2 LiuMengmei

you leaving?”

Liniang’s room.

stands by the bed, gazes at Du Liniang.

(The main male character)

Mengmei slowly fades, showing his reluctance.

Du Liniang turns, Long shot,

Scene 3 Du Liniang Du Liniang’s “I’m really room, quiet.

closes eyes, drifts into a dream.

tired. Why is it so difficult to find him?”

peaceful room contrasting with the outside.

Du Liniang slowly Full shot, Du

Scene 4 Du Liniang Du Liniang’s “Husband is in room.

Liniang lies in bed, slowly waking up.

the Yellow Hall, looking at plants...”

wakes up, stretches, and looks toward the door.

The old lady Full shot, Old

“The plants are In the garden,

Scene 5 Old Lady (The main female character’mother)

walks slowly, gazing at Du Liniang.

blooming, the birds are chirping...”

outside Du Liniang’s room.

Lady observes Du Liniang from a distance.

Scene 6 Du Liniang Du Liniang’s

“Ah...” Close-up, Du Du Liniang wakes room, near the window.

up, responds softly to her mother’s call.

Liniang closes her eyes, emotions visible.

Character Image Design The traditional Chinese opera costumes have a history of 500 years since the Ming Dynasty, and their styles and modelling have evolved through the Ming Dynasty, the Qing Dynasty and the Republic of China to the establishment of New China (Bai, 2015). Although there are no major fundamental changes, local changes have always been present in the course of continuous improvement of opera choreography. From the comparison between the traditional kun opera costumes and the youth version of “Peony Pavilion” in terms of costume pattern and layout, the youth version of “Peony Pavilion” follows the traditional pattern form of using flowers as a metaphor for human beings, and emphasises the important role of flowers and branches in expressing the characters’ comprehensive characteristics such as their identity, age, character, temperament, and appearance by means of allegories, symbols, and exaggeration, which creates a distinctive, vivid, and evocative external image of the characters. This study will mainly refer to the youth version of The Peony Pavilion and combine it with the original text of the Ming Dynasty novel The Peony Pavilion for character design, and the following table shows the role design of the main characters.

Table 4: Character design element extraction and AI commands.

Table available in the original PDF paper.

Character Costume Design Makeup Design Headwear Design Costume Patterns Du Liniang Color tone: pink,

Soft eyebrows Simple floral

Retain traditional

and eyes, pink lips, gentle and elegant

Kunqu elements, incorporate modern aesthetics,

light purple, representing purity and youth

hairpins or headbands, emphasizing beauty and elegance

Continued

AI-Driven Personalized Multisensory Design of Cultural Heritage 59

Table 4: Continued Character Costume Design Makeup Design Headwear Design Costume Patterns

Table available in the original PDF paper.

Liu Mengmei Color tone: dark Delicate facial

Scholar’s hat or Referencing the youth

green, light blue, representing restraint and depth

features, clean appearance, without excessive decoration

hairpin, highlighting fresh and refined scholarly temperament

edition of The Peony Pavilion, using the Four Gentlemen patterns (plum, orchid, bamboo, chrysanthemum) and other floral patterns Chunxiang Color tone: light

Natural makeup, Small delicate

Peony, rose, hibiscus,

clean and delicate appearance

yellow, light blue, simple and practical style

magnolia and other traditional floral patterns, enriched with chrysanthemum, pear blossom motifs

hair accessories or tied headbands, highlighting liveliness and youthful spirit

Figure 1: Anime character charts for the three main characters (generated by a deep learning model).
Original PDF Figure 1:

Music & Dubbing Based on the existing questionnaire data, this study proposes that the soundtrack design of kunqu should integrate modern music elements, and this integration can not only enhance the sense of rhythm and modernity of kunqu, but also make the plot more easily understood by the public.

AIGC Internationalised Kunqu Audio Comic Interactive Experience Design Interaction Design Overview With the Fragment of “A Dream in the Garden” as an Example 34.19% of respondents identified innovation in performing arts as the next most important area for improvement, following plot and language. Users can trigger plot progression and scene changes by touch or gesture in the dream scenes of The Garden of Dreams. This design allows users to tap on specific areas to reveal hidden plot points, or change character behaviour through action linkages to become part of the storytelling, thus dramatically

60 Zeng and Zhou

enhancing immersion and interactivity. The interactive features of sound effects and soundtracks further enhance the emotional expressiveness of this design. Table 5 demonstrates the feedback and experience in different contexts.

Table 5: Interaction design for three main contexts.

Table available in the original PDF paper.

Emotion Interaction Design Sound and Music Interaction

Effects and Experience

Sadnesss Click specific areas to

Adjust background

Music and interaction

reveal Du Liniang’s sorrow. Slow character actions show helplessness.

music tempo and intensity; deep piano and bass intensify sorrow.

deepen emotional connection, enhancing immersion. Happiness Use touch/gestures to

Adjust background

Interaction allows the

guide the plot to a joyful scene, with happy character actions.

music tempo, using bright strings and upbeat drums for joy.

Dynamic sound and

Adjust sound effects

audience to experience the character’s joy, enhancing immersion. Tension Interactions trigger

for urgency, like rapid drums or sharp sounds.

music heighten tension, intensifying the atmosphere.

quick reactions, altering the plot and building tension.

Virtual Environment and Scene Construction In terms of stage presentation, the Chinese opera stage adopts a minimalist and pictorial style, in which the dramatic situation is entirely completed by virtual movements and programmed figures in the performance, a stage in which the virtual and the real are intertwined with the real and the fake, with a strong assumption. Taking the clip of “A Dream in the Garden” as an example, the scene of “A Dream in the Garden” is constructed with the emotional tension of Du Liniang’s dream intertwined with reality as the core. The overall space is modelled on the classical gardens of Jiangnan, where pavilions, gurgling water, and flowers and trees are exquisitely laid out to create a poetic visual focal point, with the Peony Pavilion, as the core of the emotional expression, placed in the centre of the courtyard and surrounded by peonies and weeping willows, symbolising the rich emotional connotations of the metaphor.

Gesture Operation and Emotional Feedback In the case of the virtual scene of The Garden of Dreams, for example, touch and gesture operations play an important role in enhancing the audience’s interactive experience with the scene. The author presents the content of gesture interaction graphically.

AI-Driven Personalized Multisensory Design of Cultural Heritage 61

Table 6: Gesture interaction content in the Garden of Dreams scene.

Table available in the original PDF paper.

Design Element Interaction Emotional & Narrative

Technical Implementation

Audience Experience

Touchscreen Tap flowers or

Petals fall, echoing Touchscreen with

Enhances

emotion physics engine plants to trigger changes

resonance and engagement Gesture Operation Adjust lighting or

Dream transitions, Gesture

Intuitive, conveys

trigger meetings

emotions

scenes with gestures

recognition with scene control

Dynamic Feedback Touch triggers Petals, breeze,

Real-time Deepens

instant response Kunqu melody

feedback with sound effects

immersion and emotion Lighting & Music Change lighting

Soft dream light, Dynamic lighting

Strengthens emotional mood

and music with the story

darken when awake

and sound design

Multisensory Combine visual, Rich multisensory

Synchronized Bridges emotions emotion with characters

sound, and dynamic feedback

effects and dynamic feedback

Figure 2: The effect of opening the Garden of Dreams scene by gesture interaction.
Original PDF Figure 2:

Interactive Sound Design and Personalised Music Experience Taking the scene of “A Dream in the Garden” as an example, the interactive sound design strengthens the narrative and emotional expression through the in-depth combination of dynamically triggered sound effects and plot development. The personalised music experience creates a unique music presentation for each audience member through the combination of artificial intelligence technology and audience interaction behaviour. The system can adjust the melody, rhythm and timbre of the background music in real time according to the audience’s touch position. The specific design is shown in Table 7.

62 Zeng and Zhou

Table 7: Interactive sound effects and personalised music design for the Garden of Dreams scenario.

Table available in the original PDF paper.

Design Element Interaction Emotional & Narrative

Technical Implementation

Audience Experience

Music & Rhythm Step pace or scene Slow pace for

Music system Experience

triggers change music tempo

adjusts melody and rhythm based on actions

emotional fluctuations through interaction

Kunqu melody, faster pace accelerates rhythm, simulating emotional shifts

Personalized Triggering dream

Transition from Music transitions

Deeply experience

Music sequences alters music

using sound effect engine

solo to ensemble, music darkens after waking

emotional changes and dream atnosphere Emotional Interaction triggers

Nusic moves from Narrative and

Sense emotional

music changes Rhythm Progression elegant to rich, simplifies after waking

emotion algorithms drive music changes

shifts and layered progression

Figure 3: The scene uses subdued as well as sad sound effects to express the emotion.
Original PDF Figure 3:

CONCLUSION

Through AI-driven multi-sensory personalised design strategy, this study proposes an innovative approach to combine traditional art with modern technology, taking Kunqu’s Peony Pavilion as an example. Through digital expressions such as audio comics, it breaks through language and cultural differences and opens up a new path for the international dissemination of kunqu. The study shows that AI technology can parse the cultural connotations of kunqu, achieve digital reproduction of cultural elements, and enhance the emotional resonance of the audience through personalised design. In the future, AI will play a greater role in cultural heritage protection and dissemination, and provide a feasible reference for the modernisation and dissemination of other intangible cultural heritage.

REFERENCES

Bai, L. (2015). Traditional Chinese drama costumes’ evolution and innovation in the youth version of The Peony Pavilion. Drama and Cinema Monthly, 2015(5), 108–109. Retrieved from https://wenku.baidu.com/view/ c21e1a095b0216fc700abb68a98271fe910eaf82.html?_wkts_=1739287587629.

AI-Driven Personalized Multisensory Design of Cultural Heritage 63

Baidu. (n.d.). Traditional Chinese drama costumes’ evolution and innovation in the youth version of The Peony Pavilion. Baidu Wenku. Retrieved from https://wenku.baidu.com/view/ c21e1a095b0216fc700abb68a98271fe910eaf82.html?_wkts_=1739288327053. Docin. (n.d.). Analysis of the “Youthfulness” in the Costumes of the Youth Edition of The Peony Pavilion. Docin. Retrieved from https://www.docin.com/ p-1681926956.html. Jiang, F. (2015). “Kunqu Music and Vocal Style.” Yi Hai, 2015(10), 68–71. Zhou, J., Liu, X. (2024). Digitization of Cultural Relics: Augmented Reality Display

of Ru Ware. In: Stephanidis, C., Antona, M., Ntoa, S., Salvendy, G. (eds) HCI International 2024 Posters. HCII 2024. Communications in Computer and Information Science, vol. 2116. Springer, Cham.

Cover of the proceedings volume Affective and Pleasurable Design

Explore the complete digital volume

Affective and Pleasurable Design

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-40-3

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →