Siyuan Du (2024). Virtual Scene Design of Peking Opera "Farewell My Concubine" Based on Human-Computer Interaction Technology. In: Human Factors in Communication of Design. AHFE Open Access, vol. 136, pp. 138–152. https://doi.org/10.54941/ahfe1004976
In the continuous development of interactive art today, people pay more and more attention to the diversity of interactive methods, and Chinese opera performance has also derived many emerging technology carriers. For example, the virtual scene opera performance is based on human-computer interaction technology, which breaks through the limitation of time and space of opera performance, transforms from the theatre stage space to various types of virtual space, and makes use of various types of carriers and media such as image and sound to perform. Actors in the virtual space can create an interactive continuation of the virtual space through body changes or spatial scheduling. This not only enables the audience to appreciate the imagery of the "four skills" of Singing, Recitative, Acting, and Acrobatic Fighting in the choreographed movements of the opera but also to appreciate the spatial image of the "image outside of the image" and "image outside of the meaning".The virtual scene design of the Peking Opera "Farewell My Concubine" realizes the functions of multimedia participation in performance and construction of virtual scenes through digitally constructed performance scenes, which can be quickly applied to diversified opera performances, and at the same time, it has strong theoretical and practical application value in expanding the scope of the audience and expanding the effectiveness of trans-media digital opera dissemination, providing new possibilities for further combining the virtual scene design of the opera with human-computer interaction technology.
The Peking Opera “Farewell My Concubine”tells the story of General Xiang Yu, who was trapped in the war and drank wine and sang sadly. Yu Ji performed a sword dance to console Xiang Yu, and in order to relieve Xiang Yu’s worries, she cut her own throat after the sword dance. Chinese opera is a comprehensive art, it contains from the venue to the emotional aspects of the content, at first the opera stage scene is more through the enhancement of visual effects to create a variety of psychological implica- tions of the environment, but after the development of the times, the audience has been accustomed to the contemporary film and television media highly
Virtual Scene Design of Peking Opera “Farewell My Concubine” 139
visual means of expression, has been unable to fully comprehend the highly condensed generalization of the performance of the opera way.
The appearance of the virtual opera scene, so that the interactors can be integrated into the work, the original non-existent mountains, rivers and seas in the opera performance are reproduced, just like being in the realm. The vir- tual scene design of the Peking Opera “Farewell My Concubine” introduced in this paper is based on human-computer interaction technology, through the form of “scene immersion + role substitution” to reproduce the scene of Yu Ji’s performance of sword dance. Based on the human-computer interaction technology, the article discusses the mechanism of “interaction-experience”in the virtual space, and looks forward to future development and application, further discussing the design characteristics and construction requirements of the scene space.
The innovative points of the article include:
1. The virtual scene design of Peking Opera “Farewell My Concubine” based on human-computer interaction technology realizes the perfor- mance and construction of multimedia participation through the digitally constructed performance scene. 2. Kinect bone tracking technology is used to realize the spatial positioning and tracking functions of human bone joints, and provide real-time 3D bone coordinate information of pedestrians. 3. The hierarchical strategy is adopted to conduct the bone tracing determi- nation of the opera body, and the movements of the five related combi- nation modes are summarized into a category, and then each category is classified. 4. Establish Yu Ji sword dance material library to store and collect all the action effects that can be triggered during shooting, such as “Plant- ing sword” on the right side of Shabu Jianzhai, “Cross with two swords”, “Cloud hand”, “Reverse stabbing squatting Body bright”, “Back flower”, “waist down”, etc. 5. Design a variety of virtual scenes, such as the autumn scenery of the West- ern regions, the lotus pond under the moon, the hibiscus on the dance, the lake and the water and the sand, to express different emotions and atmospheres with different colors and emotions.
VIRTUAL SCENE DESIGN SYSTEM FOR HUMAN-COMPUTER INTERACTION Introduction to Kinect Sensors
Kinect is the XBOX360 somatosensory peripheral peripheral officially announced by Microsoft at the E3 exhibition on June 2, 2009, as a more mature commercial human body motion sensing device, which is currently heavily used in somatosensory games. Kinect, as a sensor, is essentially just an input device. It can provide three major types of raw data information, including depth data streams, colour video streams, raw audio data, etc., and corresponds to three kinds of functions, such as bone tracking, identity recog- nition, and voice recognition. Skeletal tracking is the foundation of Kinect’s
140 Siyuan
somatosensory operations, and it requires the system to quickly construct the player’s torso, limbs, head, and even fingers based on skeletal joints within the allowable latency (Yin, 2013).
Kinect Skeletal Tracking Principles Skeletal tracking is one of the highlights of the Kinect device. Kinect’s skeletal tracking function can realize the spatial positioning and tracking of human skeletal joints, and provide real-time 3D skeletal coordinate information of pedestrians, as shown in Figure 1, the Kinect 3D joint coordinate data is based on the Kinect camera coordinate system as the reference coordinate system.
Figure 1: Skeleton tracking map in Kinect camera coordinate system (left).
The Kinect device acquires a three-dimensional image of the person in the scene. Unlike a two-dimensional image, this three-dimensional image con- tains not only the pixel coordinates (x, y), but also the distance (z) from each point in the scene to the Kinect camera, and this image is the depth image captured by the Kinect.
The Kinect device first uses an infrared projector to emit an infrared laser into the scene, the laser will reflect on the surface of a rough object, and through the infrared camera you can find that the reflected laser will gen- erate a randomly distributed laser scattering, when the objects in the scene are at different distances from the laser source, the laser scattering pattern will also be different, so the Kinect can mark all the objects in the scene with the emitted infrared laser scattering. This allows Kinect to mark all the objects in the scene with the scattered spots of the emitted infrared laser. Then, Kinect divides multiple equidistant reference planes within its effective range of vision (the effective range of vision of Kinect is 0.8∼4m), uses the infrared camera to save the scattering images of the reference planes one by one, and performs the inter-correlation operation on all the scattering images to get the three-dimensional histogram of the whole scene, and finally uses Poisson’s equation to smooth the rough three-dimensional histogram to get the depth image.
Figure 2 shows a depth image captured by Kinect that contains the depth information of the character. The first step for Kinect to realize the skeleton tracking function of the character is to separate the character. At present, image segmentation technology is very mature, but for Kinect, before the
Virtual Scene Design of Peking Opera “Farewell My Concubine” 141
image segmentation must first find the specific position of the character in the depth of the image, Kinect is designed for this problem, a “big word” tracking mechanism, and then use image processing technology to segment it out, to get the depth of the image contains only the character. The depth image contains only the character.
Figure 2: Depth image collected by KinectV1 (Tao, 2013) (right).
According to the understanding of the human body structure, the human body can be divided into five parts, that is, the torso, the left arm, the right arm, the left leg, and the right leg, and the torso is the important part to support the human body (Hong, 2017). Some of the human body’s waist joints can reflect the information of its movement characteristics, while the movement information characteristics of the hands and feet are expressed by the joints of the limbs in this part. The results of the division of the five major parts of the human body are shown in Fig. 3.
Figure 3: The human skeleton map of Kinect.
Through the division of the human body structure, the combination of these five parts can be used to represent some basic movements of the human body, so a hierarchical strategy is used in the categorization (Caijie et al., 2015). The first level: is to categorize the movements of the five related com- bination patterns into one large class. For example, a movement with only two arms is a combination of the second and third parts, which is the result of a rough movement categorization. The second level: is to re-categorize the movements of the same combination mode to determine the detailed classifi- cation of the movements. The 20 joint angles of the human body are used to
142 Siyuan
validate the joint angle eigenvectors formed by the projections on the two- dimensional plane, and this is used as the first coarse categorization feature of human movement. The features of the human body in the same combi- nation mode are extracted according to the principle of kinematics (Guowei, 2016).
Skeletal Tracking Determination Methods for Opera Stances In the concept of anatomy, the torso refers to the remaining body parts from the cervical vertebrae to the tailbone after removing the head and limbs, also known as the “middle” of the human body, in which the “waist” is a carrier of the top and the bottom, connecting and coordinating the upper body and the lower body of the human body’s core, in the action plays a The waist, as the core of the human body, connects and coordinates the upper and lower bodies, playing a vital role in the movement (Wang, 2018). Therefore, it is necessary to explain the character colour of the torso movement in opera. The various forms of the torso in opera performances can show the differences in character and age, which are not only from life but also the purification and typicalization of the forms of characters in life. We analyze the character connotation and physical expressiveness behind each torso form with respect to the four main forms of the torso: upright, curved, neutral and twisted.
Different footwear lineages have different body rhythms and ways of expressing strength: in a physiological sense, the human torso in young and middle-aged people will show a relatively upright and erect state, with a sense of actively expanding the body’s territory and space, in order to show its youthfulness. As the main subject of the article is the young Yuji, the move- ments of Yuji are quick in coming out and quick in coming back, and skilful strength is the main focus. Their flexible posture is made more delicate by the natural twisting of their torsos, and her lively nature is reflected in her physical form as tilting her head, shifting her body, being shy of her ribs and crotch, and her posture is in a constant state of change, even when she walks, her torso is tilted forward, and both sides of her waist are twisted up and down, which reveals her flexible and playful character.
Therefore, the head, chest, small abdomen and lower limbs of Yuji’s body form of the flower shirt are combined with the division of the five major parts of the human body to make the body form of the opera more scientific and standardized, as shown in Figure 4.
Figure 4: Schematic diagram of the five major parts of the opera body form.
Virtual Scene Design of Peking Opera “Farewell My Concubine” 143
User Interaction Action Acquisition and Virtual Scene Determination The acquisition of user interaction actions is the technical centrepiece of somatosensory interaction, which is the core of interaction experience, the foundation of science learning, and the focus of interest stimulation. There are various ways to determine the user interaction actions, developers can determine them based on scientific data support, user behaviour, interaction cultivation cognition, or even create their own rules to restrict the formu- lation. This paper analyzes the Kinect hardware technology, acquires data through image sensors, and obtains user interaction actions through multiple image topology reactions on the skeletal data model and simulation confir- mation, in order to improve the efficiency of the acquisition and filter the most suitable interaction actions.
In the first stage, the digitized information of the interaction action is acquired in the following way.
After discretizing the body form of opera into five parts, the two detailed parts of hand and foot, which are not obvious in action recognition, are ignored and correspond to the five types of meta-actions one by one. Here, meta-movements refer to the movements of each limb part after the dis- cretization of the skeleton, including head movements, torso movements and limb movements. The recognition method of meta-actions is to localize the actions of these parts independently. The action recognition based on bone discretization is to recognize the overall action of the human body (vertical displacement action) under the condition of unchanged body posture, then discretize the bones for the recognition of meta-actions, and finally realize the recognition of multiple actions based on the combination of meta-actions (as in Fig. 5).
The steps to achieve this are as follows: (1) Judging body posture
Body posture is the orientation of the body in the standing state. There are three body orientations relative to the camera: facing forward (the front of the body faces the camera), facing left (the body turns to the left), and facing right (the body turns to the right). The body orientation was determined by simultaneously judging the difference between the b-values of the two shoulder joints and the two hip joints (subtracting the left side joint from the right side joint). When the difference is equal to 0, the body faces forward; when it is greater than 0, the body faces left; and when it is less than 0, the body faces right. (2) Recognize the overall action.
The overall action consists of jumping and squatting and is recognized by judging the difference in Y coordinate values of the hips minus the origin of the coordinate system. It is set to be a jumping action when it is greater than 2 and a squatting action when it is less than −1. (3) Since human body movements can have six directions (forward, back- wards, left, right, up and down) in three-dimensional space, when the body orientation changes, the coordinate components used for recogni- tion also change. The following is an example of the judgment conditions
144 Siyuan
of meta-action when the body is facing forward, and other orientations can be judged according to similar logic.
x Neck movements. The meta-movement of the neck part has three direc- tions: forward, left, and right, and is recognized by judging the difference in the position of the head minus the centres of the two shoulders. When the Y coordinate difference between the two is equal to 0, it is forward head bow- ing. When the coordinate difference between the two is greater than 0, it is a head tilt to the right, otherwise, it is a head tilt to the left.
y Trunk movements. The meta-movement of the trunk part has three directions: forward, left, and right, and is identified by judging the positional difference between the centres of the two shoulders minus the hips. (When the difference between the two Y coordinates is less than 1, it is forward bending; when the difference between the two coordinates is greater than 0, it is right bending, otherwise, it is left bending).
z Limb movements. The meta motions of the limb parts have four direc- tions: leftward, rightward, upward, and downward. The meta-movements of the four parts of the big arm, small arm, thigh, and calf are recognized by judging the positional difference between the elbow and shoulder joints, the positional difference between the wrist and axial joints, the positional differ- ence between the knee and hip joints, and the positional difference between the ankle and knee joints, respectively. The left big arm, for example, was divided into four quadrants of movement. When the x-coordinate difference of the left elbow minus the left shoulder is greater than 0, it is extended to the left, otherwise, it is extended to the right: when the y-coordinate difference is greater than 0, it is lifted upward, otherwise, it is placed downward. The meta-movement of several other limb parts is similar to that of the left great arm.
This work is expected to achieve meta-action recognition for three body orientations (forward facing, left facing, and right facing).
Figure 5: Flowchart of human-computer interaction of virtual scene.
Virtual Scene Design of Peking Opera “Farewell My Concubine” 145
In the second stage, the virtual scenarios are identified in the following way.
First of all, digitized image processing is carried out to determine the key skeletal points, calculate the difference of joint coordinates, and compare the captured actions with the standardized and scientific actions; if there is no obvious error or large error, the “standard interactive action” can be positioned; if it is far from the scientific and standardized actions, it will be re-captured. After two stages of information acquisition and data pro- cessing, an accurate library of somatosensory interaction movements can be obtained. This work is expected to establish a Yu Ji sword dance material library, storing and collecting all the action effects that can be triggered during the shooting, such as shabu shabu sword planting the right side of the “plant- ing the sword”, “double swords on the cross”, “cloud hand”, “Crouching Crouch”, “Returning Flower”, and “Lower Waist”, each of which lasts about 10 seconds, while the overall performance time is estimated to be about 3 minutes. Users can enter the virtual scene after the successful body recogni- tion decision, the output device can enter the virtual scene, and on the screen of the output device, the virtual scene corresponding to the recognized suc- cessful body posture is displayed, and the sword dance of Yu Ji is appreciated and experienced.
The performance of Yu Ji’s sword dance is the most perfect way to present Yu Ji’s role, Yu Ji’s emotional changes and the evolution of the plot are all presented in this play, and the sword dance is also one of the classic sequences of Farewell My Concubine. When Yu Ji dances with the sword and reveals her grief but must force herself to be calm and smile, the audience is able to see the delicate inner emotions of Yu Ji through the setting of the scene, which is very difficult to perform perfectly.
Therefore, in the process of scene design, it is necessary to show the entan- gled two sides of Yu Ji’s state of mind, and the scene design should have a polarized contrast in order to bring out Yu Ji’s emotions. In the scene design, the design will externalize the emotions of Yu Ji’s sword dance and can play a role in setting off the action of Yu Ji’s sword dance. Such as in the sword dance action combination [deep night] line string, the fourth subsection, Yu Ji in the mouth of the stage to turn inward to play three “back to the flower”. And then the “big sword flower” to the centre of the stage mouth on the sword, facing the front stage to play “sword flower”. In place of the left foot on the step, the right sword forward a stab, two swords on the cross face in the “waist”, turn over, the huqin line of strings to this point, in the sound of three gongs in the “cloud hand” to turn around and split the sword towards the outside of the light to stop. (The above section goes from the centre of the stage mouth to the inside in a straight line, and then from the inside to the outside in a straight line back to the centre of the stage mouth.) At the end of the sword dance here, the two swords on the ground, according to the sword, the performance of Yu Ji has been unable to support the spirit (Xiaoxiao, 2021). In order to strengthen the audience’s understanding of Yu Ji’s movements, the scene design needs to be strongly associated with Yu Ji’s inner thoughts and performances.
146 Siyuan
CASE STUDIES Emotional Color-Guided Scene Design
Colour has a special cultural connotation in Chinese local culture. However, due to the development of the times, the connotation of colour has changed under the influence of different political, economic, and life fields on the basis of inherited history and culture (Yanhong, 2008). Since it is not possible to go back directly to the past decades of people’s psychological semantics of colour in a group, it has been found that the modern Chinese dictionary, which has been finalized for many years, basically reflects the semantic culture of the Chinese language in the mid-to late-20th century. Therefore, it can be used as a retrospective and holistic research object. Table 1 shows the semantics of colour vocabulary for the eight colours that appear most frequently in the design of this scene, which serves as a reference basis for the expression of feelings to guide the colour design.
In the scene design, the romantic and dreamy style is generally utilized, with atmospheric and open imagery, and through different combinations of particle light effects, the scene presents different artistic effects. Upright mountains appear behind the actors in turn, with occasional auspicious blips, ranging from heavy mountains, and lonely smoke in the desert, to the moonlight in the lotus pond, adding a touch of mystery to China’s dazzling scenery.
A total of nine scenes are set: deep mountain streams, the flooded universe, flying flowers and butterflies, falling flowers, autumn colours of the western region, lotus ponds under the moon, hibiscus on the dance, lake and water, and flat sands and mangled.
High mountains and deep streams (as shown in Figure 6). The blue colour symbolizes the coldness of the cold pool and the peaks, this colour is sensible, and calm, and can better blend with Yu Ji’s strong and resolute will. The clouds and smoke on the mountain peaks are like the undulating waves of the sea, surging and surging, people want to find out, at this time, Yu Ji’s sword cuts through the sky, breaking through the clouds and fog, so as to set off Yu Ji’s vigorous posture and courageous spirit.
The Flooded Universe (shown in Figure 7). The Universe is an endless life, rich in power, but it is also strict order and complete harmony. The vastness of the universe is dotted with stars, representing constant and soft serenity. The whole scene is dominated by the dark purple-black colour of the illusion, with the darkness vaguely permeated by the haze, in which there are vaguely visible light gold-coloured tiny filaments floating, adding a touch of mystery to the scene, and the floating light in the distance cuts out the spatial sense of the universe.
The flower flies and the butterfly dances (as shown in Figure 8). Pure white butterflies around the same white lotus flowers rotating and fluttering, wher- ever they go raised the crystal shimmering, Yu Ji’s sword dance can still tease this small butterfly, just like her sword wind harsh but everywhere tender- ness, highlighting the Yu Ji sword skills of the high, more prominent Yu Ji soft and gentle heart.
The falling flowers are fluttering (as shown in Figure 9). Dancing under the flowers better set off Yu Ji’s softness, she turned the changing sword flower with a fluid look, the sword in her hand was sometimes as slow as
Virtual Scene Design of Peking Opera “Farewell My Concubine” 147
a swimming cloud, sometimes as fast as lightning, the flowers seemed to be unable to withstand the falling, highlighting the Yu Ji’s rigid and flexible character traits.
Autumn colours in the Western Region (shown in Figure 10). Yellow and blue tones as the main colour scheme, reminiscent of the distant Tibet, blue represents the sky above, and yellow represents the land underfoot, the har- vest season, always allows people to harvest new hope. In the scene, white clouds float in the sky, occasional birds fly through the air, and the source of life flowing from the land shines with the breeze, making the whole picture very dynamic.
A lotus pond under the moon (shown in Figure 11). “Appreciating the lotus” and “enjoying the moon” have been traditional and elegant customs in China since ancient times. The scene will be the lotus and the moon fusion in one, the night falls, looking at the bright moon in the sky; the lotus swaying with the wind, enjoying the high lotus stretch; Yu Ji in the surface of the pool dance, foot dialling the waves, ripples in the blue water.
Dancing on the Hibiscus (shown in Figure 12). The huge lotus platform in the centre of the image is visually appealing, with its petals floating in dark gold and swaying in the wind, and the swaying of the platform indicates the instability of the situation and the hidden crisis. The fact that Yu Ji is dancing on the lotus platform is also a metaphor for the dangerous journey Yu Ji has embarked on.
Lake light and watercolor (shown in Figure 13). The lake and the green mountains reflect each other, the distant mountains in the mirror-like vast lake reflection like a piece of translucent emerald shimmering glittering lustre. But this long night is a hint of the crisis, the tension is about to flare up. In the distance, daylight is about to break through, which abstractly represents the predicament Yu Ji is in, but her inner conviction is as strong as this beam of light, which will never fade away.
Flat sands are reckless. (shown in Figure 14).The sky and the earth are darkened by the yellow sand that covers the sky, revealing a layer of deep red; a disk of rounded bright sun against the prism of the cliffs, shimmering and golden, the blazing sun scorches the whole environment with layers of heat waves, presenting an illusory sense of distortion, also symbolizing the advent of the crisis. It symbolizes that Yu Ji is now in a desperate situation, but her tenacious spirit of fighting against fate is reborn in the blazing sun.
Table 1. Semantic organization of modern Chinese color words.
Table available in the original PDF paper.
Red Orange Yellow Blue Purple Black Brown Revolutionary Harvest Golden Dejected Elegant Unclear Rustic Radical Sweet Exalted Cold Mysterious Confused Natural Frenzied Warm Dazzling Wide Valued Secretive Steady Busy Rich Salubrious Romantic Gloomy Conservative Excited Puissant Sapiential Magic Lethal Festival Failed Distant Wicked Solemn Successful Erotic
148 Siyuan
Figure 6: The effect of high mountains and deep streams and color mood percentage.Figure 7: Flooded universe effect and color mood percentage.
Virtual Scene Design of Peking Opera “Farewell My Concubine” 149
Figure 8: Flower fly butterfly dance effect and color mood percentage.Figure 9: Falling effect diagram and color mood percentage.
150 Siyuan
Figure 10: Western autumn color effect and color mood percentage.Figure 11: The effect of the lotus pond under the moon and color mood percentage.
Virtual Scene Design of Peking Opera “Farewell My Concubine” 151
Figure 12: Dance on hibiscus effect and color mood percentage.Figure 13: Lake effect and color mood percentage.
CONCLUSION
Human-computer interaction design is an important research content in the field of cross-media art of opera in recent years, and the design has the advan- tage of interactability, which shortens the distance between modern audiences and the art of opera to a certain extent. The article is based on the principle of character recognition of Kinect 3D skeletal data, which not only provides
152 Siyuan
the 3D joint information of the human skeleton, but also divides the human body into five parts through the knowledge of the human body structure, and extracts the features of the human body of the same combination way in accordance with the principle of kinematics, which helps to complete the work of recognizing the body posture of the opera. At the same time, nine virtual performance scenes based on emotional colour are designed, and the user can enter the virtual scene after the body recognition decision is suc- cessful, and the device at the output end can enjoy the experience of Yu Ji’s sword dance. Based on this premise, this paper carries out theoretical analysis and creative practice of virtual scene design from the emotional colour, visual characteristics, musical connotation, creation method to presentation mode and other important aspects of virtual scene design under human-computer interaction design, so as to provide a reference for the innovative develop- ment of opera art in the virtual era. At the same time, through the creation of virtual scene design of Peking Opera “Farewell My Concubine” based on human-computer interaction technology, the virtualization dissemination and protection of opera culture is realized in the form of virtual scene installa- tion art, which enhances the interactive experience and sense of immersion of opera culture, realizes the inter-temporal dissemination of opera culture in a colourful and illustrative way, and achieves the maximization of the cultural value and artistic value of opera art.
REFERENCES
Ch’i Caijie, Dai Guobin. Design and development of Taiji (fixed-step) push hand movement recognition system[J]. Journal of Wuhan Institute of Physical Educa- tion, 2015, 16(08): 52–56. Dong Xiaoxiao. Research on the Generation and Dissemination of Farewell My
Concubine[D]. Shanxi Normal University, 2021. Hou Yanhong. Psychosemantic characteristics of color information and the influence of “implicit” color information on emotion and cognition [D]. Fourth Military Medical University, 2008. Nan Qiu Hong. Research on sports video analysis and recognition in the process of
sports training[J]. Modern Electronic Technology, 2017, 40(11): 68–71. Shi Man Yin. Research on KINECT technology and working principle[J]. Journal of Natural Science of Harbin Normal University, 2013, (3): 85–87. Wang Guowei. Simulation of track and field error movement recognition based on
visual images [J]. Computer Simulation, 2016, 33(04): 274–277. Wang X. Research on body language of traditional opera dance [D]. China Academy of Art, 2018. Yu Tao. Kinect application development in practice: The most natural way to talk to
the machine[M]. Beijing: Mechanical Industry Press, 2013: 30–69.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Communication of Design
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-12-0
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional