Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

The Effects of Happy and Sad Dynamic Digital Art on Relieving Stress

1 Department of Industrial Design, Eindhoven University of Technology, Eindhoven, Netherlands ↗
2 College of Computer Science and Technology, Zhejiang University, Hangzhou, China ↗
3 Guangzhou Wanqu Cooperative Institute of Design, Guangzhou, China ↗
Publication year
Digital Object Identifier10.54941/ahfe1004087

Abstract

Experiencing psychological stress due to the demands of modern life is common, but engaging with innovative digital art has the potential to alleviate this stress and provide a source of relaxation. Moreover, the interactive and dynamic nature of digital art offers diverse and distinctive experiences. The aim of this article is to determine if the process of transferring dynamic digital art images while playing happy and sad music can have an impact on an individual's stress level. For the experiment, two congruent audio-visual digital art videos were used as stimuli to assess their potential stress-relieving effects. A total of 24 participants were invited to participate and were divided into 3 groups. To induce a stress response in the participants, the Trier Social Stress Test was employed in the experiment. Following this, two separate groups were shown videos with different emotional tones, namely joy and sadness. The third group was the control group. The study assessed changes in participants' stress levels before and after the experiment using two tools: the State-Trait Anxiety Inventory (STAI) and Empatica E4. According to the study, both happy and sad videos were effective in reducing stress levels. The findings of this research could inform the development of digital art as a potential tool for stress management and emotional intervention.

INTRODUCTION

Research showed stress is part of everyday life (at home, at work, in per- sonal relations), as well as a psychological condition that is part of a crisis (trauma, wartime, illness, etc.) (Lazarus, 1999). In psychology and biology, the term “stress” is applied to describe a response or reaction to an exter- nal event or interference that disturbs and jeopardizes the functioning of an organism. People used tranquillizing medications to cope with stress, which had many negative contraindications and side effects (Bandelow et al., 2015). Therefore, many researchers examined the effects of non-pharmacological therapeutic interventions on the prevention and management of stress (Case and Dalley, 2006).

While humans are born with creativity in their most active minds, art can nourish the spirit of humans by its abundant spiritual nourishment in return, which enriches the human experience (Hagood, 1990). The wide variety of

© 2023. Published by AHFE Open Access. All rights reserved. 796

The Effects of Happy and Sad Dynamic Digital Art on Relieving Stress 797

art does not only give various inspiring philosophies but also gives people leisure and satisfaction. Art therapy (Martin et al., 2018) has become a com- munication bridge between art and the psychology field, which can help people to deal with stress and anxiety. In a broad sense, art therapy refers to a treatment method that includes all kinds of art forms, including pain- ting, dancing, music, etc. It regulates the stress and anxiety of life through various art activities (Lange et al., 2022).

The phenomenon of judging the emotional state of oneself and others by integrating multi-channel emotional information is called multi-sensory inte- gration (MSI) (Roswiyani et al., 2017). This phenomenon is also common in daily life and is an important topic of emotion research. When the emotional information among channels is incongruent, it will interfere with emotio- nal processing of individual’s, that is, emotional conflict effect (Misselhorn et al., 2019). When the emotional information from different sensory chan- nels is congruent, the efficiency of emotional processing can be improved, and the individual’s emotional response can be faster, more accurate and stronger. That is the advantage of multi-channel integration, also known as the mood-congruity effect (Bower and Gordon, 1981) (i.e., emotional congruence effect).

Stress causes biological responses, so it can be measured physiologically. For example, changes in skin conductivity (Setz et al., 2010), heart rate vari- ability (Sun et al., 2014) and heart rate (Sayette, 1999). Several studies have focused on the development of wearable technology that allows unobtrusive tracking of physiological measurements. Therefore, various wearable devices can track physiological indicators (e.g., Empatica E4 wristband). In our study, we collected sensor and self-reported data to verify that we did induce stress in participants throughout the experiment. We used the heart rate variability (HRV) extracted from the data of Empatica E4 and collected self-reported data using the State-Trait Anxiety Inventory (STAI) (Spielberger, 2010).

The work presented in this article was to combine audio and visual chan- nels in digital art as stimuli to find out if happy and sad videos could have the stress relieving effects. In our previous study, we have made 4 types of digital art videos containing different emotion congruence to conduct experiments on participants under pressure and analysed them through self-reported data. The study showed that happy and sad videos did have certain effects on reli- eving stress. Therefore, this study aims to combine self-reported data and physiological data for analysis.

RELATED WORK

People’s visual perception pays attention to the changes in colours and graph- ics. The dots, lines and surfaces consist of the pictures, the differences among geometric graphics, the colours, the size of space, and the textures. These factors, either alone or in combination, will produce different psychological effects of visual feedback and have a guiding role in reflecting emotions. In paintings, the colour gives people the most direct and effective intuitive fee- lings. The effect of colour on emotional perception could be found in internet web page design (Demir, 2020) and physical space (Kurt and Osueke, 2014).

Space, where its dominant colour was red, would be more possibly described as “stimulus induction”, which was identical to Birren’s viewpoint (Birren, 1950). Music is an abstract symbolic language with no specific references or asso- ciations. Nevertheless, its intrinsic pattern and structure convey meaning to our brain (Hargreaves et al., 2005). It can act as a powerful sensory stimu- lus, engaging the brain in retraining neural and behavioural functions that can be applied to non-musical contexts in everyday life. Music played in shopping malls affects customer behaviour, improves spatial abilities, increa- ses moods and arousal level, engendering an attitude towards compliance to any message mediated by music, influences pacing and timing of movements (Thompson, 2009) and conveys emotional meaning.

In comparison with traditional art, the strength of digital art is apparent. The emergence of dynamic media, e.g., animation, audio, video etc., has broken the monotonous traditional painting art, further enriched the visual sensation, and increased interactivity and entertainment. In our work, we combined music and the abstract image transfer process as video stimuli. There were 2 types of videos containing corresponding emotional congru- ence. The experimental study measured the changes of people’s HRV data before and after watching happy and sad videos while in tension.

Music not only reduces physiological arousal, but also affects emotional states. This may be attributed to the effect of music on brain areas, such as the amygdala, which is responsible for processing emotion. Pleasant musical experience increases the emotional valence (the felt happiness), which has the stress-reducing effect (Jiang et al., 2006). An increased dopamine in the mesolimbic pathway rewards brain system has been shown to be associated with these feelings of happiness in response to high-valence music (Zatorre, 2015).

STUDY DESIGN Experiment Design

We recruited 24 participants from Zhejiang University for the experiments, aged between 19 and 24 years old, with an equal number of male and female participants to avoid gender bias. They were at the education level of col- lege or above and with computer literacy. All participants reported good vision and hearing abilities and could react emotionally to different types of videos.

In this experiment, videos that evoke 2 types of emotions were used as inde- pendent variables. The level of stress was the dependent variable to explore the relationship between the emotional congruence of dual-channel audio- visual integration and stress relief. The emotions that the videos evoke were joy and sadness. We used the Trier Social Stress Test (TSST) (Kirschbaum et al., 1993) to induce stress in participants. Before and after watching the videos, the changes of stress in participants were examined. The participants were divided into 3 groups in this experiment: Joy Group, Sadness Group and Control Group. The Control Group measures the changes in pressure with no disturbance other than the time.

The Effects of Happy and Sad Dynamic Digital Art on Relieving Stress 799

Stimuli We chose 2 pieces of music which could evoke 2 types of emotions (see Table 1) respectively and produced a 2-minute clip for each piece of music. Before the formal experiment, 15 participants were randomly sele- cted to evaluate the 2-minute clips. The subjective self-rating questionnaire scaled 4 emotions, namely joy, anger, sadness and fear, with a 6-point rating scale. The higher the score, the stronger the feeling and the emotion. The results showed that the mean of the emotional valence scores of each clip was greater than 5.

The experiment applied the open data set of abstract artworks established in 2010 by Machajdik and Hanbury (Machajdik and Hanbury, 2010), inclu- ding 228 pieces of abstract artworks that contain only colour and texture, with no identified objects to evoke emotions without simulating any speci- fic objects. 280 images were rated by approximately 230 people, where each image was rated about 14 times. For each image the category with the most votes was selected as the ground truth. In this article, 2 images were chosen from the data set to represent emotions (joy and sadness). See Table 2 for votes of emotions, and Figure 1 for 2 images.

The production process was captured in 2 clips. The style transfer proces- ses were visually recorded and edited so that each image creation lasted for 2-minute in motion. ALab and SmartPainting were adopted. These 2 softw- are tools were developed by the Engineering Research Centre of Computer Aided Product Innovation Design, the Ministry of Education, China. They were used to carry out algorithm techniques to realize image style transfer

Table 1. 2 pieces of music with 2 types of emotions.

Table available in the original PDF paper.

Emotion Joy Sadness Music Beautiful Sunday Zigeunerweisen

Table 2. Ground truth of 2 chosen abstract images.

Table available in the original PDF paper.

Code Emotion Amusement Awe Disgust Fear Sadness abstract_0020.jpg Joy 9 6 0 2 1 abstract_0007.jpg Sadness 0 1 5 4 9

Figure 1: 2 images of joy and sadness.
Original PDF Figure 1:

methods meanwhile showcase the process of image manipulation with bru- shwork. At the same time, we would add congruent music into the image transfer process. Each clip was 2-minute long.

Data Collection TSST is a validated protocol used to induce stress in a study that has been widely used in psychology research. Experiments were carried out by follo- wing the 3 main phases of stress described in the original TSST protocol: a relaxation period (20 minutes), stress-inducing tasks (speech and arithmetic), and post-stress recovery. We chose speech as our task to induce stress.

Spielberger and colleagues asked participants to self-report their anxiety levels using the State-Trait Anxiety Inventory (STAI) to identify the occurre- nce of stress in their experiment. In our study, we collected self-report data using STAI to validate that we were indeed inducing stress in our participants throughout the experiment.

We used HRV to verify the presence of stress in participants and follo- wed the process described by McDuff and colleagues (McDuff et al. 2016). To generate HRV data from biological signals collected by the Empatica E4 sensors, we implemented the following generic HRV analysis method. First, we interpolated and resampled inter-beat interval (IBI) data at 4Hz to align the signals within an equal time interval. Then we obtained an HRV power spectrum in time series from the detrended interpolated IBI data, by applying the Fast Fourier Transform (FFT) algorithm with a 512-sized slide window (i.e., a 512-sample or 128-second segment of a signal). We calculated the HF (High-Frequency) powers of the HRV as the summation of the discrete points corresponding to the power spectrum under 0.15-0.40Hz. As HF HRV was previously shown to be a reliable indicator of stress occurrence (Berntson and Cacioppo, 2004), we used it for further validation of stress occurrence in our participants.

We experimented at Zhejiang University’s lab, with an experimental setup consisting of 2 adjacent rooms. One room was used for desensitization, it is for the participants to rest and relax, while the other was used for the stress-inducing tasks. We welcomed our participants upon arrival in the desensitization room.

Experiment Process The participants were divided evenly into 3 groups, 8 people in each group, 4 males and 4 females in each group, one of which was the control group, and the other 2 groups were the experimental groups. Each experimental group took 1 type of music as the experiment material.

The experiment included 4 steps and lasted 35 minutes for each partici- pant. Before the experiment, we asked participants to sign a consent form agreeing to participate in the study and collected their demographic data (age, gender, background, dominant hand). Then the participants were fitted with Empatica E4 wristbands and instructed to wear it on the non-dominated hand throughout the experiment.

The Effects of Happy and Sad Dynamic Digital Art on Relieving Stress 801

Step1: Rest and fill out the STAI form. Participants were left alone for 20 minutes to rest and stabilize their emotions. After that, the participants completed the STAI form to measure their perceived anxiety state.

Step2: Induce stress and fill out STAI form. We guided our participants to the experimental room. Participants were given 5 minutes to prepare a 5-minute speech for their dream job. To make them more nervous, they were told their performance was video-recorded and reviewed by judges. And they were then followed by completing the STAI form again.

Step3: Watch videos and fill out the STAI form. We played 2 types of videos for the 2 experimental groups and asked the participants to fill out the STAI form. The control group did not watch the videos but was asked to fill out the STAI form 2 minutes after their speech which was recorded.

Step4: Recovery. After the experiment, we taught the participants to per- form breathing exercise for relaxation to eliminate their negative emotions brought about by the experiment.

Data Analysis We examined participants’ self-reported anxiety values measured by the collected STAI forms when they completed Step1 and Step2. A one-way repeated measures ANOVA yielded a statistically significant effect of stress on self-reported anxiety values (p < 0.01). Post-hoc comparisons using the Tukey HSD test (with Bonferroni corrections) indicated that participants were significantly more anxious (p < 0.01) during the Stress Induction period (M = 37.08, SD = 11.77) of the experiment compared to the baseline values (M = 30.71, SD = 6.37). The results showed that the speech test of TSST in the experiment could effectively induce stress in the participants. See Figure 2.

We applied a one-way repeated measures ANOVA, which yielded a stati- stically significant effect of stress on HRV values (F (2, 24) = 6.54, p < 0.01). A Tukey HSD post-hoc comparison test (with Bonferroni corrections) reveals that HRV values during Stress Induction and Post-stress Recovery periods are significantly lower (p < 0.01) compared to HRV values during the baseline measurements. Since low values of HF HRV indicate a physiological prese- nce of stress (Lazarus, 1999), we can conclude from the HRV results that the participants were stressed after undergoing the speech tasks. The mean values of HF HRV are presented in Figure 3.

Figure 2: Self-reported anxiety mean values.
Original PDF Figure 2:
Figure 3: HRV(HF) mean values.
Original PDF Figure 3:

The Joy Group and Sadness Group participants reported that they felt significantly less anxious (p < 0.01) after watching the related videos compa- red to the Stress Induction period. We selected an equal number of men and women for the Joy Group and Sadness Group for an experiment on effective pressure relief. It was analysed that gender differences exist in anxiety relief with the changes in anxiety values of the selected participants before and after watching the videos. It showed that the Sadness Group had a gender difference in relieving stress (p < 0.01), and the effect on women was greater than that on men.

DISCUSSION AND FUTURE WORK The Effects of Videos of Different Emotions on Relieving Stress

Regardless of appreciating art or creating art, it can bring experiences of happiness and resonance. Although these experiences are short, they pro- foundly touch the soul. So we use digital art as stimuli to find out if it could relieve stress by combining audio and visual channels. That might be a new way to cope with daily stress by exploring the psychology and medical fields.

The digital art video of this study contained audio-visual information with congruent emotion, which enhanced the transmission of emotion and ena- bled the participants better interact and empathize. Joyful music and image made people feel positive, and this kind of emotion could effectively relieve anxiety. Some studies have shown that concerts with joyful emotions lead to increased dopamine secretion in the body, and dopamine is an essential neu- rotransmitter in the brain reward system (Sutoo and Akiyama, 2004). The- refore, it will produce happy emotions, thus alleviating the current anxiety state.

Watching sad emotional videos is generally believed to be more stressful under tense conditions. However, some studies have found that sad emotional videos could induce happiness. When participants were watching sad emoti- onal videos, they would go through a series of complex emotions, including sadness, romance, happiness, and other emotional experiences. Mori and Iwanaga believed that a happy experience induced by listening to sad music might be related to the secretion of prolactin in the individual (Mori and Iwanaga, 2014). Prolactin is a peptide hormone that could trigger feelings of

The Effects of Happy and Sad Dynamic Digital Art on Relieving Stress 803

calm and relaxation. So when an individual felt sad, prolactin might be able to bring a happy experience, which that might be part of the reasons to alle- viate the current anxiety state. The brain perceives music signals through the auditory pathway. Familiar music will involve the brain’s memory system, while an unknown melody, it will activate the right area of the hippocam- pus (Knight and Rickard, 2001). Another explanation for the positive effect of music interventions on psychological stress-related outcomes may be that listening to music can provide “distraction” from thoughts or feelings that cause stress (Chanda and Levitin, 2013). Indeed, the benefits of music to distract people from aversive states has been supported by short-term music interventions for healing acute stress disorder (Linnemann et al. 2015).

The Gender Difference There was a significant difference between men and women in relieving stress when they were watching videos with the same emotion, which indicated that the subjective perceptions of men and women in watching videos were dissimilar. Some studies (Verma et al. 2011) have found that women have more intense and frequent fluctuations in emotion than men have, and even when they have the same sentiment, men and women might have different feelings.

Limitations and Future Work The current study has some limitations that need to be mentioned. Firstly, we need to recruit more participants of different ages and occupations and add more music and images as stimuli to increase the validity and credibility of the experiment. Secondly, future research should focus on the specific cha- racteristics of the interventions on stress reduction, for example, the music tempo, music genre, the use of live music or pre-recorded music, music sele- ction, or the frequency of the music intervention sessions. What’s more, we should consider conducting comparative experiment of audio-visual infor- mation with incongruent emotions and to discover what will happen to the impact on the anxiety relief. Lastly, we will add physiological data collection in future research, e.g., skin conductance measurement, EEG etc., to coordi- nate with STAI form and HRV measurement for a comprehensive observation of the anxiety data.

CONCLUSION

This study showed that watching videos with congruent emotions of joy and sadness could significantly relieve acute anxiety. Despite the limitations, this study has important implications for future research and practical use of inte- rventions in relieving stress. Many people suffer from stress-related symptoms in their daily life and some settings (e.g., medical settings, mental health care settings, work-related settings). Since digital art intervention is accessible and affordable, it could be integrated into daily life and medical treatments, it is essential to recognize their effects.

REFERENCES

Bandelow, B., Reitt, M., Röver, C., Michaelis, S., Görlich, Y., & Wedekind, D. (2015).

Efficacy of treatments for anxiety disorders: a meta-analysis. International clinical psychopharmacology, 30(4), 183–192. Berntson, G. G., & Cacioppo, J. T. (2004). Heart rate variability: Stress and psychiatric conditions. Dynamic electrocardiography, 57–64. Birren, F. (2016). Color psychology and color therapy; a factual study of the influence

of color on human life. Pickle Partners Publishing. Bower, G. H. (1981). Mood and memory. American psychologist, 36(2), 129. Case, C., & Dalley, T. (2014). The handbook of art therapy. Routledge. Chanda, M., & Levitin, D. J. (2013). The neurochemistry of music trends in cognitive sciences. Demir, Ü. (2020). Investigation of color-emotion associations of the university

students. Color Research & Application, 45(5), 871–884. Hagood, M. M. (1990). Art therapy research in England: Impressions of an American art therapist. The arts in psychotherapy. Hargreaves, D. J., MacDonald, R., & Miell, D. (2005). How do people communicate

using music. Musical communication, 1, 1–26. Jiang, J., Rickson, D., & Jiang, C. (2016). The mechanism of music for reducing psychological stress: Music preference as a mediator. The Arts in Psychotherapy, 48, 62–68. Kirschbaum, C., Pirke, K. M., & Hellhammer, D. H. (1993). The ‘Trier Social Stress

Test’–a tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology, 28(1-2), 76–81. Knight, W. E., & Rickard, N. S. (2001). Relaxing music prevents stress-induced increases in subjective anxiety, systolic blood pressure, and heart rate in healthy males and females. Journal of music therapy, 38(4), 254–272. Kurt, S., & Osueke, K. K. (2014). The effects of color on the moods of college

students. sage Open, 4(1), 2158244014525423. Lange, E. B., Fünderich, J., & Grimm, H. (2022). Multisensory integration of musical emotion perception in singing. Psychological Research, 1–16. Lazarus, R. S. (1999). Stress and emotion. New York, NY 10036. Linnemann, A., Ditzen, B., Strahler, J., Doerr, J. M., & Nater, U. M. (2015). Music

listening as a means of stress reduction in daily life. Psychoneuroendocrinology, 60, 82–90. Machajdik, J., & Hanbury, A. (2010, October). Affective image classification using features inspired by psychology and art theory. In Proceedings of the 18th ACM international conference on Multimedia (pp. 83–92). Martin, L., Oepen, R., Bauer, K., Nottensteiner, A., Mergheim, K., Gruber, H.,

& Koch, S. C. (2018). Creative arts interventions for stress management and prevention—a systematic review. Behavioral Sciences, 8(2), 28. McDuff, D. J., Hernandez, J., Gontarek, S., & Picard, R. W. (2016, May). Cogcam: Contact-free measurement of cognitive stress during computer tasks with a digi- tal camera. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 4000–4004). Misselhorn, J., Friese, U., & Engel, A. K. (2019). Frontal and parietal alpha oscilla-

tions reflect attentional modulation of cross-modal matching. Scientific reports, 9(1), 5030. Mori, K., & Iwanaga, M. (2014). Pleasure generated by sadness: Effect of sad lyrics on the emotions induced by happy music. Psychology of Music, 42(5), 643–652. Roswiyani, R., Kwakkenbos, L., Spijker, J., & Witteman, C. L. (2019). The effe-

ctiveness of combining visual art activities and physical exercise for older adults

The Effects of Happy and Sad Dynamic Digital Art on Relieving Stress 805

on well-being or quality of life and mood: a scoping review. Journal of Applied Gerontology, 38(12), 1784–1804. Sayette, M. A. (1999). Does drinking reduce stress?. Alcohol Research & Health, 23(4), 250. Setz, C., Arnrich, B., Schumm, J., La Marca, R., Tröster, G., & Ehlert, U. (2009).

Discriminating stress from cognitive load using a wearable EDA device. IEEE Transactions on information technology in biomedicine, 14(2), 410–417. Spielberger, C. D. (1983). State-trait anxiety inventory for adults. Sun, D., Paredes, P., & Canny, J. (2014, April). MouStress: detecting stress from mouse motion. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp.61–70). Sutoo, D. E., & Akiyama, K. (2004). Music improves dopaminergic neurotransmis-

sion: demonstration based on the effect of music on blood pressure regulation. Brain research, 1016(2), 255–262. Thompson, W. F. (2015). Music, thought, and feeling: Understanding the psychology of music. Oxford university press. Verma, R., Balhara, Y. P. S., & Gupta, C. S. (2011). Gender differences in

stress response: Role of developmental and biological determinants. Industrial psychiatry journal, 20(1), 4. Zatorre, R. J. (2015). Musical pleasure and reward: mechanisms and dysfunction.

Annals of the New York Academy of Sciences, 1337(1), 202–211.

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET 2023): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET 2023): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-87-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →