Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Effects of a Person’s Facial Expressions in Video on Viewers’ Moods and Facial Expressions: Application to Interface Design

Meisei University, Tokyo, FL 1918506, Japan ↗
Publication year
Digital Object Identifier10.54941/ahfe1006864

Abstract

The number of video-sharing sites, such as YouTube, in which it is easy to post and view moving images, continues to increase. This may result in viewers unconsciously experiencing psychological effects from the videos. In this research, we aim to clarify the effects of speakers’ facial expressions in videos on viewers. Our study investigated mood ratings and changes in viewers’ facial expressions while watching different facial expressions of a real human speaker in videos. The results showed ‘Happy Feelings’ and ‘Positivity’ to be rated significantly more highly after viewing moving images in which a speaker smiled while they were talking. The results of this study reveal the potential for making and posting videos in the light of the psychological effects on viewers, and for labelling the videos to enable the viewers to select and watch such videos. It is useful to make and post videos in which smiling persons talk, aiming to make the viewers happy and positive, and for viewers to select and watch such videos when they wish to feel happy and positive. Furthermore, based on the results of our study, we created online learning materials for children to help them enjoy learning English conversation. In the online learning materials, when a user selects the correct answer, the facial expression of the character changes to a smile. This visual feedback is expected not only to inform learners that their answer was correct but also to improve their mood and promote a more positive learning experience.

INTRODUCTION

The number of video-sharing sites, such as YouTube, in which it is easy to post and view moving images, continues to increase. Because users can watch videos on their smartphones when not otherwise occupied, e.g., while on public transport, a growing number of people seek out videos for diversion and to pass time. This may result in viewers unconsciously experiencing psychological effects from the videos. The present study focuses on the facial expressions of persons in videos and investigates their effects on viewers.

Prior to our investigation, we examined previous studies on the effects of human facial expressions on other people, their impressions, their involvement with and reactions to them. Previous research has shown that people have specific facial expressions when being presented pictures of a

© 2025. Published by AHFE Open Access. All rights reserved. 522

Effects of a Person’s Facial Expressions in Video 523

happy face or an angry face (Dimberg, 1982). The subjects in the previous study adopted different facial expressions when they saw happy smiling faces and when they saw angry faces. Another report shows that when people see another person’s smiling face, they react to it and smile in response (Ichikawa & Makino, 2001). In this field of research, the Facial Feedback Hypothesis was proposed in the 19th century; it suggests that individuals’ emotional state of mind is influenced by their own facial expressions (Tomkins, 1962). It has been shown that different emotions are evoked by adopting different facial expressions such as a smiling or angry face.

Based on these previous studies, it appears that facial expressions of viewers of videos are formed unconsciously in response to those of real persons in videos, resulting in emotions that are similar to the observed facial expressions. In this study, in order to clarify the effects of a person’s facial expressions in the videos on viewers, we investigated viewers’ mood ratings and changes in facial expressions while viewing different real facial expressions in videos, focusing on smiling expressions.

EXPERIMENTAL METHOD Ten university students (six males and four females) participated in the experiment. They watched three experimental videos in random order, in which a minute-long weather forecast was read by a real person with a particular facial expression throughout: a straight face (expressionless), a Non-Duchenne Smile, or a Duchenne Smile. In this study, based on previous research on smiling and facial muscles, a Duchenne Smile is a genuine smile that appears naturally, whereas a Non-Duchenne Smile is an intentionally created smile (Duchenne, 1990; Ekman & Friesen, 1982). The sound in the video was muted to eliminate factors other than the effects of facial expression. Our intention was to exclude the effects of any vocal stimuli on viewers’ impression ratings, since the tone of voice could be linked to the different types of facial expressions. The experiment was conducted over Zoom, employing the experimental process shown in Figure 1. A reset image (Fig. 1: R) was displayed for one minute between each of the three experimental videos (Fig. 2). The subjects rated their mood before and after viewing each video (Fig. 1: Ev). The descriptive terms used in the ratings were ‘Happy Feelings’, ‘Pleasure’ and ‘Positivity,’ which are all related to smiling facial expressions. The level of ‘Happy Feelings’ was rated on a seven- point scale: ‘Very happy,’ ‘Happy,’ ’Somewhat happy,’ ‘Neutral,’ ‘Somewhat bored,’ ‘Bored’ and ‘Very bored’; the level of ‘Pleasure’ was rated as ‘Very pleased,’ ‘Pleased,’ ’Somewhat pleased,’ ‘Neutral,’ ’Somewhat sad,’ ‘Sad’ and ‘Very sad’; and the level of ‘Positivity’ was rated as ‘Very positive,’ ‘Positive,’ ’Somewhat positive,’ ‘Neutral,’ ’Somewhat disappointed,’ ‘Disappointed’ and ‘Very disappointed.’ The seven stages were categorized into seven grades (7 – 1 points).

524 Yoshioka

Figure 1: Experimental process.
Original PDF Figure 1:
Figure 2: Reset image.
Original PDF Figure 2:

Prior to creating the experimental videos, we examined previous studies on Duchenne Smiles and Non-Duchenne Smiles (Sugahara, 2014). In these studies, three facial geometric characteristics were listed: the aspect ratio of a rectangle superimposed on a smiling human face, angle of palpebral fissure, and inclination of the lower lip with the mouth open. The aspect ratio of a rectangle superimposed on a smiling human face describes the ratio of width to height of a virtual rectangle which covers the area formed between the corners of the eyes and the corners of the mouth. The rectangle formed by connecting the four points, that consist of the corners of the eyes and the corners of the mouth, comprises the aspect ratio. The angle of palpebral fissure describes the degree of openness of the eyelids. The inclination of the lower lip with the mouth open describes the angle of the lower lip. When making a Duchenne Smile, the corners of the mouth are raised, describing an inverted parabolic curve. Of these geometric characteristics, a Non-Duchenne Smile activates only the muscles around the mouth, but a Duchenne Smile activates not only the muscles around the mouth, but narrows the eyelids’ aperture to form a crescent shape. Based on this information, three types of experimental videos were created, in which a real person talks with a Straight Face, Non-Duchenne Smile or Duchenne Smile (Fig. 3).

Effects of a Person’s Facial Expressions in Video 525

Figure 3: Experimental video (opening part).
Original PDF Figure 3:

RESULTS

Mood Ratings Before and After Viewing the Videos Ratings for ‘Happy Feelings’, ‘Pleasure’ and ‘Positivity’ before and after viewing the videos were analyzed using a t-test (significance level of 5%, two-sided test and paired t-test).

In the ratings for ‘Happy Feelings’, there was no significant difference between before and after viewing the person in the video talking with a Straight Face and Non-Duchenne Smile, whereas there was a significant difference (t (9) = 3.0, p <.05) between before and after viewing the person in the video talking with a Duchenne Smile (Fig. 4). In the ratings for ‘Positivity’, there was also no significant difference between before and after viewing the person talking with a Straight Face and a Non-Duchenne Smile, whereas there was a significant difference (t (9) = 2.91, p <.05) between before and after viewing the person talking with a Duchenne Smile (Fig. 5). In the ratings for ‘Pleasure’, again there was no significant difference between before and after viewing the person talking with a Straight Face and Non-Duchenne Smile, whereas there was a marginally significant difference (t (9) = 1.92, p <.086) between before and after viewing the person talking with a Duchenne Smile (Fig. 6). These results showed that the viewers gave significantly higher ratings for ‘Happy Feelings’ and ‘Positivity’ while watching the person talking with a Duchenne Smile. Their mood changed, making them feel happy and positive. There was no interaction between the difference in facial expression (Straight Face, Non-Duchenne Smile and Duchenne Smile) of the persons and ratings for ‘Happy Feelings’, ‘Pleasure’ and ‘Positivity’ between before and after viewing the videos.

Figure 4: Ratings of ‘Happy Feelings’ before and after viewing the videos.
Original PDF Figure 4:

526 Yoshioka

Figure 5: Ratings of ‘Positivity’ before and after viewing the videos.
Original PDF Figure 5:
Figure 6: Ratings of ‘Pleasure’ before and after viewing the videos.
Original PDF Figure 6:

Changes in Facial Expression While Watching a Video To analyze changes in facial expressions while watching the videos, we recorded the subjects’ facial expressions using the camera function in Zoom. Based on previous studies of the Duchenne Smile and Non-Duchenne Smile [Note 3], changes in the cheeks, corners of the mouth and the eyes were observed by visual detection. In the experiment, changes in facial expression such as lifted cheeks, raised corners of the mouth and narrowed eyes due to a narrowed palpebral fissure were observed in half of the subjects while they watched the person talking with a Duchenne Smile (Table 1). The results showed that a mirroring reaction occurred on the part of the subjects, supporting these previous studies (Ichikawa & Makino, 2001) which reported that individual facial expressions changed due to mirroring of the other person’s smile.

Effects of a Person’s Facial Expressions in Video 527

Table 1: Changes in facial expression while watching a person in videos with different facial expressions.

Table available in the original PDF paper.

Straight Face Non-Duchenne Smile Duchenne Smile Eyes Subject Cheeks Corners of the Mouth Eyes Subject Cheeks Corners of the Mouth Eyes Subject Cheeks Corners of the Mouth

1 – – – 1 – – – 1 – – – 2 – – – 2 – – – 2 + + + 3 – – – 3 – – – 3 – – – 4 – + – 4 – – – 4 + + + 5 – – – 5 – – – 5 – – – 6 – – – 6 – – – 6 + + + 7 – – – 7 – + + 7 + + – 8 – – – 8 – – – 8 – – – 9 – – – 9 – – – 9 – – – 10 – – – 10 – – – 10 + + – −not changed, + changed

DISCUSSION

Aiming to clarify the effects of facial expressions in videos on viewers, this study investigated viewers’ mood ratings and changes in facial expressions while watching different facial expressions in a video, specifically smiling facial expressions. The viewers gave significantly higher ratings for ‘Happy Feelings’ and ‘Positivity’ while watching Duchenne Smiles in the video. Their mood changed, making them feel happy and positive. Changes in facial expression were observed in half of the subjects, such as lifted cheeks, raised corners of the mouth and narrowing of the eyes, which made a smiling face. This can be regarded as mirroring the smiling face of the person in the video. These results support the Facial Feedback Hypothesis, which suggests that individuals’ emotional states are influenced by their facial expressions. The subjects in this study observed Duchenne Smiles in the video, reacted to them by mirroring, and smiled, making them feel happy and positive.

The results of this study reveal the potential for making and posting videos in the light of the psychological effects on their viewers, and for labeling them to enable the viewers to select and watch such videos. It is useful to make and post videos in which smiling persons talk, aiming to make the viewers happy and positive, and for viewers to select and watch such videos when they wish to feel happy and positive.

DESIGN PROPOSAL Design Background

Based on the results of our study on the effects of facial expressions in the videos on the viewers, we developed online learning materials for children to help them enjoy learning English conversation.

In current online materials used in English conversation classes, a checkmark [✓] is displayed when the answer is correct, and a cross [×] appears when the answer is wrong. In Japan, however, because a circle [⃝] is very commonly used to indicate a correct answer, Japanese people tend not to intuitively recognize a checkmark as indicating a correct answer. In view of this point, we created a program in which, in addition to the circle mark, the

528 Yoshioka

character’s facial expression is changed to a smile when an answer is correct (Fig. 7).

In our previous studies, we investigated smiling face icons that were composed of simple-shaped eyes and mouths using lines or dots. In one of these studies, we investigated how subjects’ mood changed when they drew smiling face icons that consisted only of eyes and a mouth using a short version of the Profile of Mood States (POMS-SF). In the POMS- SF, which used 30 evaluation items, T-scores by sex and age class were obtained from crude scores on the following six scales: Tension-Anxiety (T- A), Depression-Dejection (D), Anger – Hostility (A-H), Vigor (V), Fatigue (F), and Confusion (C), and analysis of variance (ANOVA) was performed. The results showed that the act of drawing smile icons was found to decrease the subjects’ ‘Tension - Anxiety,’ ‘Depression - Dejection,’ ‘Anger - Hostility,’ and ‘Fatigue’ scale scores on the short version of the POMS- SF and to improve their mood. In addition, we confirmed that the subjects’ facial expressions had synchronous responses to the smiling face icons (Yoshioka, 2017a). In another of our studies, physiological and psychological evaluations were performed, and changes in participants’ facial expressions were studied during three shuffled tasks with different creativity tasks using a tablet: viewing of face icons; placing of parts to create face icons; and drawing of face icons consisting of eyes and mouth parts. The facial expressions of participants who liked looking at drawings and paintings, who enjoyed scribbling and sketching, and who were familiar with drawing responded synchronously to the smile icons they were drawing; as their facial expressions changed, their moods improved. In addition, in the task of placing eyes and mouth parts to create smile face icons, the variation in mean blood pressure increased and the tasks of excitement increased (Yoshioka, 2017b). These results support previous research that, even when interacting with face icons using simple elements, ‘When a person looks at another person’s smiling facial expressions, this causes a transmission of emotion and makes the person’s face respond synchronically to the other person’s smiling face to form a smile, thus putting the person in a good mood’ (Ichikawa & Makino, 2001). Another previous study on pictographs investigates the relationship between choosing emoticons with facial expressions that are used in emails and chats, and cerebral activity. In the study, while watching emoticons, the right fusiform gyrus, which is related to the recognition of faces, was not significantly activated; however, the right inferior frontal gyrus, which is related to discrimination of emotions, was significantly activated. It was suggested that emoticons are relevant to nonverbal communication (Uasa, et al., 2007). Based on these previous studies, illustrations of smiling animal characters with a simply-shaped eyes and mouth, rather than photos of smiling real human faces, were displayed in the online learning materials developed (Fig. 7).

Design Content These characters in the online learning materials which have simply eyes and mouth are child-friendly design elements that induce smiles in the users. In the

Effects of a Person’s Facial Expressions in Video 529

online learning materials, if the user selects the correct answer, a circle mark is displayed on the screen and the facial expression of the character changes to a smile. This allows the users to understand that the answer they selected is correct, and by seeing the video of a smiling face, their own facial expressions have a synchronous response to the smiling face, which is expected to improve their mood. We will conduct mood ratings and investigation of changes in facial expressions using this program.

Figure 7: Online learning materials using a character with a smiling face.
Original PDF Figure 7:

CONCLUSION

It is easy these days for anyone to post and watch videos online. Viewers may unconsciously experience psychological effects from the videos they watch, so there will be benefits from selecting videos that have a positive effect on them.

Aiming to clarify the effects of persons’ facial expressions in videos of this type on viewers, this study investigated viewers’ mood ratings and changes in facial expressions while watching the different facial expressions of the persons in the videos. The results showed psychological changes, specifically by becoming happier and more positive while watching persons talking with a smiling face. Half of the subjects reacted to the facial expressions of the persons talking with a Duchenne Smile and mirrored the smile. These results support the idea that it is useful to make and post videos in which smiling persons talk, aiming to make the viewers happy and positive, and for viewers to be able to select and watch such videos when they want to feel happy and positive. The results of this study can be used to create an index for making and posting videos, and for users to select material that will positively affect them. Other than facial expressions, content, stories, and voices can also have psychological effects on viewers. We will conduct further investigations by structuring various elements in video form. In addition, in this study we developed online learning materials using smiling faces of animal characters for children to help them enjoy learning English conversation. We will conduct mood ratings and investigation of changes in facial expressions using this program.

ACKNOWLEDGMENT

While this paper is based on graduation research conducted by a student under my supervision, Kouki Satake (4th-year student at Meisei University, 2024), it has been developed through new analyses and evaluations from a different perspective.

530 Yoshioka

REFERENCES

Dimberg, Ulf. (1982). Facial Reactions to Facial Expressions. Psychophysiology, 19, 643–647. https://doi.org/10.1111/j.1469–8986.1982.tb02516.x. Duchenne de Boulogne, G. B. (1990). The mechanism of human facial expression. Cambridge, Cambridge University Press. https://doi.org/10.1017/ CBO9780511752841. Ekman, P., & Friesen, W. V. (1982). Felt, false, and miserable smiles, Journal

of Nonverbal Behavior, 6, 238–252. https://link.springer.com/article/10.1007/ BF00987191. Ichikawa, H., Makino, J. (2001). Congruency of Facial Reactions to Facial Expressions, Institute of Electronics, Information and Communication Engineers (IEICE) Technical Report, 101(333), 9–15. https://doi.org/10.4992/jjpsy.75.142. Sugahara, T. (2014). Analysis of Shape and Facial Muscle Activity of Smiles, Journal of the Visualization Society of Japan, 34(133), 14–19. https://www.jstage.jst.go.jp/article/jvs/34/133/34_14/_pdf/-char/ja. Tomkins, S. S. (1962). The positive affect, Imagery, Consciousness, Vol. 1,

New York, Springer-Verlag. https://books.google.co.jp/books/about/ Affect_Imagery_Consciousness.html?id=WIpgNerqaIkC&redir_esc=y. Uasa, M., Saito, K., Mukawa, N. (2007). Brain Activity Associated with Emoticons: An fMRI Study-Effects of Facial Expressions in Personal Communications over Computer Network-, The transactions of the Institute of Electrical Engineers of Japan. C. 127(11), 1865–1870. https://doi.org/10.1541/ieejeiss.127.1865. Yoshioka, K. (2017). Mood Change Caused by Drawing of Face Icons - Characteristics of the Shapes and Facial Expression in Eyes and Mouth Parts, Bulletin of Japanese Society for the Science of Design, 63(5), 43–48. https:// doi.org/10.11247/jssdj.63.5_43. Yoshioka, K. (2017). Physiological and Psychological Evaluation and Changes in

Facial Expression by Viewing, Placing, or Drawing of Face Icons, Bulletin of Japanese Society for the Science of Design, 64(1), 11–18. https://doi.org/10.11247/ jssdj.64.1_11.

Cover of the proceedings volume Human Factors in Design, Engineering, and Computing

Explore the complete digital volume

Human Factors in Design, Engineering, and Computing

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-75-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →