Dawoon Jeong, Sung H. Han, Kimin Kwon, Wan Sun Shin (2022). Emotional and Physiological Responses to the Roll Motion Effect in 4D Movies. In: Affective and Pleasurable Design. AHFE Open Access, vol. 41, pp. 247–252. https://doi.org/10.54941/ahfe1001804
4D effects have been introduced to increase movie audience immersion. Nevertheless, it is difficult to find research studies that investigated systematically the audience’s responses to the motion effects that account for a large portion of the 4D effects. This study investigated how the motion effect of rolling an audience chair affects emotions and physiological responses. The participants, with attached sensors, experienced movie clips with or without the motion effect and responded to questionnaires about their emotions and immersion. The results showed that the motion effect amplified arousal and immersion. Changes in the blood volume pulse and skin conductance level due to motion effects were also found. It was concluded that the roll motion effect can be effectively used to arouse and immerse the audience. In addition, significant correlations were found between emotional and physiological responses.
The movie industry has begun producing 4D movies as interest in immersive content has increased. 4D movies provide 4D effects, such as motion effects (chair movements), vibrations, lights, air shots, and water sprays, along with audiovisual content. Motion effects are the most frequently used 4D effects (Lee et al., 2016).
The current methods for applying motion effects to audiovisual content are not completely effective. Most motion effects have been created based on the empirical knowledge of motion effects designers. Current 4D movies provide the audience with some satisfaction. This can be inferred from the fact that sales of 4D cinema increased, while total cinema sales decreased due to COVID-19 (Korean Film Council, 2021). However, the audience’s evaluation of the motion effects is both positive and negative. Therefore, it is necessary to find a way to reduce the negative effects and enhance the positive effects to maximize the effectiveness of motion effects.
The effectiveness of motion effects can be investigated from various per- spectives. The audiences’ emotional responses and immersion are some of these aspects. Positive effects are created by the designer, using motion effects
to make the audience feel immersed and elicit the emotions intended by the director of the movie. Nevertheless, it is difficult to find studies that systema- tically investigate the audience’s immersion and emotional responses to these motion effects.
There are subjective and objective measures of emotional responses and immersion. One of the subjective measurements involves collecting responses through questionnaires. Objective measurements include methods to mea- sure physiological signals, such as the electrocardiogram, electromyograph (EMG), electroencephalogram, and skin conductance (SC). Objective measu- rement has the advantage of collecting in real time and inferring emotional responses, although it is difficult to know the emotional response directly.
The objective of this study was to investigate how motion effects in 4D movies affects emotional responses and immersion. For this purpose, questionnaires were used to collect the subjective emotional responses and immersion resulting from motion effects. We confirmed whether the phy- siological responses were affected by motion effects. By identifying the correlation between the subjective responses and physiological responses, the possibility of inferring emotional responses and immersion through objective measurements was confirmed.
METHODS
This study focused on the motion effects that rotate once along the roll axis (the roll motion effect), among various motion effects. The roll motion effect is a movement that frequently appears in 4D movies (Jeong et al., 2021). We made 4D movie clips to elicit emotions. Four scenes that provided the roll motion effect in 4D movies (How to Train Your Dragon: The Hidden World, Harry Potter and the Chamber of Secrets, and Avengers: Endgame) were extracted, and movie clips of approximately 1 min were produced. The roll motion effect rotates up to 8° in the left or right direction for 1 s (Figure 1). The displacement of the roll motion effect followed a raised cosine waveform.
The participants subjectively evaluated valence, arousal, and immersion after watching the movie clip with (experimental condition) or without (control condition) the roll motion effect. A 101-point scale was used, with a score of 0 indicating very unpleasant, very calm, or no immersion at all. A score of 100 indicates very pleasant, very excited, or fully immersed.
For the objective measurements, blood volume pulse (BVP) measured with a photoplethysmography sensor, SC level (SCL) measured with a galvanic skin response (GSR) sensor, and EMG of corrugator muscle activity and zygomaticus major muscle activity obtained with facial EMG sensors were collected. Procomp InfinitiTM was used to collect the physiological signals. The PPG and GSR sensors were attached to fingers that the participants did not mainly use.
Thirty-two participants in their 20s and 30s were recruited (average age: 23.94 years, S.D.: 3.72). The study included 21 men and 11 women. They had no mental or physical disabilities or cardiovascular diseases.
The participants sat on a chair-shaped motion simulator (iMP6-M150, INNOSUMULATION) and experienced the movie clips and roll motion
Emotional and Physiological Responses to the Roll Motion Effect in 4D Movies 249
Figure 1: (Left) Neutral state of the motion simulator (0°) (Right) The state rotated 10° to the left of the roll axis.Figure 2: Experimental procedure.
effects. The movie clips were played on a screen at 2.45 m from the motion simulator. The sound was played using two speakers (Yamaha NS-F150) and a subwoofer (Yamaha NS-SW050).
The experimental procedure is illustrated in Figure 2. The participants experienced a movie clip with or without the roll motion effect. They then responded to questionnaires about their emotional state and immersion. The emotional state was defined as the interval from the onset of the roll motion effect to the beginning of the neutral position (washback). Upon com- pletion of the questionnaires, the participants experienced the next movie clip after resting for 90 s. The last 30 s of rest were set as the baseline state. The movie clips were counterbalanced using a balanced Latin square design.
The collected physiological signals were preprocessed into change scores and analyzed. Change score was calculated as mean physiological response in emotional state over differences in mean physiological response in emo- tional state and baseline state. Then we calculated average change score for four clips. Paired t-tests were conducted on the average change scores, and a correlation analysis (CA) between subjective responses and average change scores was conducted.
Table 1. Comparison of the emotional responses, immersion, and change scores with and without the roll motion effect confirmed by paired t-tests.
Table 1 shows the results of the paired t-tests performed to compare the responses with and without the roll motion effect. The paired t-tests reve- aled that there were significant effects of the roll motion effect on arousal [t = −1.53, p = 0.136] and immersion [t = −1.61, p = 0.117] at p<.2. However, there was no statistically significant difference in the valence [t = −0.63, p = 0.535]. Arousal and immersion were higher with the roll motion effect (arousal: 60.16, immersion: 95.70) than without it (arousal: 53.54, immersion: 93.28). There were statistically significant differences in change scores of BVP (t = 2.41, p = 0.022) and SCL (t = −2.12, p = 0.042) at p<.05. There was no statistically significant difference in the scores of facial EMG features at p<.05.
The CA revealed significant correlations between the scores of physiologi- cal responses and subjective responses (Table 2). However, there was no sta- tistically significant correlation between these change scores of physiological responses and immersion.
DISCUSSION
According to the paired t-tests results, the roll motion effect is effective in enhancing arousal and immersion. We suggest using the roll motion effect in scenes in which the director or designer wants the audience to be aroused and immersed. If they want the audience to remain in a calm emotional state, it is better to avoid the roll motion effect. However, because the roll motion effect has no significant effect on the amplification of the valence, it is better to use another 4D effect to amplify the valence. For example, the motion effect rotating along the pitch axis (Jeong et al., 2021) and the hot-air effect
Emotional and Physiological Responses to the Roll Motion Effect in 4D Movies 251
(Raheel et al., 2021) can amplify the valence depending on the scene. This result also indicates that the roll motion effect does not interfere with the valence.
The roll motion effect promotes vasoconstriction, and this response may occur when the sympathetic nervous system is stimulated or when stress, anxiety, or pain is felt (Jang et al., 2015). Therefore, the change score of BVP decreased when there was a roll motion effect. In addition, the change score of SCLs increased when there was a roll motion effect. Although not revealed in this study, emotional responses related to activation of the sym- pathetic nervous system, such as surprise, have the potential to be amplified by the roll motion effect (Jang et al., 2015).
BVP and SCL were more effective than EMG in confirming the psychophy- siological responses of the roll motion effect. This is because in the change scores of EMG, a significant effect of the roll motion effect could not be confirmed. It can be assumed that this is because the roll motion effect did not affect valence but affected arousal. Facial EMG of corrugator muscle activity and zygomaticus major muscle activity is widely used as a psych- ophysiological indicator of emotional processing valence (Potter & Bolls, 2012). It is expected that emotional responses to the roll motion effect can be inferred by collecting physiological signals. This is because weak but signifi- cant correlations were found between physiological features and subjective emotional responses. However, because it has weak correlations, it is expe- cted that emotional responses can be inferred with higher accuracy using machine learning or deep learning techniques rather than simple linear regres- sion. A more sophisticated recognition model can be created by collecting more signals or by utilizing features related to arousal.
This study had some limitations. Because only four movie clips were used for the experiment, it was necessary to check the consistency of the results with more movie clips. The results of this study may be limited to the roll motion effect, which rotates by 8° along the raised cosine waveform for 1 s. This study can be expanded by varying the amplitude or waveform. In addition, it is necessary to investigate other age groups, because the age range of the participants was limited to those in their 20s and 30s.
CONCLUSION
This study determined the effects of the roll motion effect used in 4D movies on emotional responses and immersion through questionnaires and the col- lection of physiological signals. The roll motion effect is effective in arousing and immersing the audience. It was also found that the roll motion effect decreased BVP and increased SCL. Therefore, it is thought that physiological features related to BVP and SCL are suitable for recognizing psychophysio- logical responses of the roll motion effect. This study can provide the basis for designing motion effects that enhance emotional effects and immersion. In addition, the emotional responses and physiological signals collected in this study can be used to create a model for recognizing the emotions of 4D movie audiences.
ACKNOWLEDGMENT
This work was supported by Samsung Research Funding & Incubation Cen- ter of Samsung Electronics [project number SRF-IT1802-05] and the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Techno- logy Research Center) support program (IITP-2018-0-01441) supervised by the IITP(Institute for Information & Communications Technology Planning & Evaluation)
REFERENCES
Jang, Eun-Hye. Park, Byoung-Jun. Park, Mi-Sook. Kim, Sang-Hyeob and Sohn, Jin- Hun. (2015). Analysis of physiological signals for recognition of boredom, pain, and surprise emotions. Journal of physiological anthropology Volume 32 No. 25. Jeong, Dawoon. Han, Sung H. Jeong, Dong Yeong. Kwon, Kimin and Choi, Seun-
gmoon. (2021). Investigating 4D movie audiences’ emotional responses to motion effects and empathy. Computers in Human Behavior Volume 121 No. 106797. Korean Film Council (KOFIC), Republic of Korea. (July 22, 2021) Settlement of Korean Film Industry in the First Half of 2021. KOFIC Website: https://www.ko fic.or.kr/kofic/business/board/selectBoardDetail.do?boardNumber=2 Lee, Jaebong. Han, Bohyung and Choi, Seungmoon. (2015). Motion effects synthesis
for 4D films, IEEE transactions on visualization and computer graphics Volume 22 Issue 10. Potter, Robert F., Bolls, Paul D. (2012). Psychophysiological measurement and meaning: Cognitive and emotional processing of media. New York and London: Routledge. Raheel, Aasim. Majid, Muhammad and Anwar, Syed Muhammad. (2021). Dear-
mulsemedia: Dataset for emotion analysis and recognition in response to multiple sensorial media. Information Fusion Volume 65.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Affective and Pleasurable Design
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-17-9
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional