Multimodal Deep Models for Predicting Affective Responses Evoked by Movies

Ha Thi Phuong Thao, Dorien Herremans, Gemma Roig

Introduction

While significant progress has been made in automated emotion recognition—the detection of facial expressions or vocal cues from a subject—predicting the evoked emotion (the affective response of a viewer to a stimulus) remains a formidable challenge. This task is inherently subjective, as the emotional "vibe" of a cinematic scene is constructed through a complex interplay of music, narrative pacing, and visual composition. Researchers from the Singapore University of Technology and Design (SUTD) address this by proposing a multimodal deep learning system designed to predict viewer Arousal (emotional intensity) and Valence (emotional positivity).

Methodology

The core of the proposed approach is a hybrid neural network architecture that processes three distinct modalities to capture the multifaceted nature of film.

Multimodal Feature Extraction

  1. Visual (Spatial): A pre-trained ResNet-50 model extracts features from static RGB frames, identifying objects and settings that define the scene's context.
  2. Visual (Temporal/Motion): To capture the dynamic elements of storytelling, the researchers utilized optical flow maps calculated via PWC-Net. These maps, which represent movement between frames, were processed by a ResNet-101 to extract motion-based features.
  3. Audio: Acoustic features were extracted using the OpenSMILE toolkit, generating 1,582 low-level descriptors (such as pitch, loudness, and cepstrum) from 400ms audio segments.
The proposed multimodal architecture featuring spatial, temporal, and audio processing branches.

Model Architectures

The extracted features are concatenated and passed through a classification pipeline. The researchers compared two primary architectures: a simple Fully Connected (FC) model that treats time steps independently, and a Long Short-Term Memory (LSTM) network designed to model temporal dependencies. To handle the subjectivity of the task, continuous emotion scores were quantized into seven discrete bins for classification.

Experimental Results

The study utilized the extended COGNIMUSE dataset, which includes annotations for both "intended" emotion (filmmaker's goal) and "experienced" emotion (actual viewer response) across various Hollywood film clips.

The multimodal approach, combining all three streams, achieved superior performance compared to unimodal baselines and previous state-of-the-art methods. For experienced emotion, the model reached an accuracy of 53.32% for Arousal and 43.10% for Valence. Notably, when considering a tolerance of ±1 bin, the accuracy jumped to over 90% for both dimensions.

Comparison of ground truth emotional responses versus model predictions for Arousal and Valence over time.

Discussion and Key Insights

Several critical insights emerged from the experimental data:

  • Dominance of Audio: Models relying solely on audio features consistently outperformed those using only visual features. This underscores the primary role of sound design and soundtracks in driving cinematic emotion.
  • Motion vs. Objects: Within the visual domain, optical flow (motion) was found to be more informative than static RGB frames. This suggests that camera movement and action pacing are more potent emotional cues than the specific objects depicted.
  • The Subjectivity Gap: The system was more effective at predicting intended emotions than experienced ones. This highlights the "noise" introduced by individual viewer biases and personal histories, which remain difficult for AI to model universally.

Conclusion

By integrating spatial, temporal, and acoustic data, the SUTD researchers have established a robust framework for predicting the emotional impact of movies. While challenges remain regarding the inherent subjectivity of human experience and the need for larger datasets, this work demonstrates that a multimodal approach is essential for capturing the complex "language" of film.

Summary updated

Abstract

The goal of this study is to develop and analyze multimodal models for predicting experienced affective responses of viewers watching movie clips. We develop hybrid multimodal prediction models based on both the video and audio of the clips. For the video content, we hypothesize that both image content and motion are crucial features for evoked emotion prediction. To capture such information, we extract features from RGB frames and optical flow using pre-trained neural networks. For the audio model, we compute an enhanced set of low-level descriptors including intensity, loudness, cepstrum, linear predictor coefficients, pitch and voice quality. Both visual and audio features are then concatenated to create audio-visual features, which are used to predict the evoked emotion. To classify the movie clips into the corresponding affective response categories, we propose two approaches based on deep neural network models. The first one is based on fully connected layers without memory on the time component, the second incorporates the sequential dependency with a long short-term memory recurrent neural network (LSTM). We perform a thorough analysis of the importance of each feature set. Our experiments reveal that in our set-up, predicting emotions at each time step independently gives slightly better accuracy performance than with the LSTM. Interestingly, we also observe that the optical flow is more informative than the RGB in videos, and overall, models using audio features are more accurate than those based on video features when making the final prediction of evoked emotions.
BibTeX
@inproceedings{thao2019multimodal,
  author = {Ha Thi Phuong Thao and Dorien Herremans and Gemma Roig},
  title = {{Multimodal Deep Models for Predicting Affective Responses Evoked by Movies}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  doi = {10.1109/iccvw.2019.00201},
}
Figure