Introduction
While significant progress has been made in automated emotion recognition—the detection of facial expressions or vocal cues from a subject—predicting the evoked emotion (the affective response of a viewer to a stimulus) remains a formidable challenge. This task is inherently subjective, as the emotional "vibe" of a cinematic scene is constructed through a complex interplay of music, narrative pacing, and visual composition. Researchers from the Singapore University of Technology and Design (SUTD) address this by proposing a multimodal deep learning system designed to predict viewer Arousal (emotional intensity) and Valence (emotional positivity).
Methodology
The core of the proposed approach is a hybrid neural network architecture that processes three distinct modalities to capture the multifaceted nature of film.
Multimodal Feature Extraction
- Visual (Spatial): A pre-trained ResNet-50 model extracts features from static RGB frames, identifying objects and settings that define the scene's context.
- Visual (Temporal/Motion): To capture the dynamic elements of storytelling, the researchers utilized optical flow maps calculated via PWC-Net. These maps, which represent movement between frames, were processed by a ResNet-101 to extract motion-based features.
- Audio: Acoustic features were extracted using the OpenSMILE toolkit, generating 1,582 low-level descriptors (such as pitch, loudness, and cepstrum) from 400ms audio segments.
Model Architectures
The extracted features are concatenated and passed through a classification pipeline. The researchers compared two primary architectures: a simple Fully Connected (FC) model that treats time steps independently, and a Long Short-Term Memory (LSTM) network designed to model temporal dependencies. To handle the subjectivity of the task, continuous emotion scores were quantized into seven discrete bins for classification.
Experimental Results
The study utilized the extended COGNIMUSE dataset, which includes annotations for both "intended" emotion (filmmaker's goal) and "experienced" emotion (actual viewer response) across various Hollywood film clips.
The multimodal approach, combining all three streams, achieved superior performance compared to unimodal baselines and previous state-of-the-art methods. For experienced emotion, the model reached an accuracy of 53.32% for Arousal and 43.10% for Valence. Notably, when considering a tolerance of ±1 bin, the accuracy jumped to over 90% for both dimensions.
Discussion and Key Insights
Several critical insights emerged from the experimental data:
- Dominance of Audio: Models relying solely on audio features consistently outperformed those using only visual features. This underscores the primary role of sound design and soundtracks in driving cinematic emotion.
- Motion vs. Objects: Within the visual domain, optical flow (motion) was found to be more informative than static RGB frames. This suggests that camera movement and action pacing are more potent emotional cues than the specific objects depicted.
- The Subjectivity Gap: The system was more effective at predicting intended emotions than experienced ones. This highlights the "noise" introduced by individual viewer biases and personal histories, which remain difficult for AI to model universally.
Conclusion
By integrating spatial, temporal, and acoustic data, the SUTD researchers have established a robust framework for predicting the emotional impact of movies. While challenges remain regarding the inherent subjectivity of human experience and the need for larger datasets, this work demonstrates that a multimodal approach is essential for capturing the complex "language" of film.
Summary updated