How You Feelin'? Learning Emotions and Mental States in Movie Scenes

Dhruv Srivastava, Aditya Kumar Singh, Makarand TapaswiCVPR 2023 · DOI · arXiv

Abstract

Recognizing emotions in cinematic narratives requires moving beyond simple facial expression analysis to a holistic understanding of scene context. We present EmoTx, a multimodal Transformer-based architecture designed to identify a multi-label set of emotions and complex mental states in movie scenes. By integrating video context, character facial features, and dialog utterances, EmoTx captures the nuances of human behavior that single-modality models miss. Our approach demonstrates a significant performance leap over previous state-of-the-art methods on the MovieGraphs dataset, particularly in identifying latent states like "honest" or "determined."

Introduction

Most emotion recognition research focuses on "in-the-wild" facial expressions or short clips limited to basic categories like Ekman’s six emotions. However, real-world storytelling in movies is far more complex. Characters often experience multiple emotions simultaneously—such as being both "curious" and "confused"—and their internal mental states are often invisible in a single frame. Machines need to understand emotions at the scene level (30–60 seconds), where context from dialog and the surrounding environment is crucial. The challenge lies in building a model that can ingest these disparate streams of data to predict a rich, multi-label set of attributes for every character in a scene.

Methodology

EmoTx is a multimodal Transformer encoder designed to handle the long-form nature of movie scenes. Its architecture integrates three primary data streams:

  1. Video: Spatio-temporal features extracted using a Multiscale Vision Transformer (MViT).
  2. Characters: Facial features captured via a ResNet50 pretrained on emotion datasets.
  3. Dialog: Utterance embeddings from a fine-tuned RoBERTa model.
The EmoTx architecture integrates video, character, and dialog features through a Transformer encoder with specialized learnable classifier tokens for multi-label prediction.

To optimize multi-label classification, EmoTx utilizes a "Query2Label" approach with $K$ learnable classifier tokens (one for each emotion label). This allows the model to capture correlations between different emotions, such as the frequent co-occurrence of "surprised" and "excited." The model is trained to predict emotions simultaneously for the overall scene and for each individual character, utilizing discrete time-binning and modality-specific embeddings to maintain temporal and contextual clarity.

Datasets

The study primarily utilizes the MovieGraphs dataset, which contains detailed graph-based annotations for 7,637 scenes across 51 movies. The authors evaluate on three label sets: the Top-10 most frequent emotions, the Top-25, and a mapping of 181 MovieGraphs labels to the 26 categories defined in the Emotic dataset. To ensure consistent character identification throughout a scene, a custom pipeline was developed using MTCNN for face detection, SORT for tracking, and C1C for clustering.

Results and Discussion

EmoTx demonstrates a significant performance leap over adapted state-of-the-art models like CAER, EmotionNet, and M2Fnet. On the Top-10 labels, EmoTx achieved a mean Average Precision (mAP) of 34.22, a nearly 10-point absolute improvement over the next best model (M2Fnet at 24.55).

Average Precision (AP) scores across various emotion and mental state labels, comparing EmoTx variants against baseline architectures.

Key insights from the results include:

  • Attention "Switching": Analysis of self-attention scores reveals that the model shifts focus based on the label. Expressive emotions (e.g., cheerful) rely heavily on character face tokens, while abstract mental states (e.g., helpful) depend more on dialog and video context.
  • The Multi-Label Necessity: Dataset statistics show that the average movie scene contains approximately four distinct emotions, making single-label classification fundamentally insufficient for story understanding.
  • Modality Importance: Ablation studies confirmed that removing any one modality (video, character, or dialog) led to a performance drop, proving that emotion recognition in complex narratives is a truly multimodal task.

Conclusion and Future Work

While EmoTx sets a new benchmark for scene-level emotion recognition, several avenues for improvement remain. The current model does not utilize audio features, such as tone of voice or prosody, which carry significant emotional weight. Additionally, the model treats each scene independently; future work could incorporate "long-term" memory to account for character development across an entire movie. Finally, while the model excels at identifying high-energy states like "excited," it still faces challenges with subtle or rare mental states that require a deeper semantic understanding of the plot.

Summary updated

BibTeX
@inproceedings{srivastava2023how,
  author = {Dhruv Srivastava and Aditya Kumar Singh and Makarand Tapaswi},
  title = {{How You Feelin'? Learning Emotions and Mental States in Movie Scenes}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00248},
}
Figure