AttendAffectNet-Emotion Prediction of Movie Viewers Using Multimodal Fusion with Self-Attention

Ha Thi Phuong Thao, Balamurali B. T., Gemma Roig, Dorien Herremans

TL;DR

This paper introduces AttendAffectNet (AAN), a novel multimodal deep learning architecture that leverages self-attention to predict movie viewers' emotional responses (valence and arousal) by fusing visual, audio, and text features. AAN demonstrates superior performance compared to existing methods, highlighting the significant influence of audio features and the effectiveness of self-attention in capturing cross-modal and temporal dependencies.

What Problem Does This Paper Address?

Predicting the emotional impact of media, particularly movies, on viewers is a challenging but important task with applications in film production, marketing, and understanding human-computer interaction. While previous research has focused on extracting features from video and audio and fusing them, many approaches overlook the crucial correlations between different modalities and the temporal dynamics of emotional responses. This paper aims to address this gap by proposing a multimodal fusion approach that explicitly models these interdependencies using self-attention.

Methods

The core of this work is the AttendAffectNet (AAN) architecture, inspired by the Transformer model's self-attention mechanism. AAN processes multimodal features (visual, audio, text) extracted from movie segments. The authors propose three variants:

  • Feature AAN: Applies self-attention across features from different modalities within a movie segment to capture inter-modal relationships.
  • Temporal AAN: Applies self-attention to concatenated multimodal feature vectors from sequential movie segments, accounting for temporal dependencies.
  • Mixed AAN: Combines both Feature and Temporal AAN by first applying self-attention within segments and then across segments.

Key techniques include:

  • Feature Extraction: Pre-trained deep neural networks like ResNet-50 and I3D for visual features, OpenSMILE and VGGish for audio, and BERT for text (movie subtitles).
  • Self-Attention Mechanism: Used to weigh the importance of different features and temporal segments, allowing the model to focus on the most relevant information.
  • Multimodal Fusion: Integration of features from video, audio, and text to create a comprehensive representation of the movie content.

Datasets

The proposed models were trained and evaluated on two datasets:

  • Extended COGNIMUSE Dataset: Contains 30-minute movie clips with frame-level annotations for arousal and valence. For this study, 5-second excerpts were used, with averaged intended emotion annotations.
  • EIMT16 Dataset (Global EIMT16 subset): Comprises approximately 11,000 movie excerpts (8-12 seconds each) with annotated arousal/valence values representing expected evoked emotions.

Key Insights and Ideas

  • Audio Dominance: Audio features, particularly music and speech, play a more influential role in predicting viewer emotions than visual or text features alone. This is attributed to the intentional use of audio to set emotional context.
  • Self-Attention's Power: The self-attention mechanism effectively captures complex relationships between different modalities and across temporal segments, leading to improved emotion prediction accuracy.
  • Holistic Approach Benefits: Combining features from all three modalities (visual, audio, text) consistently yields better results than using any single modality or just two.
  • Feature AAN Superiority: The Feature AAN variant, which applies self-attention across modalities within a segment, generally outperformed the Temporal and Mixed AAN variants, suggesting that inter-modal context is highly valuable.

Key Results

  • The proposed AttendAffectNet models, especially the Feature AAN, significantly outperformed baseline models (fully connected layers and LSTM) on both datasets.
  • When using only audio features, the Feature AAN achieved MSE values of 0.125 and 1.111 for arousal, and 0.185 and 0.209 for valence on the extended COGNIMUSE and Global EIMT16 datasets, respectively.
  • With all modalities combined, the Feature AAN achieved the best MSE and PCC on the extended COGNIMUSE dataset (arousal: 0.117 MSE, 0.655 PCC; valence: 0.170 MSE, 0.575 PCC).
  • The models showed better performance on the extended COGNIMUSE dataset (intended emotion) compared to the Global EIMT16 dataset (expected emotion).
  • Qualitative analysis (Figures 5 & 6) showed good correlation between predicted and ground truth arousal values, with valence showing slightly less correlation.

Limitations and Future Work

  • Feature Selection: The paper acknowledges that using a wide range of features might introduce redundancy. Future work could explore feature selection techniques to remove less informative features.
  • Standard Deviation of Features: The study used element-wise averaging of feature vectors. Incorporating the standard deviation of extracted features, as done in some prior work, could potentially further improve performance.
  • Dominance Dimension: While the Circumplex model includes dominance, it was omitted due to annotation difficulties. Future work could explore its inclusion if annotated data becomes available.
  • Model Variants: While the Mixed AAN combines temporal and feature attention, it did not yield improved accuracy over the individual variants. Further research could explore more sophisticated fusion strategies.
  • Dataset Limitations: The Global EIMT16 dataset lacks subtitle information, limiting multimodal experiments on that dataset.

Summary updated

BibTeX
@inproceedings{thao2021attendaffectnet,
  author = {Ha Thi Phuong Thao and Balamurali B. T. and Gemma Roig and Dorien Herremans},
  title = {{AttendAffectNet-Emotion Prediction of Movie Viewers Using Multimodal Fusion with Self-Attention}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=e6WDsCO01T},
}
Figure