Topic and Style-aware Transformer for Multimodal Emotion Recognition

Shuwen Qiu, Nitesh Sekhar, Prateek SinghalACL 2023 · DOI

Abstract

Multimodal Emotion Recognition (MER) often struggles to effectively utilize visual data due to its high dimensionality and inherent subjectivity. This paper presents a transformer-based framework that "normalizes" facial expressions by incorporating Topic and Speaking Style information. By leveraging the VATT backbone and a cross-attention mechanism to calibrate emotional signals against context, the proposed model achieves state-of-the-art performance on the CMU-MOSEI dataset, demonstrating that visual cues are a powerful predictive signal when properly contextualized.

Introduction

In Multimodal Emotion Recognition (MER), models frequently rely more on text and audio than visual data. Video frames are often perceived as "noise" because emotional expression is highly context-dependent and subjective. For instance, a "happy" expression during a casual movie review differs significantly from one during a formal public speech. Furthermore, individual baselines vary; a naturally calm person's excitement might look like a more expressive person's neutral state.

Illustrating how topic and style influence emotional expression across different contexts and individuals.

The authors address these challenges by introducing "content normalization," which uses topic and style as anchors to interpret visual signals more accurately.

Methodology

The proposed architecture integrates several components to achieve context-aware emotion recognition.

The Topic and Style-aware Transformer architecture showing the integration of VATT, topic embeddings, and style codes.

Multimodal Backbone: VATT

The framework utilizes the Video-Audio-Text Transformer (VATT) as its backbone. VATT projects raw signals from different modalities into a shared latent space, facilitating better alignment between visual features and their corresponding text and audio.

Topic and Style Feature Extraction

To capture context, the model employs Latent Dirichlet Allocation (LDA) to identify the topic of the transcript (e.g., Finance, Movie Reviews). Simultaneously, it extracts individual "speaking style" features using a 3D Morphable Model (3DMM). By analyzing facial expression coefficients over time, the model establishes a baseline for the speaker's natural expressiveness.

Feature Aggregation via Cross-Attention

Rather than simple concatenation, the model uses a cross-attention mechanism. The video context acts as a "query" to the topic and style features, allowing the model to calibrate raw emotional signals based on the specific context and the speaker's unique baseline.

Experimental Setup and Results

The model was evaluated on the CMU-MOSEI dataset, which contains over 23,500 sentence-level utterances from more than 1,000 YouTube speakers. The dataset is annotated for sentiment and six basic emotions.

The "Ours-Full" model demonstrated superior performance compared to existing state-of-the-art (SOTA) methods like Multilogue-Net and TBJE. Specifically, it achieved a Weighted F1 of 71.18 for "Happy" and 73.57 for "Sad," outperforming previous benchmarks. Ablation studies confirmed the importance of the visual modality; removing video features led to a significant drop in F1 scores, proving that the architecture successfully extracts meaningful information from visual cues.

Conclusion and Future Work

This research highlights that visual data in MER is not inherently noisy but requires calibration. By treating topic and style as essential metadata, the proposed transformer framework effectively normalizes emotional content across different contexts and speakers. Future work will explore more advanced deep-learning-based topic clustering and aim for a fully end-to-end trainable framework to further enhance performance across diverse emotional categories and languages.

Summary updated

BibTeX
@inproceedings{qiu2023topic,
  author = {Shuwen Qiu and Nitesh Sekhar and Prateek Singhal},
  title = {{Topic and Style-aware Transformer for Multimodal Emotion Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.130},
}
Figure