Multi-modal Multi-label Emotion Detection with Modality and Label Dependence

Dong Zhang, Xincheng Ju, Junhui Li, Shoushan Li, Qiaoming Zhu, Guodong ZhouEMNLP 2020 · OpenReview

Abstract

Human emotions are inherently complex and multi-faceted, often manifesting as a combination of distinct emotional states. Traditional emotion detection systems frequently struggle with this complexity due to a narrow focus on single modalities or an assumption of label independence. In this work, we present MMS2S (Multi-modal Seq2Set), a framework that leverages text, video, and audio to detect multiple emotions simultaneously. By incorporating a multi-head modality attention mechanism and a sequence-to-set learning objective, MMS2S effectively models the dependencies between different modalities and emotion labels. Our approach achieves state-of-the-art performance on the CMU-MOSEI dataset, demonstrating the importance of integrated multi-modal analysis for capturing the nuances of human expression.

Introduction

Human expression is a symphony of text, tone, and facial movement. While AI has made significant progress in text-based emotion detection, real-world scenarios often involve "multi-label" emotions—such as bittersweetness or angry disappointment. Existing systems face two primary hurdles: a reliance on single modalities (often ignoring visual or acoustic cues) and the treatment of emotions as independent variables. For instance, "Sadness" might be evident in a transcript, but "Disgust" often requires visual context to be accurately identified.

An example of multi-modal input (text, visual frames, and audio waveform) resulting in multiple emotion labels (Sad, Disgust).

The MMS2S Framework

To address these challenges, we propose the Multi-modal Seq2Set (MMS2S) model, built upon the Transformer architecture. The framework consists of three critical components:

1. Multi-modal Encoding

The model employs three independent Transformer encoders to process distinct data streams:

  • Textual: GloVe embeddings capture semantic meaning.
  • Visual: Facial action units and landmarks track physical expressions.
  • Acoustic: MFCCs and pitch capture tonal nuances.

2. Multi-head Modality Attention

This core innovation allows the model to dynamically weigh the importance of each modality during the decoding phase. Instead of using a static fusion of data, the attention mechanism determines which sense—be it a facial scowl or a sarcastic tone—is most relevant for the specific emotion being predicted at any given step.

3. Sequence-to-Set (Seq2Set) Learning

Traditional sequence-to-sequence models are often biased by the arbitrary order of labels in training data. MMS2S treats emotion labels as a set rather than a sequence. By considering all possible permutations during training and summing their probabilities during inference, the model achieves greater robustness and better captures inter-label synergies (e.g., the co-occurrence of "Sad" and "Disgust").

Experimental Results

We evaluated MMS2S on CMU-MOSEI, the largest benchmark for multi-modal emotion detection. The dataset contains utterance-level video clips annotated with six emotion categories: Happiness, Sadness, Anger, Fear, Disgust, and Surprise.

MMS2S significantly outperformed state-of-the-art baselines, including MulT and RAVEN:

  • Accuracy: MMS2S reached 0.475, compared to MulT's 0.445.
  • F1 Score: The model achieved 0.560, a notable improvement over the previous best of 0.531.

Ablation Analysis

Ablation studies confirmed the necessity of both modality and label modeling. Removing Modality Attention (MMS2S w/o M) caused accuracy to drop to 0.421, while removing Label Dependence modeling (MMS2S w/o L) resulted in a drop to 0.417.

Conclusion and Future Work

MMS2S demonstrates that modeling the dependencies between modalities and labels is essential for accurate emotion detection. Future research will explore contextual awareness—incorporating the emotions of previous speakers in a conversation—and the use of Graph Neural Networks (GNNs) to more explicitly map the relationships between different emotional states. Additionally, testing the model's robustness against noisy, unaligned "in-the-wild" data remains a priority for real-world application.

Summary updated

BibTeX
@inproceedings{zhang2020multi,
  author = {Dong Zhang and Xincheng Ju and Junhui Li and Shoushan Li and Qiaoming Zhu and Guodong Zhou},
  title = {{Multi-modal Multi-label Emotion Detection with Modality and Label Dependence}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=5i_za-lGdnn},
}
Figure