Abstract
Human emotions are inherently complex and multi-faceted, often manifesting as a combination of distinct emotional states. Traditional emotion detection systems frequently struggle with this complexity due to a narrow focus on single modalities or an assumption of label independence. In this work, we present MMS2S (Multi-modal Seq2Set), a framework that leverages text, video, and audio to detect multiple emotions simultaneously. By incorporating a multi-head modality attention mechanism and a sequence-to-set learning objective, MMS2S effectively models the dependencies between different modalities and emotion labels. Our approach achieves state-of-the-art performance on the CMU-MOSEI dataset, demonstrating the importance of integrated multi-modal analysis for capturing the nuances of human expression.
Introduction
Human expression is a symphony of text, tone, and facial movement. While AI has made significant progress in text-based emotion detection, real-world scenarios often involve "multi-label" emotions—such as bittersweetness or angry disappointment. Existing systems face two primary hurdles: a reliance on single modalities (often ignoring visual or acoustic cues) and the treatment of emotions as independent variables. For instance, "Sadness" might be evident in a transcript, but "Disgust" often requires visual context to be accurately identified.
The MMS2S Framework
To address these challenges, we propose the Multi-modal Seq2Set (MMS2S) model, built upon the Transformer architecture. The framework consists of three critical components:
1. Multi-modal Encoding
The model employs three independent Transformer encoders to process distinct data streams:
- Textual: GloVe embeddings capture semantic meaning.
- Visual: Facial action units and landmarks track physical expressions.
- Acoustic: MFCCs and pitch capture tonal nuances.
2. Multi-head Modality Attention
This core innovation allows the model to dynamically weigh the importance of each modality during the decoding phase. Instead of using a static fusion of data, the attention mechanism determines which sense—be it a facial scowl or a sarcastic tone—is most relevant for the specific emotion being predicted at any given step.
3. Sequence-to-Set (Seq2Set) Learning
Traditional sequence-to-sequence models are often biased by the arbitrary order of labels in training data. MMS2S treats emotion labels as a set rather than a sequence. By considering all possible permutations during training and summing their probabilities during inference, the model achieves greater robustness and better captures inter-label synergies (e.g., the co-occurrence of "Sad" and "Disgust").
Experimental Results
We evaluated MMS2S on CMU-MOSEI, the largest benchmark for multi-modal emotion detection. The dataset contains utterance-level video clips annotated with six emotion categories: Happiness, Sadness, Anger, Fear, Disgust, and Surprise.
MMS2S significantly outperformed state-of-the-art baselines, including MulT and RAVEN:
- Accuracy: MMS2S reached 0.475, compared to MulT's 0.445.
- F1 Score: The model achieved 0.560, a notable improvement over the previous best of 0.531.
Ablation Analysis
Ablation studies confirmed the necessity of both modality and label modeling. Removing Modality Attention (MMS2S w/o M) caused accuracy to drop to 0.421, while removing Label Dependence modeling (MMS2S w/o L) resulted in a drop to 0.417.
Conclusion and Future Work
MMS2S demonstrates that modeling the dependencies between modalities and labels is essential for accurate emotion detection. Future research will explore contextual awareness—incorporating the emotions of previous speakers in a conversation—and the use of Graph Neural Networks (GNNs) to more explicitly map the relationships between different emotional states. Additionally, testing the model's robustness against noisy, unaligned "in-the-wild" data remains a priority for real-world application.
Summary updated