TL;DR
This paper introduces MER-CLIP, a novel framework for multimodal emotion recognition that leverages the powerful semantic understanding of CLIP. By using a label encoder to guide the learning process and a cross-modal decoder to fuse information, MER-CLIP achieves state-of-the-art performance on benchmark datasets, even with limited and potentially uncertain training data.
What Problem Does This Paper Address?
Multimodal emotion recognition (MER) aims to understand human emotions by combining information from various sources like text, audio, and video. While effective, MER research faces a significant hurdle: the scarcity of large-scale, high-quality datasets with multiple modalities. Creating such datasets is challenging due to the subjective nature of emotion annotation and the complexity of collecting synchronized data across different channels. This data limitation hinders the development of robust MER models. Existing approaches often struggle to effectively fuse information from different modalities and generalize to a wide range of emotion labels.
Methods
The core of MER-CLIP lies in its innovative use of the CLIP (Contrastive Language-Image Pre-training) model. Instead of relying on traditional modality-specific encoders, MER-CLIP adapts CLIP's architecture for each modality (vision, language, audio).
Here's a breakdown of the key components:
- CLIP-based Modality Encoders: The paper utilizes the pre-trained weights of CLIP's image and text encoders to extract rich semantic features from visual, language, and audio inputs. This leverages CLIP's extensive knowledge gained from massive internet-scale datasets.
- Label Encoder (LE): A crucial innovation is the introduction of a label encoder, also based on CLIP's text encoder. This encoder treats emotion labels (e.g., "happy," "sad") as text embeddings, allowing the model to understand their semantic meaning. This is a departure from traditional methods that often treat labels as simple categorical identifiers.
- Cross-Modal Decoder (CMD): To effectively fuse the modality features and align them with emotion labels, a cross-modal decoder is employed. This decoder uses an attention mechanism, where emotion-aligned embeddings from the label encoder act as queries. This allows the model to learn correlations across modalities in an emotion-aware manner.
- Label Encoder-Guided Prediction: Instead of a standard fully-connected layer for prediction, MER-CLIP uses the cosine similarity between the final multimodal representation and the label embeddings from the LE. This approach enhances generalization, enabling the model to handle diverse label types, including phrases and sentences, not just single words.
Datasets
The paper evaluates MER-CLIP on two widely used multimodal datasets:
- CMU-MOSI: A multimodal sentiment analysis dataset containing short monologue video clips.
- CMU-MOSEI: A larger multimodal dataset for sentiment and emotion analysis, featuring video clips from numerous YouTube speakers.
Both datasets provide annotations for sentiment (on a scale of -3 to +3) and CMU-MOSEI also includes labels for six emotion categories (happy, sad, angry, surprise, disgust, fear). The authors generated utterance-level data from these datasets to ensure a perfect match with preprocessed data for fair comparison.
Key Insights and Ideas
- CLIP's Semantic Power for MER: The paper demonstrates that CLIP's pre-trained semantic understanding can be effectively transferred to the MER task, overcoming the limitations of smaller, specialized MER datasets.
- Label Semantics Matter: Treating emotion labels as semantic embeddings (via the label encoder) significantly improves the model's ability to understand and predict emotions, especially when dealing with diverse or nuanced labels.
- Emotion-Guided Fusion: The cross-modal decoder, guided by emotion queries from the label encoder, allows for a more targeted and effective fusion of multimodal information, leading to richer representations.
- Generalization Beyond Words: The label encoder-guided prediction mechanism allows the model to generalize to phrase-level and even sentence-level descriptions of emotions, showcasing its adaptability.
Key Results
MER-CLIP achieves state-of-the-art performance on both CMU-MOSI and CMU-MOSEI datasets for multimodal emotion recognition.
- On CMU-MOSEI, MER-CLIP achieved the highest accuracy (49.3%) and Micro-F1 (57.8%) among the compared methods for emotion recognition.
- For sentiment analysis on CMU-MOSI and CMU-MOSEI, MER-CLIP also demonstrated superior performance, achieving an accuracy of 84.0% and 85.3% respectively, and F1 scores of 84.0% and 85.1%.
- Ablation studies confirmed the effectiveness of both the cross-modal decoder (CMD) and the label encoder (LE), with MER-CLIP (including both) outperforming models with only one or neither component.
- The model showed comparable performance when using sentence or phrase descriptions for labels, highlighting its generalization capabilities.
Limitations and Future Work
- Data Uncertainty: The paper acknowledges that label uncertainties in datasets can still pose a challenge, even with the proposed methods.
- Modality Fusion Order: While the LVA (Language, Vision, Audio) fusion order in the cross-modal decoder yielded the best results, the paper notes that changing this order can lead to decreased performance, suggesting further exploration into optimal fusion strategies.
- Generalization to Different Labels: While the LE shows good generalization, the paper suggests further investigation into its capabilities with even more complex or abstract label types.
- Computational Cost: Leveraging large pre-trained models like CLIP can be computationally intensive, which might be a consideration for real-time applications.
The authors suggest that their framework's ability to leverage rich semantics from pre-trained models and its flexible label encoder-guided prediction scheme open avenues for more robust and adaptable multimodal emotion recognition systems.
Summary updated