Abstract
Multimodal Emotion Recognition (MER) aims to automatically identify and understand human emotional states by integrating information from various modalities. However, the scarcity of annotated multimodal data significantly hinders the advancement of this research field. This paper presents a solution for the MER-SEMI sub-challenge of MER 2024. First, to better adapt acoustic modality features for the MER task, we evaluate the contributions of different layers of the pre-trained speech model HuBERT. Based on these observations, we perform Parameter-Efficient Fine-Tuning (PEFT) on the layers identified as most effective for emotion recognition. Second, leveraging the strengths of the acoustic modality, we propose a feature alignment pre-training method. This approach uses large-scale unlabeled data to train a visual encoder, promoting semantic alignment of visual features within the acoustic feature space. Finally, using adapted acoustic features, aligned visual features, and lexical features, we employ an attention mechanism for feature fusion. On the MER2024-SEMI test set, the proposed method achieves a weighted F1 score of 88.90%, ranking fourth among all participating teams.
Introduction
Understanding human emotions through Multimodal Emotion Recognition (MER) involves the complex integration of audio, video, and textual data. Despite its potential, the field faces two primary challenges: data scarcity and the modality gap. Annotating emotional data is labor-intensive, resulting in small labeled datasets that are prone to overfitting. Furthermore, visual information, such as facial expressions, often underperforms compared to acoustic signals in emotion-related tasks. Simply concatenating these disparate modalities is often ineffective because they exist in different semantic spaces. Our approach addresses these issues by leveraging a massive pool of unlabeled data to align the visual modality with the more descriptive acoustic modality.
Methodology
The proposed framework is structured into three distinct stages: acoustic feature adaptation, visual feature alignment, and multimodal feature fusion.
1. Acoustic Feature Adaptation
Rather than treating the pre-trained HuBERT-large model as a static feature extractor, we conducted an empirical study to identify which transformer layers are most sensitive to emotional content. Our findings indicate that intermediate layers (specifically layers 16–21) provide a "sweet spot" that balances low-level physical signals (pitch, energy) with abstract linguistic meaning.
To adapt these layers efficiently, we utilized Parameter-Efficient Fine-Tuning (PEFT) by inserting small, trainable "bottleneck" adapter layers. This allows the model to specialize in emotion recognition without retraining the entire 300M+ parameter backbone. We employed a hybrid loss function—combining supervised Cross-Entropy loss with an unsupervised masked reconstruction loss—to ensure robust acoustic representations.
2. Visual Feature Alignment
To bridge the semantic gap between visual (CLIP-large) and acoustic modalities, we introduced a contrastive pre-training stage. Using over 115,000 unlabeled video/audio pairs, we trained a Vision MLP to map visual features into the fine-tuned HuBERT feature space. By maximizing the similarity between corresponding audio and video clips, the visual encoder learns to interpret emotion in a way that complements the acoustic signal, effectively using the stronger acoustic modality as an anchor.
3. Multimodal Feature Fusion
Lexical features were extracted using the Baichuan2 Large Language Model (LLM). These textual features, along with the adapted acoustic and aligned visual features, were integrated using an attention-based fusion module. This module dynamically assigns weights to each modality, allowing the system to prioritize the most reliable information source for any given video clip.
Results and Discussion
The efficacy of our framework was validated on the MER 2024 dataset, which includes 5,030 labeled clips and a massive unlabeled set for pre-training. Our system achieved a weighted F1 score of 88.90% on the MER2024-SEMI test set.
Key performance gains were observed from our specific adaptation and alignment strategies:
- Acoustic Gains: Fine-tuning HuBERT layers 16–21 with adapters improved the validation F1 score from a baseline of 72.82% to 80.24%.
- Alignment Impact: Utilizing aligned visual features (CLIPL-A) instead of raw CLIP features resulted in a 3.64% improvement in unimodal test performance, confirming that semantic alignment successfully bridges the modality gap.
- Synergy: The final multimodal fusion outperformed all single-modality benchmarks, demonstrating that the alignment strategy effectively integrated complementary emotional cues.
Conclusion
By focusing on targeted acoustic adaptation and visual-acoustic alignment, we developed a robust MER pipeline that excels even in data-scarce environments. Our results highlight the importance of identifying emotion-sensitive layers in pre-trained models and the power of using stronger modalities as anchors for cross-modal learning. Future work will explore deeper alignment between lexical features and other modalities to further enhance recognition accuracy.
Summary updated