Improving Multimodal Emotion Recognition by Leveraging Acoustic Adaptation and Visual Alignment

Zhixian Zhao, Haifeng Chen, Xi Li, Dongmei Jiang, Lei Xie

TL;DR

Researchers have developed a high-performing multimodal emotion recognition (MER) framework that tackles the "data scarcity" problem by focusing on two key strategies: fine-tuning specific intermediate layers of a speech model (HuBERT) using parameter-efficient adapters and aligning visual features to the acoustic space through contrastive learning on large-scale unlabeled data. Their approach secured 4th place in the MER 2024 competition with a weighted F1 score of 88.90%.

What Problem Does This Paper Address?

Multimodal Emotion Recognition (MER) aims to understand human emotions by combining audio, video, and text. However, the field faces two major hurdles:

  1. Data Scarcity: Annotating emotional data is incredibly expensive and time-consuming, leading to small labeled datasets.
  2. The Modality Gap: Visual information (like facial expressions) often performs worse than acoustic information in emotion tasks. Simply concatenating these different "languages" of data doesn't work well because they exist in different semantic spaces.

The authors address these by leveraging the MER-SEMI challenge's massive pool of unlabeled data to "teach" the visual modality how to align with the more descriptive acoustic modality.

Methods

The framework consists of three distinct stages:

1. Acoustic Feature Adaptation

Instead of using the entire pre-trained HuBERT-large model as a black box, the authors conducted an empirical study to find which layers actually "care" about emotion. They discovered that intermediate layers (16–21) are the most effective.

  • Adapters: They inserted small, trainable "bottleneck" layers (adapters) into these specific transformer layers. This allows for Parameter-Efficient Fine-Tuning (PEFT)—adapting the model to emotion without retraining all 300M+ parameters.
  • Hybrid Loss: They used a combination of supervised Cross-Entropy loss and an unsupervised masked reconstruction loss to make the acoustic representations more robust.

2. Visual Feature Alignment

To bridge the gap between the visual (CLIP-large) and acoustic modalities, the authors proposed a pre-training step.

  • Contrastive Learning: Using unlabeled video/audio pairs, they trained a Vision MLP to map visual features into the same semantic space as the fine-tuned HuBERT features.
  • Similarity Scoring: By maximizing the similarity between corresponding audio and video clips (and minimizing it for mismatched ones), the visual encoder learns to "see" emotion in a way that complements what the model "hears."

3. Multimodal Feature Fusion

Finally, they extracted lexical (text) features using the Baichuan2 LLM. All three modalities (audio, visual, text) were fed into an attention-based fusion module. This module dynamically assigns weights to each modality, allowing the model to decide which source is most reliable for a given clip.

Datasets

The study used the MER 2024 dataset:

  • Train & Val: 5,030 video clips with discrete emotion labels (neutral, anger, happiness, sadness, worry, surprise).
  • Unlabeled Data: A massive set of 115,595 unlabeled video clips used for the contrastive alignment and unsupervised acoustic training.
  • MER-SEMI Test Set: 1,169 unlabeled clips used for final evaluation.

Key Insights and Ideas

  • The "Goldilocks" Layers: Shallow layers in speech models capture low-level physics (pitch, energy), while the deepest layers capture abstract linguistics. The authors found that the "sweet spot" for emotion lies in the middle-to-late layers (16–21), which balance physical signal and semantic meaning.
  • Acoustic as the Anchor: Because the acoustic modality is naturally stronger for emotion recognition, it serves as the "anchor" for the visual modality. Aligning CLIP features to HuBERT features significantly boosted performance compared to using raw CLIP features.
  • Efficiency Matters: By using adapters and only fine-tuning specific layers, the researchers achieved state-of-the-art results while keeping the number of learnable parameters low, preventing the model from over-fitting on the small labeled dataset.

Key Results

  • Overall Performance: The system achieved a weighted F1 score of 88.90% on the MER2024-SEMI test set.
  • Acoustic Gains: Fine-tuning HuBERT with adapters (HLFT 16-21) improved the F1 score from 72.82% to 80.24% on the validation set compared to the baseline.
  • Alignment Impact: Using the aligned visual features (CLIPL-A) instead of raw features (CLIPL) improved the test set F1 score by 3.64% in unimodal tests and contributed to higher multimodal accuracy.
  • Synergy: The fusion of all three modalities outperformed any single modality, proving that the alignment strategy successfully integrated complementary information.

Limitations and Future Work

While the paper demonstrates a robust pipeline, a few areas remain open:

  • Lexical Integration: The lexical features (from Baichuan2) were extracted from transcripts, but the paper doesn't explore deep alignment between text and audio/video in the same way it does for visual-acoustic alignment.
  • Computational Cost: While PEFT is efficient, the initial contrastive pre-training on 115k clips still requires significant compute.
  • Future Directions: The authors suggest that exploring different transformer backbones or more complex fusion mechanisms (like cross-modal transformers) could further push the boundaries of MER.

Summary updated

Abstract

Multimodal Emotion Recognition (MER) aims to automatically identify and understand human emotional states by integrating information from various modalities. However, the scarcity of annotated multimodal data significantly hinders the advancement of this research field. This paper presents our solution for the MER-SEMI sub-challenge of MER 2024. First, to better adapt acoustic modality features for the MER task, we experimentally evaluate the contributions of different layers of the pre-trained speech model HuBERT in emotion recognition. Based on these observations, we perform Parameter-Efficient Fine-Tuning (PEFT) on the layers identified as most effective for emotion recognition tasks, thereby achieving optimal adaptation for emotion recognition with a minimal number of learnable parameters. Second, leveraging the strengths of the acoustic modality, we propose a feature alignment pre-training method. This approach uses large-scale unlabeled data to train a visual encoder, thereby promoting the semantic alignment of visual features within the acoustic feature space. Finally, using the adapted acoustic features, aligned visual features, and lexical features, we employ an attention mechanism for feature fusion. On the MER2024-SEMI test set, the proposed method achieves a weighted F1 score of 88.90%, ranking fourth among all participating teams, validating the effectiveness of our approach.
BibTeX
@inproceedings{zhao2024improving,
  author = {Zhixian Zhao and Haifeng Chen and Xi Li and Dongmei Jiang and Lei Xie},
  title = {{Improving Multimodal Emotion Recognition by Leveraging Acoustic Adaptation and Visual Alignment}},
  booktitle = {MRAC@MM},
  year = {2024},
  doi = {10.1145/3689092.3689407},
}

Loading related content...

Figure