Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning

Sijie Mai, Ya Sun, Haifeng Hu

Abstract

Multimodal Sentiment Analysis (MSA) often struggles with noisy data and suboptimal training orders. This paper introduces WSCL-CL, a framework combining Weakly Supervised Correlation Learning with an adaptive Curriculum Learning strategy. By sorting training pairs by difficulty and employing a dynamic feeding function that filters noise, the model achieves state-of-the-art performance on the CMU-MOSI and CMU-MOSEI benchmarks.

Introduction

Understanding human emotion requires the integration of language, audio, and visual signals. However, existing self-supervised methods face two primary challenges. First, random feeding of training data can lead models to get stuck in local optima if they encounter complex, ambiguous samples (like sarcasm) too early. Second, noisy data—where modalities from the same sample are contradictory or modalities from different samples are coincidentally similar—can degrade feature extraction.

Methodology: WSCL-CL

The authors propose a dual-component architecture to address these issues: Weakly Supervised Correlation Learning (WSCL) and a Curriculum Learning (CL) strategy.

Figure 2: The WSCL-CL architecture, showing the integration of multimodal fusion with a curriculum-driven correlation learning path.

Weakly Supervised Correlation Learning (WSCL)

Rather than binary matching (same vs. different person), WSCL utilizes sentiment labels to generate nuanced correlation scores. Modalities from different individuals expressing similar emotional intensities (e.g., "extreme joy") are treated as more correlated than those with opposing sentiments. This approach creates a more discriminative and semantically meaningful embedding space.

The Curriculum Learning Strategy

The core innovation is a "lesson plan" that organizes training data from easy to hard.

Figure 1: Visualization of the curriculum learning process, where training pairs are sorted by difficulty and selected adaptively over time.
  1. Difficulty Scoring: Training pairs are ranked based on loss values derived from both a pre-trained "teacher" model and the active "student" model.
  2. Adaptive Feeding Function: The system dynamically adjusts the training set using three actions:
  3. Stay: Continue at the current difficulty level if learning progress is high.
  4. Step Forward: Advance to harder examples once performance stabilizes.
  5. Step Back: Revert to easier examples if performance drops, allowing the model to "regroup."
  6. Noise Filtering: To prevent "poisoning" the model, the hardest 5% of pairs—likely representing noise or misaligned data—are discarded entirely.

Experimental Results

The framework was evaluated on two major benchmarks: CMU-MOSI (~2,000 clips) and CMU-MOSEI (>20,000 utterances).

  • State-of-the-Art Performance: WSCL-CL outperformed previous leaders like HyCon. On CMU-MOSI, it achieved a 0.9% improvement in 7-way accuracy and a 1.1% boost in binary accuracy. On CMU-MOSEI, it saw a 1.2% increase in F1 score.
  • Ablation Insights: Removing the CL component resulted in a ~1.4% drop in accuracy, highlighting that the order of data presentation is critical.
  • Modality Asymmetry: Analysis revealed that correlating language and audio (L+A) is significantly harder than other pairs, likely due to frequent misalignments between literal text and vocal tone.

Conclusion

WSCL-CL demonstrates that an adaptive, difficulty-aware training regimen significantly improves multimodal sentiment analysis. While the method introduces some computational overhead and depends on a pre-trained teacher model, its ability to filter noise and navigate complex data distributions sets a new standard for the field. Future work may explore applying this curriculum strategy to broader tasks like conversational emotion recognition.

Summary updated

Abstract

In the field of multimodal sentiment analysis (MSA), a few studies have leveraged the inherent modality correlation information stored in samples for self-supervised learning. However, they feed the training pairs in a random order without consideration of difficulty. Without human annotation, the generated training pairs of self-supervised learning often contain noise. If noisy or hard pairs are used for training at the easy stage, the model might be stuck in bad local optimum. In this paper, we inject curriculum learning into weakly supervised multimodal correlation learning. The weakly supervised correlation learning leverages the label information to generate scores for negative pairs to learn a more discriminative embedding space, where negative pairs are defined as two unimodal embeddings from different samples. To assist the correlation learning, we feed the training pairs to the model according to difficulty by the proposed curriculum learning, which consists of elaborately designed scoring and feeding functions. The scoring function computes the difficulty of pairs using pre-trained and current correlation predictors, where the pairs with large losses are defined as hard pairs. Notably, the hardest pairs are discarded in our algorithm, which are assumed as noisy pairs. Moreover, the feeding function takes the difference of correlation losses as feedback to determine the feeding actions (‘stay’, ‘step back’, or ‘step forward’). The proposed method reaches state-of-the-art performance on MSA.
BibTeX
@inproceedings{mai2022curriculum,
  author = {Sijie Mai and Ya Sun and Haifeng Hu},
  title = {{Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.209},
}
Figure