Abstract
Multimodal Sentiment Analysis (MSA) often struggles with noisy data and suboptimal training orders. This paper introduces WSCL-CL, a framework combining Weakly Supervised Correlation Learning with an adaptive Curriculum Learning strategy. By sorting training pairs by difficulty and employing a dynamic feeding function that filters noise, the model achieves state-of-the-art performance on the CMU-MOSI and CMU-MOSEI benchmarks.
Introduction
Understanding human emotion requires the integration of language, audio, and visual signals. However, existing self-supervised methods face two primary challenges. First, random feeding of training data can lead models to get stuck in local optima if they encounter complex, ambiguous samples (like sarcasm) too early. Second, noisy data—where modalities from the same sample are contradictory or modalities from different samples are coincidentally similar—can degrade feature extraction.
Methodology: WSCL-CL
The authors propose a dual-component architecture to address these issues: Weakly Supervised Correlation Learning (WSCL) and a Curriculum Learning (CL) strategy.
Weakly Supervised Correlation Learning (WSCL)
Rather than binary matching (same vs. different person), WSCL utilizes sentiment labels to generate nuanced correlation scores. Modalities from different individuals expressing similar emotional intensities (e.g., "extreme joy") are treated as more correlated than those with opposing sentiments. This approach creates a more discriminative and semantically meaningful embedding space.
The Curriculum Learning Strategy
The core innovation is a "lesson plan" that organizes training data from easy to hard.
- Difficulty Scoring: Training pairs are ranked based on loss values derived from both a pre-trained "teacher" model and the active "student" model.
- Adaptive Feeding Function: The system dynamically adjusts the training set using three actions:
- Stay: Continue at the current difficulty level if learning progress is high.
- Step Forward: Advance to harder examples once performance stabilizes.
- Step Back: Revert to easier examples if performance drops, allowing the model to "regroup."
- Noise Filtering: To prevent "poisoning" the model, the hardest 5% of pairs—likely representing noise or misaligned data—are discarded entirely.
Experimental Results
The framework was evaluated on two major benchmarks: CMU-MOSI (~2,000 clips) and CMU-MOSEI (>20,000 utterances).
- State-of-the-Art Performance: WSCL-CL outperformed previous leaders like HyCon. On CMU-MOSI, it achieved a 0.9% improvement in 7-way accuracy and a 1.1% boost in binary accuracy. On CMU-MOSEI, it saw a 1.2% increase in F1 score.
- Ablation Insights: Removing the CL component resulted in a ~1.4% drop in accuracy, highlighting that the order of data presentation is critical.
- Modality Asymmetry: Analysis revealed that correlating language and audio (L+A) is significantly harder than other pairs, likely due to frequent misalignments between literal text and vocal tone.
Conclusion
WSCL-CL demonstrates that an adaptive, difficulty-aware training regimen significantly improves multimodal sentiment analysis. While the method introduces some computational overhead and depends on a pre-trained teacher model, its ability to filter noise and navigate complex data distributions sets a new standard for the field. Future work may explore applying this curriculum strategy to broader tasks like conversational emotion recognition.
Summary updated