TL;DR
This paper introduces a novel approach to multimodal sentiment analysis by combining curriculum learning with weakly supervised modality correlation learning. By dynamically feeding training pairs based on their difficulty and discarding noisy examples, the method achieves state-of-the-art performance, overcoming limitations of random data sampling in existing self-supervised learning frameworks.
What Problem Does This Paper Address?
Multimodal sentiment analysis (MSA) aims to understand human emotions from various data sources like text, audio, and video. While self-supervised learning has shown promise by leveraging inherent correlations between modalities within a sample, a key challenge is the random order in which these correlated unimodal data pairs are presented to the model. This can lead to models getting stuck in suboptimal solutions, especially when noisy or difficult examples are encountered early in training. Without human annotation, these self-supervised tasks often generate noisy data, further exacerbating the problem. This paper addresses the need for a more structured and intelligent way to train multimodal models by introducing curriculum learning to guide the learning process.
Methods
The core innovation lies in the Weakly Supervised Correlation Learning with Curriculum Learning (WSCL-CL) framework.
- Weakly Supervised Modality Correlation Learning: This task aims to learn a discriminative embedding space by capturing shared information across modalities. It uses label information to score negative pairs (unimodal embeddings from different samples), making it "weakly supervised." The goal is to pull embeddings from the same sample closer and push those from different samples apart.
- Curriculum Learning: This component dynamically controls the order of training data presentation. It consists of two key functions:
- Difficulty Scoring Function: This function determines the "hardness" of training pairs. It leverages both pre-trained and current correlation predictors to calculate the loss associated with each pair. Pairs with higher losses are considered harder. Notably, the hardest pairs (assumed to be noisy) are discarded.
- Pair Feeding Function: This function adaptively controls the pace at which training pairs are fed to the model. It has three actions: 'stay' (continue with the current difficulty level), 'step forward' (move to more challenging pairs), or 'step backward' (revisit easier pairs). This decision is based on the difference in correlation losses between consecutive iterations, allowing for a more refined learning strategy.
Datasets
The proposed method was evaluated on two widely-used multimodal sentiment analysis datasets:
- CMU-MOSI: A collection of over 2,000 video clips with sentiment annotations on a [-3, 3] Likert scale.
- CMU-MOSEI: A larger dataset with over 20,000 utterances from more than 1,000 YouTube speakers, covering diverse topics.
Key Insights and Ideas
- Curriculum Learning is Crucial for Multimodal Self-Supervised Learning: Simply feeding correlated multimodal data randomly is suboptimal. A structured, difficulty-aware approach significantly improves performance.
- Discarding Noisy Pairs is Beneficial: The paper hypothesizes that the hardest pairs are often noisy and demonstrates that discarding them improves model performance. This is a key advantage over methods that don't explicitly handle noise.
- Adaptive Feeding Strategy: The 'stay', 'step forward', 'step backward' feeding mechanism allows the model to adapt to its learning progress, preventing it from getting stuck in local optima or oscillating too much.
- Weak Supervision for Correlation: Leveraging existing label information to score negative pairs provides a "weak supervision" signal that helps learn a more discriminative embedding space without requiring full manual annotation.
Key Results
The proposed WSCL-CL method achieved state-of-the-art performance on both CMU-MOSI and CMU-MOSEI datasets across various metrics, including Acc7, Acc2, F1 score, MAE, and Correlation. For instance, on CMU-MOSI, it outperformed the strong baseline HyCon by up to 1.1% on Acc2 and F1 score. On CMU-MOSEI, it achieved a 1.2% improvement in F1 score over HyCon. Ablation studies confirmed the significant contributions of both the correlation learning task and the curriculum learning components.
Limitations and Future Work
- Learned Predictor Imperfection: The authors acknowledge that the learned correlation predictor might not be a perfect scoring function, as its learning loss may not reach zero, potentially leading to misclassification of pair difficulty.
- Pre-training Time Complexity: The use of a pre-trained correlation predictor for initial difficulty scoring increases the overall training time.
- Future Work: The paper suggests that the difficulty scoring function could be improved. Additionally, the visualization of the choosing index
cifor different training pairs revealed that identifying modalities belonging to the same sample is easier than distinguishing modalities from different samples. This points to future work in improving how the model learns to correctly locate samples in the embedding space.
Summary updated