Curriculum Learning Meets Weakly Supervised Modality Correlation Learning

Sijie Mai, Ya Sun, Haifeng HuarXiv 2022 · DOI · OpenReview

TL;DR

This paper introduces a novel approach to multimodal sentiment analysis by combining curriculum learning with weakly supervised modality correlation learning. By dynamically feeding training pairs based on their difficulty and discarding noisy examples, the method achieves state-of-the-art performance, overcoming limitations of random data sampling in existing self-supervised learning frameworks.

What Problem Does This Paper Address?

Multimodal sentiment analysis (MSA) aims to understand human emotions from various data sources like text, audio, and video. While self-supervised learning has shown promise by leveraging inherent correlations between modalities within a sample, a key challenge is the random order in which these correlated unimodal data pairs are presented to the model. This can lead to models getting stuck in suboptimal solutions, especially when noisy or difficult examples are encountered early in training. Without human annotation, these self-supervised tasks often generate noisy data, further exacerbating the problem. This paper addresses the need for a more structured and intelligent way to train multimodal models by introducing curriculum learning to guide the learning process.

Methods

The core innovation lies in the Weakly Supervised Correlation Learning with Curriculum Learning (WSCL-CL) framework.

  • Weakly Supervised Modality Correlation Learning: This task aims to learn a discriminative embedding space by capturing shared information across modalities. It uses label information to score negative pairs (unimodal embeddings from different samples), making it "weakly supervised." The goal is to pull embeddings from the same sample closer and push those from different samples apart.
  • Curriculum Learning: This component dynamically controls the order of training data presentation. It consists of two key functions:
  • Difficulty Scoring Function: This function determines the "hardness" of training pairs. It leverages both pre-trained and current correlation predictors to calculate the loss associated with each pair. Pairs with higher losses are considered harder. Notably, the hardest pairs (assumed to be noisy) are discarded.
  • Pair Feeding Function: This function adaptively controls the pace at which training pairs are fed to the model. It has three actions: 'stay' (continue with the current difficulty level), 'step forward' (move to more challenging pairs), or 'step backward' (revisit easier pairs). This decision is based on the difference in correlation losses between consecutive iterations, allowing for a more refined learning strategy.

Datasets

The proposed method was evaluated on two widely-used multimodal sentiment analysis datasets:

  • CMU-MOSI: A collection of over 2,000 video clips with sentiment annotations on a [-3, 3] Likert scale.
  • CMU-MOSEI: A larger dataset with over 20,000 utterances from more than 1,000 YouTube speakers, covering diverse topics.

Key Insights and Ideas

  • Curriculum Learning is Crucial for Multimodal Self-Supervised Learning: Simply feeding correlated multimodal data randomly is suboptimal. A structured, difficulty-aware approach significantly improves performance.
  • Discarding Noisy Pairs is Beneficial: The paper hypothesizes that the hardest pairs are often noisy and demonstrates that discarding them improves model performance. This is a key advantage over methods that don't explicitly handle noise.
  • Adaptive Feeding Strategy: The 'stay', 'step forward', 'step backward' feeding mechanism allows the model to adapt to its learning progress, preventing it from getting stuck in local optima or oscillating too much.
  • Weak Supervision for Correlation: Leveraging existing label information to score negative pairs provides a "weak supervision" signal that helps learn a more discriminative embedding space without requiring full manual annotation.

Key Results

The proposed WSCL-CL method achieved state-of-the-art performance on both CMU-MOSI and CMU-MOSEI datasets across various metrics, including Acc7, Acc2, F1 score, MAE, and Correlation. For instance, on CMU-MOSI, it outperformed the strong baseline HyCon by up to 1.1% on Acc2 and F1 score. On CMU-MOSEI, it achieved a 1.2% improvement in F1 score over HyCon. Ablation studies confirmed the significant contributions of both the correlation learning task and the curriculum learning components.

Limitations and Future Work

  • Learned Predictor Imperfection: The authors acknowledge that the learned correlation predictor might not be a perfect scoring function, as its learning loss may not reach zero, potentially leading to misclassification of pair difficulty.
  • Pre-training Time Complexity: The use of a pre-trained correlation predictor for initial difficulty scoring increases the overall training time.
  • Future Work: The paper suggests that the difficulty scoring function could be improved. Additionally, the visualization of the choosing index ci for different training pairs revealed that identifying modalities belonging to the same sample is easier than distinguishing modalities from different samples. This points to future work in improving how the model learns to correctly locate samples in the embedding space.

Summary updated

Abstract

In the field of multimodal sentiment analysis (MSA), a few studies have leveraged the inherent modality correlation information stored in samples for self-supervised learning. However, they feed the training pairs in a random order without consideration of difficulty. Without human annotation, the generated training pairs of self-supervised learning often contain noise. If noisy or hard pairs are used for training at the easy stage, the model might be stuck in bad local optimum. In this paper, we inject curriculum learning into weakly supervised modality correlation learning. The weakly supervised correlation learning leverages the label information to generate scores for negative pairs to learn a more discriminative embedding space, where negative pairs are defined as two unimodal embeddings from different samples. To assist the correlation learning, we feed the training pairs to the model according to difficulty by the proposed curriculum learning, which consists of elaborately designed scoring and feeding functions. The scoring function computes the difficulty of pairs using pre-trained and current correlation predictors, where the pairs with large losses are defined as hard pairs. Notably, the hardest pairs are discarded in our algorithm, which are assumed as noisy pairs. Moreover, the feeding function takes the difference of correlation losses as feedback to determine the feeding actions (`stay', `step back', or `step forward'). The proposed method reaches state-of-the-art performance on MSA.
BibTeX
@misc{mai2022curriculum,
  author = {Sijie Mai and Ya Sun and Haifeng Hu},
  title = {{Curriculum Learning Meets Weakly Supervised Modality Correlation Learning}},
  year = {2022},
  eprint = {2212.07619},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.07619},
}
Figure