Weakly-supervised Multi-task Learning for Multimodal Affect Recognition

Wenliang Dai, Samuel Cahyawijaya, Yejin Bang, Pascale FungarXiv 2021 · arXiv

Introduction

Multimodal affect recognition—the identification of human emotions, sentiments, or sarcasm through text, audio, and video—is a fundamental challenge in natural language processing and computer vision. Despite its importance, the field is consistently hindered by data scarcity. Annotating multimodal data is a resource-intensive process, resulting in datasets that are often too small to train complex state-of-the-art (SOTA) models effectively. This leads to two primary issues: significant overfitting, where models memorize small training sets instead of learning generalizable features, and extreme instability, where performance varies wildly based on random initialization.

Proposed Methodology

To address these challenges, researchers from HKUST proposed a framework that combines a novel fusion architecture with a weakly-supervised multi-task learning (MTL) strategy.

Hybrid Early-Late Fusion (EF-LF)

Traditional multimodal models typically employ either Early Fusion (concatenating raw features at the input level) or Late Fusion (fusing high-level representations at the output level). The authors introduce a Hybrid Fusion model that integrates both approaches. This architecture allows the system to simultaneously capture low-level cross-modal interactions and high-level semantic features, providing a more robust representation of the affective state.

Comparison of Early, Late, and Hybrid fusion architectures.

Weakly-supervised Multi-task Learning (MTL)

The core innovation lies in leveraging "weak labels" to augment small datasets. Since most benchmarks only provide labels for a single task (e.g., sarcasm), the authors utilize external models trained on related source tasks (like sentiment analysis) to generate labels for the target dataset.

  1. Weak Label Acquisition: A model trained on a large source task (e.g., CMU-MOSEI for sentiment) predicts labels for a target dataset (e.g., MUStARD for sarcasm).
  2. Joint Training: A backbone model is then trained to solve multiple tasks simultaneously using a mix of "strong" (human-annotated) and "weak" (model-generated) labels. This multi-tasking acts as a powerful regularizer, forcing the model to learn shared, generalizable representations across different affective domains.
Multimodal affect recognition framework leveraging auxiliary tasks.

Experimental Setup and Datasets

The framework was evaluated on three primary benchmarks:

  • IEMOCAP: Focused on multimodal emotion recognition (Neutral, Happy, Sad, Angry).
  • CMU-MOSEI: A large-scale dataset used for both sentiment and emotion analysis.
  • MUStARD: A specialized dataset for audiovisual sarcasm discovery.

Key Insights

The study revealed several critical findings regarding the current state of multimodal research:

  • Complexity vs. Performance: Benchmarking 12 different models showed that complex SOTA architectures often underperform compared to simple baselines on small datasets. The proposed "naive" hybrid fusion baseline frequently outperformed more sophisticated alternatives.
  • The Reliability Gap: The authors identified a "random seed trap," where changing the seed could alter accuracy by up to 4.7%, highlighting the instability of models trained on limited data.
  • Effectiveness of Weak Supervision: Weak labels provided performance gains nearly identical to human-annotated labels, provided the auxiliary tasks were sufficiently correlated (e.g., emotion and sentiment).
  • Task Correlation: Analysis confirmed that sentiment has a higher correlation with emotion than with sarcasm, explaining why emotion-based auxiliary tasks are particularly effective for sentiment models.

Results and Impact

The MTL approach yielded significant improvements:

  • Accuracy & F1-Score: Sarcasm recognition saw an accuracy increase of up to 2.9% and an F1-score boost of 3.3%.
  • Model Stability: The variance in results across different random seeds was significantly reduced. In emotion recognition tasks, the standard deviation dropped by 0.4%, indicating a more reliable and reproducible model.

Conclusion and Future Directions

While weakly-supervised MTL is a powerful tool for overcoming data scarcity, the authors caution that the choice of auxiliary tasks is crucial. Tasks that are too distant from the target can introduce noise rather than helpful signals. Future research will focus on dynamic weighting of weak labels based on model confidence and extending the framework to other affective tasks such as humor or stress detection.

Summary updated

Abstract

Multimodal affect recognition constitutes an important aspect for enhancing interpersonal relationships in human-computer interaction. However, relevant data is hard to come by and notably costly to annotate, which poses a challenging barrier to build robust multimodal affect recognition systems. Models trained on these relatively small datasets tend to overfit and the improvement gained by using complex state-of-the-art models is marginal compared to simple baselines. Meanwhile, there are many different multimodal affect recognition datasets, though each may be small. In this paper, we propose to leverage these datasets using weakly-supervised multi-task learning to improve the generalization performance on each of them. Specifically, we explore three multimodal affect recognition tasks: 1) emotion recognition; 2) sentiment analysis; and 3) sarcasm recognition. Our experimental results show that multi-tasking can benefit all these tasks, achieving an improvement up to 2.9% accuracy and 3.3% F1-score. Furthermore, our method also helps to improve the stability of model performance. In addition, our analysis suggests that weak supervision can provide a comparable contribution to strong supervision if the tasks are highly correlated.
BibTeX
@misc{dai2021weakly,
  author = {Wenliang Dai and Samuel Cahyawijaya and Yejin Bang and Pascale Fung},
  title = {{Weakly-supervised Multi-task Learning for Multimodal Affect Recognition}},
  year = {2021},
  eprint = {2104.11560},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.11560},
}
Figure