Introduction
Multimodal affect recognition—the identification of human emotions, sentiments, or sarcasm through text, audio, and video—is a fundamental challenge in natural language processing and computer vision. Despite its importance, the field is consistently hindered by data scarcity. Annotating multimodal data is a resource-intensive process, resulting in datasets that are often too small to train complex state-of-the-art (SOTA) models effectively. This leads to two primary issues: significant overfitting, where models memorize small training sets instead of learning generalizable features, and extreme instability, where performance varies wildly based on random initialization.
Proposed Methodology
To address these challenges, researchers from HKUST proposed a framework that combines a novel fusion architecture with a weakly-supervised multi-task learning (MTL) strategy.
Hybrid Early-Late Fusion (EF-LF)
Traditional multimodal models typically employ either Early Fusion (concatenating raw features at the input level) or Late Fusion (fusing high-level representations at the output level). The authors introduce a Hybrid Fusion model that integrates both approaches. This architecture allows the system to simultaneously capture low-level cross-modal interactions and high-level semantic features, providing a more robust representation of the affective state.
Weakly-supervised Multi-task Learning (MTL)
The core innovation lies in leveraging "weak labels" to augment small datasets. Since most benchmarks only provide labels for a single task (e.g., sarcasm), the authors utilize external models trained on related source tasks (like sentiment analysis) to generate labels for the target dataset.
- Weak Label Acquisition: A model trained on a large source task (e.g., CMU-MOSEI for sentiment) predicts labels for a target dataset (e.g., MUStARD for sarcasm).
- Joint Training: A backbone model is then trained to solve multiple tasks simultaneously using a mix of "strong" (human-annotated) and "weak" (model-generated) labels. This multi-tasking acts as a powerful regularizer, forcing the model to learn shared, generalizable representations across different affective domains.
Experimental Setup and Datasets
The framework was evaluated on three primary benchmarks:
- IEMOCAP: Focused on multimodal emotion recognition (Neutral, Happy, Sad, Angry).
- CMU-MOSEI: A large-scale dataset used for both sentiment and emotion analysis.
- MUStARD: A specialized dataset for audiovisual sarcasm discovery.
Key Insights
The study revealed several critical findings regarding the current state of multimodal research:
- Complexity vs. Performance: Benchmarking 12 different models showed that complex SOTA architectures often underperform compared to simple baselines on small datasets. The proposed "naive" hybrid fusion baseline frequently outperformed more sophisticated alternatives.
- The Reliability Gap: The authors identified a "random seed trap," where changing the seed could alter accuracy by up to 4.7%, highlighting the instability of models trained on limited data.
- Effectiveness of Weak Supervision: Weak labels provided performance gains nearly identical to human-annotated labels, provided the auxiliary tasks were sufficiently correlated (e.g., emotion and sentiment).
- Task Correlation: Analysis confirmed that sentiment has a higher correlation with emotion than with sarcasm, explaining why emotion-based auxiliary tasks are particularly effective for sentiment models.
Results and Impact
The MTL approach yielded significant improvements:
- Accuracy & F1-Score: Sarcasm recognition saw an accuracy increase of up to 2.9% and an F1-score boost of 3.3%.
- Model Stability: The variance in results across different random seeds was significantly reduced. In emotion recognition tasks, the standard deviation dropped by 0.4%, indicating a more reliable and reproducible model.
Conclusion and Future Directions
While weakly-supervised MTL is a powerful tool for overcoming data scarcity, the authors caution that the choice of auxiliary tasks is crucial. Tasks that are too distant from the target can introduce noise rather than helpful signals. Future research will focus on dynamic weighting of weak labels based on model confidence and extending the framework to other affective tasks such as humor or stress detection.
Summary updated