TL;DR
This paper introduces EASUM, a novel two-phase training scheme that bridges the gap between sentiment analysis and emotion recognition by jointly modeling both. By first training a domain-general model and then using its outputs as pseudo-labels for a domain-specific model, EASUM significantly improves performance on multimodal affective state understanding tasks, achieving state-of-the-art results on several benchmarks while using fewer parameters.
What Problem Does This Paper Address?
Understanding human affective states is crucial, but traditional approaches often treat sentiment analysis (e.g., positive/negative) and emotion recognition (e.g., happy/sad/angry) as separate tasks. This is problematic because sentiment and emotion are deeply intertwined; a positive sentiment can stem from joy, and anger can lead to a negative sentiment. Relying on only one modality (like text) or treating these tasks independently can lead to incomplete or suboptimal understanding of a person's true affective state. While some work has explored combining these tasks, it's often hindered by the lack of datasets with both sentiment and emotion annotations, which are labor-intensive to create. This paper tackles this by proposing a method to leverage existing datasets for both tasks to improve performance on each, even when they lack joint annotations.
Methods
EASUM employs a two-phase training strategy:
- Phase 1: Domain General (DG) Model Training: The core idea here is to train a single model that can generalize across different datasets and tasks (MSA and MER). This is achieved by:
- Data Alignment: Using four benchmark datasets (CMU-MOSI, CMU-MOSEI for MSA; MELD, IEMOCAP for MER) and employing a domain alignment technique. Specifically, they use Moment Matching to minimize the distribution discrepancies between the feature representations from different datasets.
- Feature Extraction & Fusion: Standard feature extractors for audio (librosa) and video (efficientNet) are used, along with text embeddings. A Multimodal Adaptation Gate (MAG) is used to fuse these modalities, allowing acoustic and visual information to influence the textual representation. The fused features are then processed by a ConvBERT encoder to learn a multimodal representation.
- Classification: Two separate "fully connected" (FC) layers are used to predict sentiment (for MSA datasets) and emotion (for MER datasets) from the domain-invariant features. The objective function includes both classification loss and the domain alignment loss (weighted by
w). - Phase 2: Domain Specific (DS) Model Training: In this phase, the focus shifts to improving performance on each specific task (MSA or MER).
- Two-Stream Structure: A DG stream (using the frozen, pretrained DG model from Phase 1) generates soft pseudo-labels for the task that is not annotated in the dataset. For example, for CMU-MOSI (which has sentiment labels but not emotion labels), the DG stream generates emotion pseudo-labels.
- Auxiliary Supervision: These soft pseudo-labels are then used as auxiliary supervision during the training of a domain-specific model. The DS model uses DeBERTa (an advanced BERT variant) for multimodal representation learning.
- Objective Function: The DS model's training objective combines the primary task loss (e.g., MSE for MSA, cross-entropy for MER) with a KL divergence loss that encourages the DS model's predictions to match the soft pseudo-labels.
Datasets
The paper utilizes four publicly available multimodal datasets:
- CMU-MOSI: Primarily for sentiment analysis (sentiment intensity).
- CMU-MOSEI: For sentiment analysis (sentiment intensity) and emotion recognition (six discrete emotions). The authors only use sentiment for this dataset.
- MELD: For emotion recognition (seven emotion classes) and sentiment polarity. The authors only use emotion labels.
- IEMOCAP: For emotion recognition (six emotion labels) and dimensional labels (valence, activation, dominance). The authors only use emotion labels.
The authors note that MELD and IEMOCAP have incongruent emotion label spaces, which they merge into a common 9-emotion category space for the MER task.
Key Insights and Ideas
- Sentiment and Emotion are Synergistic: The core premise is that sentiment and emotion are deeply linked, and jointly modeling them can lead to a richer understanding of affective states than treating them separately.
- Domain Generalization as a Bridge: By training a single model that can generalize across diverse MSA and MER datasets, the authors effectively bridge the "domain gap" and learn robust, invariant features. This is a novel application of domain generalization in this specific multimodal affective computing context.
- Pseudo-Labels as Auxiliary Teachers: Using the DG model to generate soft pseudo-labels for the missing annotation type (e.g., emotion for sentiment-only datasets) provides valuable, albeit imperfect, supervision. This allows the DS model to learn from information it wouldn't otherwise have access to.
- Soft Pseudo-Labels are Key: The use of "soft" pseudo-labels (probability distributions over classes) rather than hard labels is crucial. This provides a more nuanced signal and can reduce biases associated with specific datasets, leading to better generalization.
- Visual Modality's Strong Impact: An ablation study suggests that the visual modality contributes more to performance gains than the acoustic modality when combined with text.
Key Results
- State-of-the-Art Performance: EASUM achieves new state-of-the-art (SOTA) results on CMU-MOSI and MELD datasets across multiple metrics. It also performs comparably to, and sometimes surpasses, SOTA on CMU-MOSEI and IEMOCAP.
- Efficiency: The proposed method achieves these results while using approximately 40% fewer parameters compared to previous SOTA models (e.g., UniMSE, which uses a T5-base model).
- Pseudo-Label Quality: Qualitative analysis shows that the generated soft pseudo-labels are generally in parallel with the ground truth annotations, demonstrating their adequacy and reliability.
- Ablation Studies: Confirm the importance of incorporating multiple modalities (text, audio, visual) and the effectiveness of the soft pseudo-labeling strategy.
Limitations and Future Work
- Dataset Limitations: The authors acknowledge that the availability of datasets with joint sentiment and emotion annotations is limited, which is a primary motivation for their approach.
- Parameter Efficiency: While the model is more parameter-efficient than some baselines, the paper doesn't explicitly detail the exact parameter count for EASUM itself, only comparing it to models like T5-base.
- Future Work: The paper suggests that the EASUM training scheme is viable for other sentiment and emotion datasets and can be expanded to incorporate more datasets, implying a generalizable framework. Further research could explore more sophisticated domain alignment techniques or different pseudo-labeling strategies.
Summary updated