TL;DR
This paper introduces CLGSI, a novel multimodal sentiment analysis framework that leverages contrastive learning guided by sentiment intensity to better capture nuanced emotional expressions. By intelligently selecting and weighting sample pairs based on their sentiment differences, CLGSI achieves state-of-the-art performance on several benchmark datasets, demonstrating its effectiveness in understanding sentiment across different modalities and cultures.
What Problem Does This Paper Address?
Multimodal sentiment analysis (MSA) aims to understand human emotions by integrating information from various sources like text, video, and audio. While contrastive learning has shown promise in MSA, existing methods often treat all sample pairs equally, failing to account for the subtle differences in sentiment intensity. This can lead to misinterpretations, where samples with slightly different negative sentiments are treated the same, hindering a deeper understanding of emotional nuances. Furthermore, how to effectively fuse the representations learned through contrastive learning across different modalities remains a challenge.
Methods
CLGSI tackles these issues with two main innovations:
- Contrastive Learning Guided by Sentiment Intensity: Instead of simply using binary labels (positive/negative), CLGSI considers the difference in sentiment intensity between sample pairs. It assigns weights to these pairs, giving more importance to pairs with larger sentiment discrepancies. This fine-grained approach enriches the contrastive learning process.
- Global-Local-Fine-Knowledge (GLFK) Fusion Mechanism: This novel mechanism mimics human cognitive processes for understanding information. It first extracts common features across modalities (Global, Local, Fine) and then consolidates this knowledge (Knowledge) to create a comprehensive representation. Simultaneously, a Subnet module extracts modality-specific features. These common and specific features are then jointly learned to predict sentiment intensity.
Datasets
The framework was evaluated on three widely used datasets:
- MOSI: An English dataset with text, video, and audio modalities, featuring sentiment scores from -3 (strongly negative) to +3 (strongly positive).
- MOSEI: A larger English dataset, also with text, video, and audio, covering a wider range of topics and sentiment scores.
- SIMS: A Chinese multimodal dataset with text, video, and audio, using sentiment scores from -1 (strongly negative) to +1 (strongly positive).
Key Insights and Ideas
- Sentiment Intensity Matters: The paper strongly advocates for moving beyond simple positive/negative labels in contrastive learning for MSA. Recognizing and weighting sentiment intensity differences is crucial for capturing subtle emotional nuances.
- Human-like Information Processing: The GLFK fusion mechanism, inspired by how humans read and process information (abstract first, then details), proves effective in integrating multimodal data. This suggests that mimicking cognitive processes can lead to more robust feature fusion.
- Generalization Across Cultures: CLGSI demonstrates strong performance on both English and Chinese datasets, indicating its ability to generalize across different cultural expressions of sentiment.
Key Results
CLGSI achieved state-of-the-art results on the MOSI and MOSEI datasets, outperforming several strong baselines in various metrics, including Acc-2, F1, Acc-7, and MAE. For instance, on MOSI, CLGSI achieved Acc-7 surpassing baselines by at least 1.36%. On MOSEI, it improved Acc-2 and F1 by at least 0.5% over other methods. While it slightly underperformed ConFEDE on the SIMS dataset, this was attributed to ConFEDE's use of additional unimodal labels for pre-training, a resource not available for MOSI/MOSEI. CLGSI's performance on MOSI and MOSEI without such pre-training highlights its robust generalization capabilities.
Limitations and Future Work
The authors acknowledge that CLGSI's contrastive learning approach, guided by sentiment intensity, might not be directly applicable to multimodal emotion recognition (MER) tasks where labels are discrete emotions rather than continuous intensity scores. Future work could explore mechanisms to adapt CLGSI for MER. Additionally, the paper suggests further investigation into the usability of this approach with large language models (LLMs).
Summary updated