TL;DR
This paper introduces a novel Hierarchical Graph Contrastive Learning (HGCL-LG) framework for multimodal sentiment analysis (MSA). By constructing both local and global graphs from multimodal data and employing contrastive learning at multiple levels, the model effectively captures both within-utterance and across-utterance sentiment patterns, leading to improved performance.
What Problem Does This Paper Address?
Multimodal sentiment analysis (MSA) aims to understand human emotions expressed through a combination of text, audio, and visual cues. While existing methods excel at learning representations within a single utterance, they often overlook the crucial "global co-occurrence characteristics" of the entire dataset. This means they might miss how certain patterns or sentiment expressions tend to appear together across many different utterances, which can be vital for accurate sentiment prediction. The paper addresses this gap by proposing a framework that explicitly models both local (within-utterance) and global (across-utterance) sentiment information.
Methods
The core of the proposed HGCL-LG framework lies in its hierarchical graph construction and contrastive learning approach.
- Graph Construction:
- Local Graph: For each utterance, a "local graph" is built using the raw multimodal sequence features. This graph captures the intricate relationships within that specific utterance.
- Global Graph: To capture dataset-wide patterns, a "global graph" is constructed. This is achieved by first using a discrete Variational Autoencoder (dVAE) to learn a shared embedding space that encodes global co-occurrence features across the dataset. Then, each modality's sequence features are mapped to this embedding space to form the global graph.
- Hierarchical Graph Contrastive Learning:
- Local-Level Contrastive Learning: This strategy uses graph augmentations to create different views of the local graph. Contrastive loss is applied at both the node and graph levels to learn invariant representations.
- Global-Level Contrastive Learning: Similar to local-level learning, this applies contrastive loss to augmented global graphs to capture global sentiment patterns.
- Cross-Level Contrastive Learning: This novel component aims to bridge the gap between local and global representations. It uses a shared projection network to make the local and global representations of the same utterance closer in the learned embedding space.
- Automatic Graph Augmentation: To enhance the graph contrastive learning, an automatic graph augmentation strategy is introduced. This strategy adaptively samples augmentation operations (like node dropping or masking) for each node, creating diverse yet semantically similar graph structures for contrastive learning.
- Fusion and Sentiment Prediction: Finally, the learned local and global representations are concatenated and fed into a classifier to predict the final sentiment intensity.
Datasets
The proposed HGCL-LG framework is evaluated on two widely used multimodal sentiment analysis datasets:
- CMU-MOSI: Contains 2199 short monologue video clips from YouTube movie reviews, with sentiment scores ranging from -3 (strongly negative) to +3 (strongly positive).
- CMU-MOSEI: An enlarged version of CMU-MOSI, featuring 16,326 utterances for training, 1871 for validation, and 4659 for testing.
Key Insights and Ideas
- Global Context is Crucial: The paper strongly emphasizes that considering global co-occurrence patterns across the entire dataset, not just within individual utterances, significantly boosts MSA performance. This is a key departure from many prior works.
- Hierarchical Graph Representation: Building both local and global graphs allows the model to capture sentiment at different granularities, from fine-grained intra-utterance interactions to broader inter-utterance relationships.
- Cross-Level Learning Bridges the Gap: The cross-level contrastive learning is a critical innovation, effectively aligning local and global representations and ensuring they contribute synergistically to sentiment prediction.
- Adaptive Graph Augmentation: The introduction of an automatic, adaptive graph augmentation strategy is a novel contribution to MSA, allowing for more robust and effective graph contrastive learning.
Key Results
The HGCL-LG model demonstrates superior performance on both the CMU-MOSI and CMU-MOSEI datasets compared to several state-of-the-art baselines. For instance, on the CMU-MOSI dataset, the proposed model achieved an Acc7 of 41.7, Acc2 of 81.8, F1 of 81.7, MAE of 0.752, and Corr of 0.784. These results are highlighted as the best among the tested models. The paper also shows that the model performs well on both aligned and unaligned datasets, though slightly better on aligned ones. Ablation studies confirm the importance of each component of the HGCL-LG framework, with the removal of any module leading to a performance decline.
Limitations and Future Work
- Unaligned Datasets: While the model performs well on unaligned datasets, the paper notes that results are slightly worse than on aligned datasets, suggesting that explicitly modeling aligned data could further improve performance.
- Edge Construction: The study found that the "V→TA" edge construction method was the most effective, implying that other edge types might introduce noise. Future work could explore more sophisticated edge construction strategies.
- Generalizability: While tested on two benchmark datasets, further evaluation on a wider variety of multimodal sentiment analysis tasks and datasets would be beneficial to assess the broader applicability of the HGCL-LG framework.
Summary updated