Hierarchical graph contrastive learning of local and global presentation for multimodal sentiment analysis

Jun Du, Jianhang Jin, Jian Zhuang, Cheng ZhangScientific 2024 · DOI

TL;DR

This paper introduces a novel Hierarchical Graph Contrastive Learning (HGCL-LG) framework for multimodal sentiment analysis (MSA). By constructing both local and global graphs from multimodal data and employing contrastive learning at multiple levels, the model effectively captures both within-utterance and across-utterance sentiment patterns, leading to improved performance.

What Problem Does This Paper Address?

Multimodal sentiment analysis (MSA) aims to understand human emotions expressed through a combination of text, audio, and visual cues. While existing methods excel at learning representations within a single utterance, they often overlook the crucial "global co-occurrence characteristics" of the entire dataset. This means they might miss how certain patterns or sentiment expressions tend to appear together across many different utterances, which can be vital for accurate sentiment prediction. The paper addresses this gap by proposing a framework that explicitly models both local (within-utterance) and global (across-utterance) sentiment information.

Methods

The core of the proposed HGCL-LG framework lies in its hierarchical graph construction and contrastive learning approach.

  1. Graph Construction:
    • Local Graph: For each utterance, a "local graph" is built using the raw multimodal sequence features. This graph captures the intricate relationships within that specific utterance.
  2. Global Graph: To capture dataset-wide patterns, a "global graph" is constructed. This is achieved by first using a discrete Variational Autoencoder (dVAE) to learn a shared embedding space that encodes global co-occurrence features across the dataset. Then, each modality's sequence features are mapped to this embedding space to form the global graph.
  3. Hierarchical Graph Contrastive Learning:
    • Local-Level Contrastive Learning: This strategy uses graph augmentations to create different views of the local graph. Contrastive loss is applied at both the node and graph levels to learn invariant representations.
  4. Global-Level Contrastive Learning: Similar to local-level learning, this applies contrastive loss to augmented global graphs to capture global sentiment patterns.
  5. Cross-Level Contrastive Learning: This novel component aims to bridge the gap between local and global representations. It uses a shared projection network to make the local and global representations of the same utterance closer in the learned embedding space.
  6. Automatic Graph Augmentation: To enhance the graph contrastive learning, an automatic graph augmentation strategy is introduced. This strategy adaptively samples augmentation operations (like node dropping or masking) for each node, creating diverse yet semantically similar graph structures for contrastive learning.
  7. Fusion and Sentiment Prediction: Finally, the learned local and global representations are concatenated and fed into a classifier to predict the final sentiment intensity.

Datasets

The proposed HGCL-LG framework is evaluated on two widely used multimodal sentiment analysis datasets:

  • CMU-MOSI: Contains 2199 short monologue video clips from YouTube movie reviews, with sentiment scores ranging from -3 (strongly negative) to +3 (strongly positive).
  • CMU-MOSEI: An enlarged version of CMU-MOSI, featuring 16,326 utterances for training, 1871 for validation, and 4659 for testing.

Key Insights and Ideas

  • Global Context is Crucial: The paper strongly emphasizes that considering global co-occurrence patterns across the entire dataset, not just within individual utterances, significantly boosts MSA performance. This is a key departure from many prior works.
  • Hierarchical Graph Representation: Building both local and global graphs allows the model to capture sentiment at different granularities, from fine-grained intra-utterance interactions to broader inter-utterance relationships.
  • Cross-Level Learning Bridges the Gap: The cross-level contrastive learning is a critical innovation, effectively aligning local and global representations and ensuring they contribute synergistically to sentiment prediction.
  • Adaptive Graph Augmentation: The introduction of an automatic, adaptive graph augmentation strategy is a novel contribution to MSA, allowing for more robust and effective graph contrastive learning.

Key Results

The HGCL-LG model demonstrates superior performance on both the CMU-MOSI and CMU-MOSEI datasets compared to several state-of-the-art baselines. For instance, on the CMU-MOSI dataset, the proposed model achieved an Acc7 of 41.7, Acc2 of 81.8, F1 of 81.7, MAE of 0.752, and Corr of 0.784. These results are highlighted as the best among the tested models. The paper also shows that the model performs well on both aligned and unaligned datasets, though slightly better on aligned ones. Ablation studies confirm the importance of each component of the HGCL-LG framework, with the removal of any module leading to a performance decline.

Limitations and Future Work

  • Unaligned Datasets: While the model performs well on unaligned datasets, the paper notes that results are slightly worse than on aligned datasets, suggesting that explicitly modeling aligned data could further improve performance.
  • Edge Construction: The study found that the "V→TA" edge construction method was the most effective, implying that other edge types might introduce noise. Future work could explore more sophisticated edge construction strategies.
  • Generalizability: While tested on two benchmark datasets, further evaluation on a wider variety of multimodal sentiment analysis tasks and datasets would be beneficial to assess the broader applicability of the HGCL-LG framework.

Summary updated

Abstract

Multi-modal sentiment analysis (MSA) aims to regress or classify the overall sentiment of utterances through acoustic, visual, and textual cues. However, most of the existing efforts have focused on developing the expressive ability of neural networks to learn the representation of multi-modal information within a single utterance, without considering the global co-occurrence characteristics of the dataset. To alleviate the above issue, in this paper, we propose a novel hierarchical graph contrastive learning framework for MSA, aiming to explore the local and global representations of a single utterance for multimodal sentiment extraction and the intricate relations between them. Specifically, regarding to each modality, we extract the discrete embedding representation of each modality, which includes the global co-occurrence features of each modality. Based on it, for each utterance, we build two graphs: local level graph and global level graph to account for the level-specific sentiment implications. Then, two graph contrastive learning strategies is adopted to explore the different potential presentations based on graph augmentations respectively. Furthermore, we design a cross-level comparative learning for learning local and global potential representations of complex relationships.
BibTeX
@inproceedings{du2024hierarchical,
  author = {Jun Du and Jianhang Jin and Jian Zhuang and Cheng Zhang},
  title = {{Hierarchical graph contrastive learning of local and global presentation for multimodal sentiment analysis}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-54872-6},
}
Figure