G^2SAM: Graph-Based Global Semantic Awareness Method for Multimodal Sarcasm Detection

Yiwei Wei, Shaozu Yuan, Hengyang Zhou, Longbiao Wang, Zhiling Yan, Ruosong Yang, Meng ChenAAAI 2024 · DOI

Abstract

Multimodal sarcasm detection is a challenging task that requires capturing the subtle incongruity between textual and visual information. While existing methods focus on local incongruity within a single post, they often neglect global semantic patterns shared across different sarcastic instances. This paper introduces $G^2SAM$ (Graph-Based Global Semantic Awareness Method), which enhances sarcasm detection by incorporating a "global" perspective. $G^2SAM$ utilizes a fine-grained graph-aligned model to represent multimodal inputs and a $k$-Nearest Neighbor ($k$NN) retrieval strategy to leverage a library of known examples. To optimize this retrieval, we propose Label-aware Graph Contrastive Learning (LGCL) to structure the semantic space. Experimental results on the HFM benchmark demonstrate that $G^2SAM$ achieves state-of-the-art performance, highlighting the importance of global semantic congruity.

Introduction

Sarcasm is a sophisticated form of irony where the intended meaning is the opposite of the literal words. In social media, this often manifests as a "multimodal" phenomenon—for instance, a caption saying "Great weather!" paired with a photo of a torrential downpour. Detecting such sarcasm is difficult for machines because it relies on identifying incongruity.

Most current research focuses on local incongruity, searching for clashes within a single text-image pair. However, sarcastic posts often share underlying semantic structures or "vibes" globally. If a model can recognize that a new post resembles a known sarcastic pattern from its training history, its predictive power increases. This paper proposes $G^2SAM$, which shifts the focus from purely local analysis to a combination of local graph-based features and global semantic awareness.

Conceptual overview of global semantic awareness: the model identifies a query's sentiment by retrieving similar sarcastic and non-sarcastic examples from a global semantic space.

Methodology

The $G^2SAM$ framework is built upon three core components designed to capture both fine-grained relationships and global context.

1. Fine-grained Graph-aligned Model (FGM)

To capture the nuances of text and images, $G^2SAM$ treats them as graphs rather than flat vectors.

  • Textual Graph: Using BERT and the spaCy library, the model extracts word features and builds a graph where edges represent syntactic dependencies.
  • Visual Graph: The model uses a Vision Transformer (ViT) and object detection to identify key regions. These regions become nodes, with edges established based on visual similarity.
  • Fusion: A stacked multi-head self-attention module aligns these graphs, allowing the model to learn cross-modal relationships (e.g., how a specific word relates to a specific object in the image).

2. Global Semantic Awareness via $k$NN Retrieval

Instead of relying solely on a final classifier layer, $G^2SAM$ projects the fused multimodal representation into a "semantic space." During inference, the model performs a $k$-Nearest Neighbor ($k$NN) search to find the $k$ most similar instances from the training set. A weighted voting mechanism then integrates the labels of these neighbors into the final prediction.

3. Label-aware Graph Contrastive Learning (LGCL)

For $k$NN retrieval to be effective, the semantic space must be well-organized. The authors introduced a novel contrastive loss (LGCL) that explicitly uses labels during training. This loss forces sarcastic examples to cluster together while pushing them away from non-sarcastic ones, ensuring that the "neighbors" retrieved during inference are semantically relevant.

The $G^2SAM$ architecture: (Left) The Fine-grained Graph-aligned Model fuses text and image features. (Right) The three-step workflow involving training, space construction, and kNN-based prediction.

Experiments and Results

The researchers evaluated $G^2SAM$ on the HFM (Hierarchical Fusion Model) dataset, a standard benchmark consisting of nearly 25,000 Twitter posts.

Performance

$G^2SAM$ achieved state-of-the-art (SOTA) results, outperforming previous models like DIP:

  • Accuracy: 90.48% (an 0.89% improvement).
  • Macro-F1 Score: 89.65% (a 1.31% improvement).

Ablation Study

Ablation tests confirmed the importance of the global components. Removing the $k$NN retrieval or the LGCL training led to a significant drop in accuracy (approx. 1-1.5%), proving that "global awareness" is a key differentiator for the model's success.

Conclusion

$G^2SAM$ demonstrates that sarcasm detection benefits significantly from a global perspective. By combining fine-grained graph representations with a label-aware semantic space for $k$NN retrieval, the model captures patterns that local-only models miss. Future work may involve optimizing the retrieval process for real-time applications and testing the framework's generalizability to other social media platforms and tasks like hate speech detection.

Summary updated

BibTeX
@inproceedings{wei2024g,
  author = {Yiwei Wei and Shaozu Yuan and Hengyang Zhou and Longbiao Wang and Zhiling Yan and Ruosong Yang and Meng Chen},
  title = {{G^2SAM: Graph-Based Global Semantic Awareness Method for Multimodal Sarcasm Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i8.28766},
}
Figure