Abstract
Multimodal sarcasm detection is a challenging task that requires capturing the subtle incongruity between textual and visual information. While existing methods focus on local incongruity within a single post, they often neglect global semantic patterns shared across different sarcastic instances. This paper introduces $G^2SAM$ (Graph-Based Global Semantic Awareness Method), which enhances sarcasm detection by incorporating a "global" perspective. $G^2SAM$ utilizes a fine-grained graph-aligned model to represent multimodal inputs and a $k$-Nearest Neighbor ($k$NN) retrieval strategy to leverage a library of known examples. To optimize this retrieval, we propose Label-aware Graph Contrastive Learning (LGCL) to structure the semantic space. Experimental results on the HFM benchmark demonstrate that $G^2SAM$ achieves state-of-the-art performance, highlighting the importance of global semantic congruity.
Introduction
Sarcasm is a sophisticated form of irony where the intended meaning is the opposite of the literal words. In social media, this often manifests as a "multimodal" phenomenon—for instance, a caption saying "Great weather!" paired with a photo of a torrential downpour. Detecting such sarcasm is difficult for machines because it relies on identifying incongruity.
Most current research focuses on local incongruity, searching for clashes within a single text-image pair. However, sarcastic posts often share underlying semantic structures or "vibes" globally. If a model can recognize that a new post resembles a known sarcastic pattern from its training history, its predictive power increases. This paper proposes $G^2SAM$, which shifts the focus from purely local analysis to a combination of local graph-based features and global semantic awareness.
Methodology
The $G^2SAM$ framework is built upon three core components designed to capture both fine-grained relationships and global context.
1. Fine-grained Graph-aligned Model (FGM)
To capture the nuances of text and images, $G^2SAM$ treats them as graphs rather than flat vectors.
- Textual Graph: Using BERT and the spaCy library, the model extracts word features and builds a graph where edges represent syntactic dependencies.
- Visual Graph: The model uses a Vision Transformer (ViT) and object detection to identify key regions. These regions become nodes, with edges established based on visual similarity.
- Fusion: A stacked multi-head self-attention module aligns these graphs, allowing the model to learn cross-modal relationships (e.g., how a specific word relates to a specific object in the image).
2. Global Semantic Awareness via $k$NN Retrieval
Instead of relying solely on a final classifier layer, $G^2SAM$ projects the fused multimodal representation into a "semantic space." During inference, the model performs a $k$-Nearest Neighbor ($k$NN) search to find the $k$ most similar instances from the training set. A weighted voting mechanism then integrates the labels of these neighbors into the final prediction.
3. Label-aware Graph Contrastive Learning (LGCL)
For $k$NN retrieval to be effective, the semantic space must be well-organized. The authors introduced a novel contrastive loss (LGCL) that explicitly uses labels during training. This loss forces sarcastic examples to cluster together while pushing them away from non-sarcastic ones, ensuring that the "neighbors" retrieved during inference are semantically relevant.
Experiments and Results
The researchers evaluated $G^2SAM$ on the HFM (Hierarchical Fusion Model) dataset, a standard benchmark consisting of nearly 25,000 Twitter posts.
Performance
$G^2SAM$ achieved state-of-the-art (SOTA) results, outperforming previous models like DIP:
- Accuracy: 90.48% (an 0.89% improvement).
- Macro-F1 Score: 89.65% (a 1.31% improvement).
Ablation Study
Ablation tests confirmed the importance of the global components. Removing the $k$NN retrieval or the LGCL training led to a significant drop in accuracy (approx. 1-1.5%), proving that "global awareness" is a key differentiator for the model's success.
Conclusion
$G^2SAM$ demonstrates that sarcasm detection benefits significantly from a global perspective. By combining fine-grained graph representations with a label-aware semantic space for $k$NN retrieval, the model captures patterns that local-only models miss. Future work may involve optimizing the retrieval process for real-time applications and testing the framework's generalizability to other social media platforms and tasks like hate speech detection.
Summary updated