Abstract
Multimodal sarcasm detection models often suffer from overfitting on in-domain data due to the lack of accurate external knowledge and the relatively low-quality data in existing benchmarks, leading to poor model robustness. To address this, we propose to use multimodal large language models (MLLMs) as a knowledge source to extract relevant knowledge step by step, in which Chain-of-Thought (CoT) is used to understand rich semantics. Additionally, we design a multi-stage fusion module that interactively and incrementally integrates generated external knowledge with the original data, aiming to uncover deeper levels of sarcastic semantics. Extensive experiments show that our approach not only significantly advances the state-of-the-art results but also demonstrates exceptional robustness and noise resistance in out-of-domain scenarios. Furthermore, our method exhibits sarcasm detection capabilities that substantially outperform other models, even when trained on a very limited amount of data.
Introduction: The Challenge of Sarcastic Incongruity
Detecting sarcasm is a complex task for artificial intelligence because it hinges on the incongruity between literal expression and intended sentiment. In a multi-modal context—such as a social media post containing both text and an image—the sarcasm often resides in a subtle contradiction between the visual evidence and the textual claim.
Current Multi-modal Sarcasm Detection (MSD) systems face three primary hurdles:
- Overfitting on Spurious Cues: Models often rely on "easy" features like specific hashtags (#sarcasm) or emojis. When these are removed (as in the MMSD2.0 benchmark), performance typically drops significantly.
- Low-Quality External Knowledge: Previous attempts to incorporate external data (like object labels) often introduced noise because they failed to capture internet slang or complex cultural nuances.
- Underutilization of LLMs: While LLMs have been used for simple captioning, their capacity for complex reasoning about the relationship between image and text has remained largely untapped.
Methodology: CoTMSD
To address these challenges, we introduce CoTMSD, a framework that treats sarcasm detection as a reasoning task rather than a simple classification problem. The system consists of two primary phases:
1. Knowledge Acquisition via Chain-of-Thought (CoT)
CoTMSD utilizes a Multimodal Large Language Model (specifically InternLM-Xcomposer2-vl-7b) to generate three distinct "views" of the input data:
- Image View ($I^m$): A concise description of the visual scene.
- Text View ($T^m$): An interpretation of both the literal and implicit meanings of the text.
- Mixture View ($Mix^m$): A reasoning step where the MLLM analyzes the relationship between the image and text to explain the underlying sarcastic semantics.
2. Multi-Stage Fusion Module
The framework integrates this generated knowledge with the original data through a sophisticated multi-stage fusion pipeline.
The fusion process involves:
- Information Interaction Layer: Parallel paths use Transformer encoders to fuse original image/text data with their respective MLLM-generated descriptions.
- Information Fusion Layer: A sequential process that first merges the processed image and text features, then incorporates the "Mixture View" to capture deep-level sarcastic semantics.
- Backbone: The system employs a frozen CLIP (ViT-B/32) model as the primary encoder for both modalities.
Evaluation and Results
CoTMSD was evaluated on three major benchmarks: MMSD (standard Twitter data), MMSD2.0 (a "cleaned" version without misleading cues), and RedEval (an out-of-domain Reddit dataset).
Performance and Robustness
- State-of-the-Art Results: CoTMSD achieved 95.11% accuracy on MMSD and 93.40% on MMSD2.0, representing a significant 6.58% improvement over previous benchmarks on the harder MMSD2.0 set.
- Exceptional Generalization: On the out-of-domain RedEval dataset, CoTMSD achieved a near-perfect 99.90% accuracy. In contrast, competing models like DynRT-Net saw their accuracy drop to approximately 58.57%.
- Data Efficiency: Remarkably, CoTMSD trained on only 10% of the training data outperformed traditional SOTA models trained on the full dataset, highlighting the value of high-quality, reasoned knowledge over raw data volume.
Conclusion
CoTMSD demonstrates that the "secret sauce" for effective sarcasm detection is reasoning. By leveraging the internal world knowledge of MLLMs through Chain-of-Thought prompting, the model learns why a pair is sarcastic. While the computational cost of large MLLMs remains a consideration, the gains in robustness and accuracy mark a significant step forward for multimodal understanding.
Summary updated