Abstract
Sarcasm is a "linguistic flip" where the intended meaning contradicts the literal words, posing a significant challenge for traditional AI models. This paper provides a comprehensive survey of Multimodal Sarcasm Detection (MSD), an emerging field that integrates text with images, audio, and video to capture subtle cues—such as facial expressions or mocking tones—that text-only models often miss. We categorize the technical evolution of MSD from traditional deep learning to multimodal transformers and LLM-based prompting. Furthermore, we examine key benchmarks like MUStARD and MMSD, highlighting the "incongruity principle" as the core mechanism for identifying sarcastic intent across different modalities.
The Rise of Multimodal Sarcasm Detection
Detecting sarcasm is a cornerstone of sentiment analysis, mental health monitoring, and hate speech detection. However, sarcasm is inherently contextual. While text-based sarcasm detection is well-studied, it often hits a performance ceiling because it lacks access to external cues. As shown in the recent publication trends, there has been a significant surge in research focusing on multimodal approaches to bridge this gap.
Technical Methodologies
The authors categorize the evolution of MSD into four technical eras:
- Traditional Deep Learning: Early models used separate encoders (e.g., ResNet for images, BERT for text) combined via simple fusion techniques like concatenation.
- Multimodal Transformers: Architectures like CLIP and ViLBERT project different data types into a shared feature space, making incongruities easier to detect.
- Graph-Based & Attention Models: These use Graph Convolutional Networks (GCNs) or inter-modality attention to map dependencies between specific visual objects and transcript words.
- LLM-Based Prompting: The latest frontier uses Large Language Models (LLMs) like GPT-4 or LLaVA for zero-shot reasoning about sarcastic intent.
Datasets and the Incongruity Principle
The field relies on several key benchmarks. For visuo-textual data, the MMSD2.0 dataset is a refined standard that removes spurious cues. For video-based detection, the MUStARD dataset—featuring clips from sitcoms like Friends—is the gold standard. These datasets allow models to learn the "Incongruity Principle," where a positive text paired with a negative facial expression or tone provides a strong signal for sarcasm.
Key Insights and Results
- Performance Gains: Multimodal models consistently outperform text-only baselines. On the MMSD dataset, top-tier models achieve F1-scores of ~94%.
- Complexity of Video: Detecting sarcasm in video remains harder than in static images, with state-of-the-art F1-scores on MUStARD hovering around 77-79%, indicating that temporal dynamics are still difficult for AI to master.
- Context and Auxiliary Tasks: Models perform better when they have access to previous lines of dialogue or are simultaneously trained on emotion recognition tasks.
Limitations and Future Directions
Despite progress, the field faces several challenges. Most research is biased toward English, ignoring the cultural dependency of sarcasm. Furthermore, sarcasm is subjective; future models should account for multiple human perspectives rather than a single "ground truth." Finally, improving the reasoning capabilities of LLMs regarding figurative language remains a major future direction.
Summary updated