Abstract
Sarcasm is a rhetorical device used to convey the opposite of the literal meaning of an utterance. Widely used in social media and computer-mediated communication, its detection is a critical task for computational models. While early approaches focused primarily on text, sarcasm often relies on cues from tonality, facial expressions, and visual context. This paper presents a comprehensive survey of Multimodal Sarcasm Detection (MSD), reviewing research from 2018 to 2023. We categorize the models and datasets used in the field and outline future research directions.
Introduction
Sarcasm represents a sophisticated linguistic phenomenon where the intended meaning contradicts the literal words spoken. In human interaction, these nuances are often signaled through non-verbal cues such as a raised eyebrow or a specific vocal lilt. For Artificial Intelligence, however, decoding this "linguistic dance" is notoriously difficult. Traditional sentiment analysis often fails when faced with sarcasm because the literal sentiment (e.g., "Great weather we're having!") contradicts the reality (e.g., a torrential downpour).
This survey explores the transition from text-only models to multimodal approaches that integrate images, audio, and video to bridge the "sarcasm gap."
The Core Challenge: Incongruity
The fundamental difficulty in sarcasm detection lies in identifying incongruity. In text-only data, this incongruity is often implicit or requires extensive world knowledge. Multimodal settings provide additional channels to identify these contradictions:
- Visuo-Textual Incongruity: A positive caption paired with a negative image (e.g., a photo of a burnt pizza captioned "Another perfect dinner!").
- Audio-Visual Incongruity: A sarcastic tone of voice or a specific facial expression that contradicts the spoken transcript.
Methodological Evolution
The survey categorizes MSD methods into three primary evolutionary stages:
1. Traditional Deep Learning
Early models utilized separate encoders—such as ResNet for images and LSTMs for text—combining them through simple feature concatenation. As the field matured, researchers shifted toward hierarchical fusion to better capture the complex interactions between different modalities.
2. Multimodal Transformers
The current state-of-the-art involves architectures like VisualBERT, ViLBERT, and CLIP. These models employ attention mechanisms to align text and visual features within a shared embedding space. This allows the model to "attend" to specific image patches or audio segments that directly contradict the textual content.
3. LLM-based Approaches
The most recent frontier involves prompting Large Language Models (LLMs) and Vision-Language Models (VLMs) like GPT-4 or InstructBLIP. While some researchers use prompt tuning to frame sarcasm detection as a masked language prediction task, these general-purpose models often struggle in zero-shot scenarios compared to specialized, task-specific architectures.
Benchmark Datasets
The development of robust MSD models relies on high-quality, annotated datasets. Key benchmarks include:
- MSDD & MSDD 2.0: Primary datasets for image-text sarcasm collected from Twitter. MSDD 2.0 is a refined version that removes "spurious cues" (like hashtags) that previously allowed models to achieve high accuracy without truly understanding the content.
- MUStARD / MUStARD++: The standard for video and audio sarcasm, featuring clips from sitcoms like Friends and The Big Bang Theory.
- MORE: A dataset providing natural language explanations for sarcasm, moving beyond binary classification to explain why a pair is sarcastic.
- Spanish Multimodal Sarcasm: One of the few non-English datasets, utilizing dubbed clips from shows like South Park.
Key Insights
- Incongruity as a Feature: The most successful models are designed to detect inter-modal conflict (between modalities) or intra-modal conflict (within one modality).
- The "Cheating" Problem: Early datasets were often "leaky." If a tweet contained #sarcasm, models could "cheat" by identifying the tag. Modern datasets force models to process the actual content.
- Multi-task Learning: Training models to detect sentiment or emotion alongside sarcasm significantly boosts performance, as sarcasm is deeply tied to emotional states like contempt or frustration.
- Perspectivism: Sarcasm is inherently subjective. The authors suggest that ground truth should be based on an aggregate of multiple perspectives rather than a single label.
Performance Snapshot
On the Visuo-Textual MSDD dataset, specialized models (e.g., Ding et al., 2022) have achieved F1-scores as high as 94.06. In contrast, GPT-4 achieved an accuracy of 75.88, suggesting that general-purpose LLMs still lag behind specialized multimodal transformers in nuanced figurative language detection. Performance on Audio-Visual datasets (MUStARD) remains lower, with F1-scores in the 70s, reflecting the complexity of temporal data.
Future Directions
The survey identifies four critical areas for future research:
- Multilingualism: Expanding research beyond English to account for cultural dependencies in sarcasm.
- Subjectivity: Incorporating "perspectivism" to handle annotator disagreement.
- Inter-task Dependencies: Modeling the relationships between sarcasm, humor, irony, and hate speech.
- Reasoning: Moving toward models that can generate human-like explanations for sarcastic content.
Summary updated