A Survey of Multimodal Sarcasm Detection

Shafkat Farabi, Tharindu Ranasinghe, Diptesh Kanojia, Yu Kong, Marcos Zampieri

Abstract

Sarcasm is a rhetorical device used to convey the opposite of the literal meaning of an utterance. Widely used on social media and other forms of computer-mediated communication, sarcasm presents a significant challenge for computational models. While early approaches focused solely on text, sarcasm often relies on tonality, facial expression, and contextual imagery. This has led to the emergence of Multimodal Sarcasm Detection (MSD), which integrates audio, images, text, and video. This paper presents a comprehensive survey of MSD research published between 2018 and 2023, discussing the models, datasets, and future directions in the field.

Introduction

Sarcasm acts as a "linguistic wild card," where intended meaning flips the literal words. Detecting it is crucial for sentiment analysis, mental health monitoring, and hate speech detection. In a text-only environment, a phrase like "What a beautiful day!" appears positive; however, when paired with a photo of a hurricane or spoken with a biting, low-pitched tone, the meaning becomes ironic.

The core challenge in MSD is identifying incongruity—the conflict between different modalities, such as a positive caption paired with a negative image. As shown in the publication trends, research interest in this area has grown significantly as researchers move beyond text to capture these complex signals.

Publication trends in Multimodal Sarcasm Detection (MSD) and general Sarcasm Detection from 2018 to 2023.

Methodologies in MSD

MSD approaches are generally categorized into two streams based on the data types involved:

Visuo-Textual (Image + Text)

  • Traditional Deep Learning: Early models utilized separate encoders (e.g., ResNet for images, BERT for text) fused via simple concatenation or hierarchical structures.
  • Multimodal Transformers: Modern approaches use architectures like ViLBERT or CLIP to project text and images into a shared feature space, enabling the model to correlate specific words with image regions.
  • LLM-based: Recent experiments leverage Large Language Models (LLMs) like GPT-4 with prompt engineering to identify sarcasm in complex social media posts and memes.

Audio-Visual & Textual (Video + Audio + Text)

  • Feature Extraction: Models extract acoustic features (pitch, tempo), visual features (facial expressions), and textual transcripts.
  • Attention Mechanisms: Techniques like the Inter-modal Word-level Attention Network (IWAN) help models focus on specific moments where vocal tone contradicts the spoken words.
  • Multi-task Learning: High-performing models often treat sentiment or emotion detection as auxiliary tasks to better understand the speaker's underlying state.

Benchmark Datasets

Several key datasets have shaped the MSD landscape:

  • MSDD & MSDD 2.0: Large-scale Twitter datasets containing text-image pairs. MSDD 2.0 is refined to remove "spurious cues" like #sarcasm hashtags that previously allowed models to "cheat."
  • MUStARD: The gold standard for video MSD, featuring clips from sitcoms like Friends and The Big Bang Theory. It includes conversational context, which is vital as sarcasm is often a reaction to previous statements.
  • MORE: Provides natural language explanations for why a post is considered sarcastic.
  • Spanish Multimodal Sarcasm: One of the few non-English datasets, highlighting a gap in current multilingual research.
A multimodal example from the MUStARD dataset, combining visual cues, text, and audio waveforms.

Key Insights and Performance

Research indicates that sarcasm detection is less about finding "negative" signals and more about measuring the distance or conflict between modalities—the Incongruity Principle. Furthermore, there is a high correlation between sarcasm and specific emotional states like contempt or mock-joy; training models to recognize these emotions first significantly improves performance.

In terms of results, Visuo-Textual models on the MSDD dataset have achieved F1 scores as high as 94.06%. Video-based detection remains more challenging; the best models on the MUStARD dataset currently hover around 77-79% F1, indicating that the nuances of human timing and expression are difficult to capture. Interestingly, zero-shot GPT-4 performance (~75% F1) currently lags behind smaller, fine-tuned specialized models.

Future Research Directions

Despite rapid progress, several gaps remain:

  • Multilingual Gap: Most research is English-centric, yet sarcasm is deeply culturally dependent.
  • Perspectivism: Sarcasm is subjective. Future models should account for annotator disagreement rather than assuming a single ground truth.
  • Inter-task Dependencies: Modeling sarcasm jointly with humor and hate speech could help distinguish between "friendly jokes" and "toxic irony."
  • LLM Reasoning: Exploring how complex prompt engineering and chain-of-thought reasoning can be applied to multimodal irony is a burgeoning frontier.

Summary updated

Abstract

Sarcasm is a rhetorical device that is used to convey the opposite of the literal meaning of an utterance. Sarcasm is widely used on social media and other forms of computer-mediated communication motivating the use of computational models to identify it automatically. While the clear majority of approaches to sarcasm detection have been carried out on text only, sarcasm detection often requires additional information present in tonality, facial expression, and contextual images. This has led to the introduction of multimodal models, opening the possibility to detect sarcasm in multiple modalities such as audio, images, text, and video. In this paper, we present the first comprehensive survey on multimodal sarcasm detection - henceforth MSD - to date. We survey papers published between 2018 and 2023 on the topic, and discuss the models and datasets used for this task. We also present future research directions in MSD.
BibTeX
@inproceedings{farabi2024survey,
  author = {Shafkat Farabi and Tharindu Ranasinghe and Diptesh Kanojia and Yu Kong and Marcos Zampieri},
  title = {{A Survey of Multimodal Sarcasm Detection}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=vLr9mXIlHP},
}
Figure