Can Large Vision-Language Models Understand Multimodal Sarcasm?

Xinyu Wang, Yue Zhang, Liqiang Jing

Abstract

Sarcasm is a complex linguistic phenomenon that involves a disparity between literal and intended meanings, making it challenging for sentiment analysis and other emotion-sensitive tasks. While traditional sarcasm detection methods primarily focus on text, recent approaches have incorporated multimodal information. However, the application of Large Visual Language Models (LVLMs) in Multimodal Sarcasm Analysis (MSA) remains underexplored. In this paper, we evaluate LVLMs in MSA tasks, specifically focusing on Multimodal Sarcasm Detection and Multimodal Sarcasm Explanation. Through comprehensive experiments, we identify key limitations, such as insufficient visual understanding and a lack of conceptual knowledge. To address these issues, we propose a training-free framework that integrates in-depth object extraction and external conceptual knowledge to improve the model's ability to interpret and explain sarcasm in multimodal contexts. The experimental results on multiple models show the effectiveness of our proposed framework.

Introduction

Sarcasm represents a linguistic "double-cross" where the intended meaning contradicts the literal expression. In multimodal social media environments—such as Twitter or Instagram—sarcasm frequently emerges from the incongruity between a text caption and its accompanying image. For instance, a grey, rainy beach scene captioned "What a beautiful day!" is immediately recognized as sarcastic by humans, yet Large Vision-Language Models (LVLMs) often struggle with such nuances, frequently taking the text at face value.

The Multimodal Sarcasm Analysis task involves both detection (MSD) and explanation (MSE).

Current LVLMs face two primary hurdles in Multimodal Sarcasm Analysis (MSA):

  1. Insufficient Vision Ability: Models often overlook fine-grained visual details or suffer from hallucinations, missing the visual cues that contradict the text.
  2. Lack of External Knowledge: Models struggle to bridge the gap between literal objects and their latent associations (e.g., linking "hospital" to "sickness"), which are often essential for decoding sarcastic intent.

Methodology

To address these challenges, we propose a training-free, multi-source semantic-enhanced framework. This pipeline enhances LVLM performance without requiring additional fine-tuning through three key stages:

  • Fine-grained Object Extraction: We utilize Fast-RCNN to explicitly identify objects and their attributes (color, shape) within the image. This grounds the model in the actual visual content, reducing reliance on the internal vision encoder's potentially flawed perceptions.
  • External Knowledge Acquisition: Extracted objects and text keywords are mapped to ConceptNet, a structured commonsense knowledge graph. This provides semantic "neighbors"—for example, linking "beautiful" to "pleasing" or "foggy" to "unclear"—providing the necessary conceptual bridge for sarcasm detection.
  • Enhanced Prompting: The original post is augmented with the extracted objects and retrieved conceptual links. This enriched context is fed into the LVLM to perform Multimodal Sarcasm Detection (MSD) and Multimodal Sarcasm Explanation (MSE).
Overview of the proposed framework, integrating object extraction and external knowledge acquisition.

Experimental Setup

The framework was evaluated using two established benchmarks:

  • MSDD (Multimodal Sarcasm Detection Dataset): A collection of 2,409 Twitter posts labeled as "sarcastic" or "unsarcastic."
  • MORE: A dataset of 352 sarcastic posts featuring human-annotated explanations of the sarcasm.

Results and Discussion

The proposed framework yielded significant performance gains across various models, including LLaVA, MiniGPT, InstructBLIP, and GPT-4o.

Sarcasm Detection (MSD)

Baseline results indicated that even advanced models like GPT-4o struggle with zero-shot sarcasm detection. However, our framework allowed GPT-4o to achieve a peak accuracy of 75.3%, representing a 14.2% relative improvement. LLaVA showed even more dramatic progress, with accuracy rising from 41.1% to 60.3%.

Sarcasm Explanation (MSE)

The improvements in explanation tasks were particularly noteworthy. For MiniGPT, the BLEU-1 score (measuring text similarity) increased from 7.25 to 21.17. These results suggest that providing structured knowledge is more effective than raw text retrieval (like standard RAG), as ConceptNet’s triples provide a cleaner, more interpretable "bridge" for understanding latent associations.

Conclusion

Our research demonstrates that the "vision gap" in LVLMs is a significant barrier to understanding sarcasm. By forcing models to "see" specific objects and providing them with structured symbolic knowledge, we can significantly improve their ability to navigate multimodal incongruity. Future work will explore more advanced vision models to further enrich context and investigate specialized knowledge bases to capture cultural and regional nuances in sarcasm.

Summary updated

Abstract

Sarcasm is a complex linguistic phenomenon that involves a disparity between literal and intended meanings, making it challenging for sentiment analysis and other emotion-sensitive tasks. While traditional sarcasm detection methods primarily focus on text, recent approaches have incorporated multimodal information. However, the application of Large Visual Language Models (LVLMs) in Multimodal Sarcasm Analysis (MSA) remains underexplored. In this paper, we evaluate LVLMs in MSA tasks, specifically focusing on Multimodal Sarcasm Detection and Multimodal Sarcasm Explanation. Through comprehensive experiments, we identify key limitations, such as insufficient visual understanding and a lack of conceptual knowledge. To address these issues, we propose a training-free framework that integrates in-depth object extraction and external conceptual knowledge to improve the model's ability to interpret and explain sarcasm in multimodal contexts. The experimental results on multiple models show the effectiveness of our proposed framework. The code is available at https://github.com/cp-cp/LVLM-MSA.
BibTeX
@inproceedings{wang2025can,
  author = {Xinyu Wang and Yue Zhang and Liqiang Jing},
  title = {{Can Large Vision-Language Models Understand Multimodal Sarcasm?}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2025},
  doi = {10.1145/3746252.3760864},
}
Figure