DocMSU: A Comprehensive Benchmark for Document-Level Multimodal Sarcasm Understanding

Hang Du, Guoshun Nan, Sicheng Zhang, Binzhu Xie, Junrui Xu, Hehe Fan, Qimei Cui, Xiaofeng Tao, Xudong Jiang

Introduction

Multimodal Sarcasm Understanding (MSU) is a critical task in natural language processing and computer vision, requiring models to detect the subtle incongruity between text and imagery. While previous research has focused on short-form content like tweets or Reddit comments, real-world journalism often employs sarcasm at a document level. In these contexts, sarcastic clues are frequently sparse, buried within long paragraphs, and tied to highly localized visual elements. This paper introduces DocMSU, a massive new benchmark designed to push MSU research into the domain of long-form news articles.

The Challenge of Document-Level Sarcasm

Existing models designed for short texts struggle with document-level MSU because they often rely on global feature representations. In a news article, the sarcasm might hinge on a single word in a 500-word text or a tiny label within a complex image. For example, a photo of a shattered CD might only be sarcastic if the model can identify a small "unbreakable" label on the packaging.

Examples of document-level multimodal sarcasm showing localized visual and textual clues.

As shown in the figure above, the contradiction between the textual claim (e.g., "unbreakable") and the visual reality (the broken disc) is the key to detection. Standard foundation models like CLIP and ViLT often fail to capture these fine-grained, cross-modal incongruities when the context is long and the clues are localized.

The DocMSU Dataset

To address these limitations, the authors curated DocMSU, a dataset of 102,588 news articles across nine diverse topics, including Science, Health, Business, and Politics. Unlike previous datasets like MSTI or MUStARD, DocMSU features significantly longer texts, with an average of 63 tokens across five sentences.

Comprehensive statistics of the DocMSU dataset, including topic distribution and object categories.

The dataset includes high-quality, semi-manual annotations for:

  1. Binary Sarcasm Detection: Identifying if a text-image pair is sarcastic.
  2. Textual Sarcastic Spans: Localizing the specific words that convey the satire.
  3. Visual Bounding Boxes: Pinpointing the sarcastic objects within the image.

Methodology: Fine-Grained Multimodal Fusion

The authors propose a novel architecture designed for pixel-to-word alignment. The model uses a BERT encoder for text and the early convolutional layers of a ResNet for images to preserve high spatial resolution. The core innovation is the use of a Swin-Transformer for fusion. By reshaping the 1D document tokens into a 2D matrix and adding them to sliding image windows, the model employs shifted window attention to compute interactions between every word and every pixel efficiently. This allows the model to catch subtle contradictions that global encoders miss.

Key Results and Insights

The proposed model significantly outperformed existing baselines across all tasks:

  • Detection: Achieved 97.83% accuracy and an 87.25% F1-score, vastly superior to ViLT's 41.44% F1-score.
  • Localization: The model reached an AP50 of 35.29 for image localization and an EM50 of 52.19% for text span localization.

One of the most striking findings is that while Large Language Models (LLMs) like GPT-4 excel at explaining the cultural context of sarcasm once it is pointed out, they struggle to identify the underlying sarcastic causes autonomously in long-form documents. This highlights a significant gap in the reasoning capabilities of current generative models regarding subtle satire.

Conclusion

DocMSU establishes a more realistic and challenging standard for multimodal sarcasm understanding. By moving from sentence-level to document-level analysis and emphasizing fine-grained alignment, this work provides a foundation for more sophisticated AI capable of navigating the nuances of human communication in complex media environments.

Summary updated

Abstract

Multimodal Sarcasm Understanding (MSU) has a wide range of applications in the news field such as public opinion analysis and forgery detection. However, existing MSU benchmarks and approaches usually focus on sentence-level MSU. In document-level news, sarcasm clues are sparse or small and are often concealed in long text. Moreover, compared to sentence-level comments like tweets, which mainly focus on only a few trends or hot topics (e.g., sports events), content in the news is considerably diverse. Models created for sentence-level MSU may fail to capture sarcasm clues in document-level news. To fill this gap, we present a comprehensive benchmark for Document-level Multimodal Sarcasm Understanding (DocMSU). Our dataset contains 102,588 pieces of news with text-image pairs, covering 9 diverse topics such as health, business, etc. The proposed large-scale and diverse DocMSU significantly facilitates the research of document-level MSU in real-world scenarios. To take on the new challenges posed by DocMSU, we introduce a fine-grained sarcasm comprehension method to properly align the pixel-level image features with word-level textual features in documents. Experiments demonstrate the effectiveness of our method, showing that it can serve as a baseline approach to the challenging DocMSU.
BibTeX
@inproceedings{du2024docmsu,
  author = {Hang Du and Guoshun Nan and Sicheng Zhang and Binzhu Xie and Junrui Xu and Hehe Fan and Qimei Cui and Xiaofeng Tao and Xudong Jiang},
  title = {{DocMSU: A Comprehensive Benchmark for Document-Level Multimodal Sarcasm Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29748},
}
Figure