Introduction
Multimodal Sarcasm Understanding (MSU) is a critical task in natural language processing and computer vision, requiring models to detect the subtle incongruity between text and imagery. While previous research has focused on short-form content like tweets or Reddit comments, real-world journalism often employs sarcasm at a document level. In these contexts, sarcastic clues are frequently sparse, buried within long paragraphs, and tied to highly localized visual elements. This paper introduces DocMSU, a massive new benchmark designed to push MSU research into the domain of long-form news articles.
The Challenge of Document-Level Sarcasm
Existing models designed for short texts struggle with document-level MSU because they often rely on global feature representations. In a news article, the sarcasm might hinge on a single word in a 500-word text or a tiny label within a complex image. For example, a photo of a shattered CD might only be sarcastic if the model can identify a small "unbreakable" label on the packaging.
As shown in the figure above, the contradiction between the textual claim (e.g., "unbreakable") and the visual reality (the broken disc) is the key to detection. Standard foundation models like CLIP and ViLT often fail to capture these fine-grained, cross-modal incongruities when the context is long and the clues are localized.
The DocMSU Dataset
To address these limitations, the authors curated DocMSU, a dataset of 102,588 news articles across nine diverse topics, including Science, Health, Business, and Politics. Unlike previous datasets like MSTI or MUStARD, DocMSU features significantly longer texts, with an average of 63 tokens across five sentences.
The dataset includes high-quality, semi-manual annotations for:
- Binary Sarcasm Detection: Identifying if a text-image pair is sarcastic.
- Textual Sarcastic Spans: Localizing the specific words that convey the satire.
- Visual Bounding Boxes: Pinpointing the sarcastic objects within the image.
Methodology: Fine-Grained Multimodal Fusion
The authors propose a novel architecture designed for pixel-to-word alignment. The model uses a BERT encoder for text and the early convolutional layers of a ResNet for images to preserve high spatial resolution. The core innovation is the use of a Swin-Transformer for fusion. By reshaping the 1D document tokens into a 2D matrix and adding them to sliding image windows, the model employs shifted window attention to compute interactions between every word and every pixel efficiently. This allows the model to catch subtle contradictions that global encoders miss.
Key Results and Insights
The proposed model significantly outperformed existing baselines across all tasks:
- Detection: Achieved 97.83% accuracy and an 87.25% F1-score, vastly superior to ViLT's 41.44% F1-score.
- Localization: The model reached an AP50 of 35.29 for image localization and an EM50 of 52.19% for text span localization.
One of the most striking findings is that while Large Language Models (LLMs) like GPT-4 excel at explaining the cultural context of sarcasm once it is pointed out, they struggle to identify the underlying sarcastic causes autonomously in long-form documents. This highlights a significant gap in the reasoning capabilities of current generative models regarding subtle satire.
Conclusion
DocMSU establishes a more realistic and challenging standard for multimodal sarcasm understanding. By moving from sentence-level to document-level analysis and emphasizing fine-grained alignment, this work provides a foundation for more sophisticated AI capable of navigating the nuances of human communication in complex media environments.
Summary updated