DocMSU: A Comprehensive Benchmark for Document-level Multimodal Sarcasm Understanding

Hang Du, Guoshun Nan, Sicheng Zhang, Binzhu Xie, Junrui Xu, Hehe Fan, Qimei Cui, Xiaofeng Tao, Xudong Jiang

Abstract

Multimodal Sarcasm Understanding (MSU) has a wide range of applications in the news field such as public opinion analysis and forgery detection. However, existing MSU benchmarks and approaches usually focus on sentence-level MSU. In document-level news, sarcasm clues are sparse or small and are often concealed in long text. Moreover, compared to sentence-level comments like tweets, which mainly focus on only a few trends or hot topics (e.g., sports events), content in the news is considerably diverse. Models created for sentence-level MSU may fail to capture sarcasm clues in document-level news. To fill this gap, we present a comprehensive benchmark for Document-level Multimodal Sarcasm Understanding (DocMSU). Our dataset contains 102,588 pieces of news with text-image pairs, covering 9 diverse topics such as health, business, etc. The proposed large-scale and diverse DocMSU significantly facilitates the research of document-level MSU in real-world scenarios. To take on the new challenges posed by DocMSU, we introduce a fine-grained sarcasm comprehension method to properly align the pixel-level image features with word-level textual features in documents. Experiments demonstrate the effectiveness of our method, showing that it can serve as a baseline approach to the challenging DocMSU.

Introduction: The Challenge of Document-Level Sarcasm

Most AI models trained to recognize sarcasm focus on "sentence-level" data, such as short tweets or Reddit comments. However, in professional news articles, sarcasm is often much more subtle. Clues are "sparse," meaning they might be buried within a long paragraph or hidden in a tiny detail of an accompanying photograph.

Example of visual irony where a fire extinguisher is on fire, requiring cross-modal reasoning to detect sarcasm.

Existing benchmarks fail to capture this complexity because they lack length and topical diversity. Furthermore, sarcasm in news often requires "cross-modal" reasoning: the text might seem perfectly positive on its own, but it becomes sarcastic only when contrasted with the image. The goal of DocMSU is to bridge this gap by moving from simple sentence classification to complex document-level understanding and localization.

The DocMSU Dataset

The researchers introduced the DocMSU Dataset, which significantly expands the scale and scope of MSU research. It contains 102,588 news samples, with 34,130 labeled as sarcastic and 68,458 as non-sarcastic.

Distribution of sarcastic and non-sarcastic samples across nine diverse topics in the DocMSU dataset.

The dataset covers nine diverse topics, including Science, Health, Business, and Entertainment. Each sample averages 63 tokens across five sentences—significantly longer than the ~20 tokens found in previous datasets like MSTI. A rigorous three-round manual annotation process involving 15 workers ensured high-quality labels, including "confidence scores" to account for the subjective nature of irony.

Methodology: Fine-Grained Sarcasm Comprehension

To handle the "needle in a haystack" nature of document-level irony, the authors propose a novel fine-grained sarcasm comprehension method designed to align text and imagery at a granular level.

  1. Document Encoding: BERT extracts word-level features, which are then reshaped into a square document representation matrix to better align with 2D image features.
  2. Image Encoding: Early layers of a ResNet maintain high spatial resolution, ensuring small visual clues are preserved.
  3. Multimodal Fusion: A Swin-Transformer with a sliding window mechanism performs "pixel-word" feature fusion. By adding the square document matrix to patches of the image feature map, the model can attend to specific words and image regions simultaneously.

The model is trained for two primary tasks: Sarcasm Detection (binary classification) and Sarcasm Localization (identifying specific sarcastic text spans and drawing bounding boxes around ironic objects).

Key Results and Insights

The proposed model (Ours_SB) achieved an F1-score of 87.25% on sarcasm detection, significantly outperforming multimodal baselines like CLIP (77.62%) and ViLT (41.44%). In the localization task, it surpassed the previous state-of-the-art (MSTI) by over 7 points in EM50 (Exact Match with 50% overlap).

Key insights from the study include:

  • Sarcasm is Localized: Irony usually resides in a few specific words or a small image region. Focusing on localization significantly improves detection accuracy.
  • Spatial Alignment: Reshaping text embeddings into a 2D square allows vision-based Transformers to treat words like pixels, facilitating natural spatial alignment.
  • LLM Limitations: While LLMs like GPT-4 can explain obvious sarcasm, they struggle with cases requiring intricate cultural knowledge or subtle visual-textual contradictions.

Conclusion and Future Work

DocMSU provides a robust foundation for studying document-level sarcasm. However, challenges remain. About 5% of the data is considered "challenging" due to the inherent subjectivity of irony. Future work should explore how sarcasm manifests across different cultures and languages, as well as potential demographic differences in how sarcasm is perceived and produced in professional news contexts.

Summary updated

Abstract

Multimodal Sarcasm Understanding (MSU) has a wide range of applications in the news field such as public opinion analysis and forgery detection. However, existing MSU benchmarks and approaches usually focus on sentence-level MSU. In document-level news, sarcasm clues are sparse or small and are often concealed in long text. Moreover, compared to sentence-level comments like tweets, which mainly focus on only a few trends or hot topics (e.g., sports events), content in the news is considerably diverse. Models created for sentence-level MSU may fail to capture sarcasm clues in document-level news. To fill this gap, we present a comprehensive benchmark for Document-level Multimodal Sarcasm Understanding (DocMSU). Our dataset contains 102,588 pieces of news with text-image pairs, covering 9 diverse topics such as health, business, etc. The proposed large-scale and diverse DocMSU significantly facilitates the research of document-level MSU in real-world scenarios. To take on the new challenges posed by DocMSU, we introduce a fine-grained sarcasm comprehension method to properly align the pixel-level image features with word-level textual features in documents. Experiments demonstrate the effectiveness of our method, showing that it can serve as a baseline approach to the challenging DocMSU. Our code and dataset are available at https://github.com/Dulpy/DocMSU.
BibTeX
@inproceedings{du2024docmsu,
  author = {Hang Du and Guoshun Nan and Sicheng Zhang and Binzhu Xie and Junrui Xu and Hehe Fan and Qimei Cui and Xiaofeng Tao and Xudong Jiang},
  title = {{DocMSU: A Comprehensive Benchmark for Document-level Multimodal Sarcasm Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29748},
}
Figure