Describe Images in a Boring Way: Towards Cross-Modal Sarcasm Generation

Jie Ruan, Yue Wu, Xiaojun Wan, Yuesheng ZhuWACV 2024 · OpenReview

Abstract

Sarcasm generation has been investigated in previous studies by considering it as a text-to-text generation problem, i.e., generating a sarcastic sentence for an input sentence. In this paper, we study a new problem of cross-modal sarcasm generation (CMSG), i.e., generating a sarcastic description for a given image. CMSG is challenging as models need to satisfy the characteristics of sarcasm, as well as the correlation between different modalities. In addition, there should be some inconsistency between the two modalities, which requires imagination. Moreover, high-quality training data is insufficient. To address these problems, we take a step toward generating sarcastic descriptions from images without paired training data and propose an Extraction-Generation-Ranking based Modular method (EGRM) for CMSG. Specifically, EGRM first extracts diverse information from an image at different levels and uses the obtained image tags, sentimental descriptive caption, and commonsense-based consequence to generate candidate sarcastic texts. Then, a comprehensive ranking algorithm, which considers image-text relation, sarcasticness, and grammaticality, is proposed to select a final text from the candidate texts. Human evaluation at five criteria on a total of 2100 generated image-text pairs and auxiliary automatic evaluation show the superiority of our method.

Introduction

While AI has made strides in detecting sarcasm or generating sarcastic text from other text, generating sarcasm from visual input is a new frontier. Sarcasm is inherently difficult for machines because it requires "imagination": the text must be related to the image but also semantically inconsistent with it (e.g., describing a rainy, miserable day as "perfect weather for a tan").

Researchers identify three main hurdles in Cross-Modal Sarcasm Generation (CMSG):

  1. Cross-modal Correlation: The text must actually relate to the image.
  2. Intentional Inconsistency: There must be a "reversal of valence" (flipping positive to negative or vice versa) to create the ironic effect.
  3. Data Scarcity: There are no high-quality, large-scale datasets of images paired with sarcastic captions to train standard supervised models.

Methodology: The EGRM Framework

To solve this without a dedicated dataset, the authors proposed the Extraction-Generation-Ranking based Modular method (EGRM). This unsupervised approach bypasses the need for scarce labeled training data by combining sentimental captioning, commonsense reasoning, and a multi-criteria ranking system.

The EGRM framework consists of three phases: Image Information Extraction, Sarcastic Text Generation, and Comprehensive Ranking.

The framework operates in three distinct phases:

  1. Image Information Extraction: The system uses YOLOv5 to extract object tags and SentiCap to generate a "sentimental descriptive caption" (e.g., "a bad rainy day").
  2. Sarcastic Text Generation:
    • Reverse the Valence (RTV): It flips the sentiment of the initial caption (e.g., "bad" becomes "good").
  3. Commonsense Reasoning: It uses COMET to infer potential consequences of the scene (e.g., a man surfing might "crash").
  4. Constrained Generation: Using CBART, it combines these elements into several candidate sarcastic sentences.
  5. Comprehensive Ranking: A ranking module selects the best candidate based on Image-Text Relation (via CLIPScore), Sarcasticness (via a fine-tuned RoBERTa model), and Grammaticality (via BERT/PPL).

Experimental Results and Examples

The researchers conducted a massive human evaluation involving 2,100 image-text pairs across 14 different systems. EGRM significantly outperformed baselines like BLIP and SC-R3. On a 0-5 scale, EGRM scored 2.85 in sarcasticness, compared to 2.22 for the best baseline. It also achieved a higher overall quality score of 2.90.

Examples of sarcastic captions generated by the EGRM framework across different scenarios.

As shown in the examples above, the model successfully creates friction between the visual modality and the textual description. For instance, in Case 4, the model identifies a rainy scene and sarcastically suggests it is a "great time to go hiking." This "reversal of valence" is the core mechanism for creating the ironic effect.

Conclusion

The title "Describe Images in a Boring Way" is itself a sarcastic nod to the task—the goal is actually to make descriptions much more interesting and human-like than standard, literal captions. While EGRM is a strong modular pipeline, future work remains to improve style diversity (e.g., self-deprecating or hyperbolic sarcasm) and to develop higher-quality multi-modal sarcasm benchmarks.

Summary updated

Abstract

Sarcasm generation has been investigated in previous studies by considering it as a text-to-text generation problem, i.e., generating a sarcastic sentence for an input sentence. In this paper, we study a new problem of cross-modal sarcasm generation (CMSG), i.e., generating a sarcastic description for a given image. CMSG is challenging as models need to satisfy the characteristics of sarcasm, as well as the correlation between different modalities. In addition, there should be some inconsistency between the two modalities, which requires imagination. Moreover, high-quality training data is insufficient. To address these problems, we take a step toward generating sarcastic descriptions from images without paired training data and propose an Extraction-Generation-Ranking based Modular method (EGRM) for CMSG. Specifically, EGRM first extracts diverse information from an image at different levels and uses the obtained image tags, sentimental descriptive caption, and commonsense-based consequence to generate candidate sarcastic texts. Then, a comprehensive ranking algorithm, which considers image-text relation, sarcasticness, and grammaticality, is proposed to select a final text from the candidate texts. Human evaluation at five criteria on a total of 2100 generated image-text pairs and auxiliary automatic evaluation show the superiority of our method. Code and data are publicly available 1 .
BibTeX
@inproceedings{ruan2024describe,
  author = {Jie Ruan and Yue Wu and Xiaojun Wan and Yuesheng Zhu},
  title = {{Describe Images in a Boring Way: Towards Cross-Modal Sarcasm Generation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=6k54JUcwsz},
}
Figure