Abstract
Although automated image captioning methods have benefited considerably from the development of large language models (LLMs), generating humorous captions is still a challenging task. Humorous captions generated by humans are unique to the image and reflect the content of the image. However, captions generated using previous captioning models tend to be generic. Therefore, we propose incongruity-resolution chain-of-thought (IRCoT) as a novel prompting framework that creates content-specific resolutions from fine details extracted from an image. Furthermore, we integrate logit bias and negative sampling to suppress the output of generic resolutions. The results of experiments with GPT4-V demonstrate that our proposed framework effectively generated humorous captions tailored to the content of specific input images.
Introduction
Researchers have introduced IRCoT (Incongruity Resolution Chain-of-Thought), a prompting framework that helps Large Multimodal Models (LMMs) like GPT-4V generate humor that is actually specific to an image's content. By forcing the model to explicitly identify and resolve visual contradictions, and using "logit bias" to suppress generic clichés, the system produces captions that are significantly more tailored and often funnier than standard "one-shot" attempts.
The Problem: The "Genericness" Trap
If you’ve ever asked an AI to write a caption for a weird meme, you’ve likely noticed a pattern: it tends to be generic. A model might see a surreal cartoon and output something like, "When you walk into the wrong meeting and it's too late to leave." This is the "genericness" problem. While technically a joke, it ignores the specific visual nuances—like a character having a lion's face—that make that specific image unique.
The challenge is that humor often relies on incongruity-resolution theory: humor arises when we perceive an unexpected contradiction (incongruity) and then find a cognitive rule that explains it (resolution). Current LMMs often skip the "resolution" step or provide "lazy" resolutions (e.g., "it's just a dream"), leading to stale, repetitive humor.
The IRCoT Framework
The proposed IRCoT pipeline consists of five distinct steps designed to bridge the gap between visual perception and humorous reasoning:
- Image Description: The model extracts fine-grained details from the image.
- Incongruity Extraction: The model identifies what is "wrong" or contradictory in the scene.
- Incongruity Resolution: The model generates multiple potential "rules" or explanations for the contradiction.
- Humor Generation: It drafts captions based on those specific resolutions.
- Caption Selection: The best candidate is chosen.
To prevent the model from falling back on boring, catch-all explanations (like "this is a metaphor" or "it's a symbol"), the team implemented Logit Bias and Negative Sampling. These techniques mathematically penalize the model for using "penalty tokens"—words associated with generic resolutions—forcing it to find more creative, content-specific explanations. They also fine-tuned a LLaVA 1.5 model using a specialized "negative sampling loss" to achieve similar results without needing the massive compute of GPT-4V.
Experimental Setup and Datasets
The study utilized several diverse sources of humor to validate the framework:
- Imgflip: A popular English-language meme site.
- Bokete: A Japanese site where users provide humorous answers to image prompts.
- The New Yorker Cartoon Caption Contest: A source of sophisticated visual humor.
- OxfordTVG-HIC: A large-scale collection of image-text pairs.
- Resolution Dataset: A new dataset of 10,000 pairs created by the authors, where GPT-4V generated resolutions for images to help train the LLaVA model.
Key Results and Insights
- Specificity Boost: GPT-4V using IRCoT with Logit Bias achieved a SelfCIDEr score of 0.839, significantly higher than the simple prompt baseline (0.803). This metric measures how unique a caption is to its specific image.
- Human Preference: In a head-to-head evaluation, captions generated by the fine-tuned LLaVA model were voted more humorous than human-generated captions from the New Yorker contest in a significant number of cases.
- Reasoning as a Bridge: Humor isn't just a mapping from pixels to a punchline; it's a reasoning process. By making that process explicit (Chain-of-Thought), the model's "vision" and "logic" are better aligned.
- Safety via Explanation: Interestingly, the intermediate reasoning steps in IRCoT can help detect offensive content. If the "resolution" step involves a harmful stereotype, a safety filter can catch it more easily than it could by looking at a vague punchline alone.
Limitations and Future Work
While IRCoT represents a significant step forward, vision bottlenecks remain a challenge. If the LMM misidentifies an object, the resulting joke will be nonsensical. Furthermore, deep cultural references and the "uncanny valley" of humor—where a joke makes sense to a model but not to a human—are areas for future research. Future work will likely focus on better aligning the "resolution" step with human social common sense.
Summary updated