Abstract
Humans possess multimodal literacy, allowing them to actively integrate information from various modalities to form reasoning. Faced with challenges like lexical ambiguity in text, we supplement this with other modalities, such as thumbnail images or textbook illustrations. Is it possible for machines to achieve a similar multimodal understanding capability? In response, we present Understanding Pun with Image Explanations (UNPIE), a novel benchmark designed to assess the impact of multimodal inputs in resolving lexical ambiguities. Puns serve as the ideal subject for this evaluation due to their intrinsic ambiguity. Our dataset includes 1,000 puns, each accompanied by an image that explains both meanings. We pose three multimodal challenges with the annotations to assess different aspects of multimodal literacy; Pun Grounding, Disambiguation, and Reconstruction. The results indicate that various Socratic Models and Visual-Language Models improve over the text-only models when given visual context, particularly as the complexity of the tasks increases.
Introduction
Humans are naturally "multimodal literate." We use visual context to resolve lexical ambiguity without being explicitly prompted. For instance, if someone says, "I saw a man with a telescope," and shows a picture of a man holding the device, the visual cue immediately clarifies the syntactic ambiguity.
Current AI models often operate under the assumption that all provided images are relevant and all instructions are explicit. They frequently fail at "active understanding"—the ability to seek out and integrate visual information to clarify confusing text. Puns represent the ultimate stress test for this capability: they are inherently ambiguous, requiring the listener to grasp two meanings of a single word or phrase simultaneously. Researchers from Yonsei University argue that if a model cannot "get" a visual pun, it hasn't truly mastered multimodal reasoning.
The UNPIE Benchmark
To address this, the authors introduced the Understanding Pun with Image Explanations (UNPIE) benchmark. This dataset consists of 1,000 English puns, split evenly between homographic (e.g., "bank" vs. "bank") and heterographic (e.g., "gecko" vs. "echo") types.
Each pun is paired with:
- A "pun explanation" image, generated via DALL-E 3, designed to capture both meanings in a single frame.
- Two "pun disambiguator" images retrieved from LAION-2B, each depicting only one of the possible meanings.
- Human-verified translations into German, French, and Korean, which strip away the original English ambiguity to test the model's ability to reconstruct the joke.
Methodology
The researchers evaluated two primary architectures: Monolithic VLMs (end-to-end models like GPT-4V, LLaVA, and Qwen-VL) and Socratic Models (modular systems using a separate vision model like BLIP-2 to generate captions for an LLM).
These models were tested across three tasks of increasing difficulty:
- Pun Grounding: Identifying the specific punning phrase in a sentence with the help of an explanation image.
- Pun Disambiguation: Selecting the correct translation of a pun based on an image depicting only one meaning.
- Pun Reconstruction: Recreating the original English pun from a non-ambiguous foreign translation using a visual guide.
Key Insights and Results
The study yielded several critical findings regarding the state of multimodal AI:
- The "Multimodal Alignment Tax": Fine-tuning on standard multimodal machine translation (MMT) datasets like Multi30k actually decreased performance on pun reconstruction. This suggests current MMT datasets lack the deep visual-textual dependency required for complex reasoning, leading models to ignore nuanced visual cues.
- Monolithic Superiority: VLMs generally outperformed Socratic models. This indicates that simple text captions are insufficient for capturing the nuance required to understand wordplay; direct visual processing is essential.
- GPT-4V Leadership: GPT-4V emerged as the leader, achieving 97.6% accuracy in pun grounding and significantly outperforming smaller models in disambiguation tasks.
- The Complexity Gap: While models excel at grounding (identifying the pun), they struggle with reconstruction. This highlights that "generative multimodal literacy" remains an open challenge.
Conclusion and Future Work
While UNPIE provides a rigorous testbed, it is currently limited by its English-centricity and relatively small size for fine-tuning. Future work aims to expand the benchmark to include puns from diverse cultures and languages to test non-Western linguistic nuances. Ultimately, UNPIE serves as a starting point for developing models that can actively navigate and integrate multimodal information to solve real-world ambiguities.
Summary updated