Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!

Jiwan Chung, Seungwon Lim, Jaehyun Jeon, Seungbeen Lee, Youngjae Yu

Abstract

Humans possess multimodal literacy, allowing them to actively integrate information from various modalities to form reasoning. Faced with challenges like lexical ambiguity in text, we supplement this with other modalities, such as thumbnail images or textbook illustrations. Is it possible for machines to achieve a similar multimodal understanding capability? In response, we present Understanding Pun with Image Explanations (UNPIE), a novel benchmark designed to assess the impact of multimodal inputs in resolving lexical ambiguities. Puns serve as the ideal subject for this evaluation due to their intrinsic ambiguity. Our dataset includes 1,000 puns, each accompanied by an image that explains both meanings. We pose three multimodal challenges with the annotations to assess different aspects of multimodal literacy: Pun Grounding, Disambiguation, and Reconstruction. The results indicate that various Socratic Models and Visual-Language Models improve over the text-only models when given visual context, particularly as the complexity of the tasks increases.

Introduction

Humans are naturally "multimodal literate." When encountering ambiguous language, we instinctively seek out visual context to resolve it. For instance, the phrase "I saw a man with a telescope" is clarified instantly if we see an image of a man looking through a lens. Current AI models, however, often operate under the "lazy" assumption that visual input is merely decorative, lacking the "active understanding" required to seek out specific information in an image to clarify a confusing sentence.

A visual pun illustrating the ambiguity between 'leak' and 'leek' under a sink.

Puns represent a perfect stress test for this capability because they are inherently ambiguous. To "get" a pun, a model must grasp two meanings at once and use a visual cue to decide which one fits the context. This study introduces the UNPIE benchmark to evaluate whether Vision-Language Models (VLMs) can leverage images to resolve these textual ambiguities.

The UNPIE Benchmark

The UNPIE (Understanding Pun with Image Explanations) dataset consists of 1,000 English puns, split evenly between homographic (e.g., "bank" as a shore vs. a building) and heterographic (e.g., "buoy" vs. "boy") types. Each pun is paired with an AI-generated "explanation image" (via DALL-E 3) depicting both meanings, and two "disambiguator images" showing individual meanings. To test multilingual capabilities, each pun was translated into German, French, and Korean by human annotators who were specifically instructed to remove the ambiguity in the target language.

The three hierarchical tasks of the UNPIE benchmark: Pun Grounding, Disambiguation, and Reconstruction.

The benchmark poses three hierarchical tasks:

  1. Pun Grounding: Identifying the specific word or phrase that creates the pun.
  2. Pun Disambiguation: Choosing the correct non-ambiguous translation based on an image depicting only one of the pun's meanings.
  3. Pun Reconstruction: Recreating the original English pun from a "flat" (non-ambiguous) translation in another language, aided by an image explaining the joke.

Methodology

The researchers evaluated two main types of architectures:

  • Socratic Models (SM): A modular approach where one model generates a text caption for an image, and a Large Language Model (like GPT-4) uses that caption to solve the task.
  • Monolithic VLMs: End-to-end models (like LLaVA or Qwen-VL) that process raw pixels and text simultaneously.

Results and Discussion

The results indicate that visual context significantly aids models, with the benefit becoming more pronounced as task complexity increases.

Key Performance Metrics

  • Grounding: Adding images led to a consistent performance jump. For example, Qwen-VL saw its accuracy increase by 19.8% when provided with visual context.
  • Disambiguation: GPT-4 (Socratic) achieved the highest scores (up to 72.9% accuracy in French), but even smaller open-source models like LLaVA demonstrated the ability to use images to select the correct translation over 65% of the time.
  • Reconstruction: This was the most challenging task. GPT-4 with visual cues achieved a 13.7% improvement over its text-only baseline in German-to-English reconstruction, proving that the image provided the "missing link" to rediscover the pun.

Insights

  • The "MMT Paradox": Interestingly, fine-tuning models on standard Multimodal Machine Translation (MMT) datasets actually decreased performance on pun reconstruction. This suggests that current MMT datasets are too simple and fail to teach models to rely on visual dependencies.
  • Monolithic > Socratic: End-to-end VLMs generally outperformed the Socratic approach, suggesting that raw visual features capture nuances that text captions lose.
  • Heterographic Difficulty: Models found it much more difficult to reconstruct puns where words are spelled differently (e.g., buoy vs. boy), highlighting a gap in bridging phonetics and semantics.

Conclusion

While VLMs are improving, a significant "human-machine gap" remains; models still lack the nuanced "aha!" moment that humans experience when a visual cue makes a joke click. UNPIE serves as a starting point for developing models that can actively navigate the messy, ambiguous world of human communication. Future work could involve expanding the dataset to include puns native to other languages and cultures.

Summary updated

Abstract

Humans possess multimodal literacy, allowing them to actively integrate information from various modalities to form reasoning. Faced with challenges like lexical ambiguity in text, we supplement this with other modalities, such as thumbnail images or textbook illustrations. Is it possible for machines to achieve a similar multimodal understanding capability?In response, we present Understanding Pun with Image Explanations (UNPIE), a novel benchmark designed to assess the impact of multimodal inputs in resolving lexical ambiguities. Puns serve as the ideal subject for this evaluation due to their intrinsic ambiguity. Our dataset includes 1,000 puns, each accompanied by an image that explains both meanings. We pose three multimodal challenges with the annotations to assess different aspects of multimodal literacy; Pun Grounding, Disambiguation, and Reconstruction. The results indicate that various Socratic Models and Visual-Language Models improve over the text-only models when given visual context, particularly as the complexity of the tasks increases.
BibTeX
@inproceedings{chung2024can,
  author = {Jiwan Chung and Seungwon Lim and Jaehyun Jeon and Seungbeen Lee and Youngjae Yu},
  title = {{Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=KOkTMkjno6},
}
Figure