Augmenters at SemEval-2023 Task 1: Enhancing CLIP in Handling Compositionality and Ambiguity for Zero-Shot Visual WSD through Prompt Augmentation and Text-To-Image Diffusion

Jie S. Li, Yow-Ting Shiue, Yong-Siang Shih, Jonas Geiping

Abstract

This paper describes zero-shot approaches for the Visual Word Sense Disambiguation (VWSD) Task in English. Our preliminary study shows that the simple approach of matching candidate images with the phrase using CLIP suffers from the many-to-many nature of image-text pairs. We find that the CLIP text encoder may have limited abilities in capturing the compositionality in natural language. Conversely, the descriptive focus of the phrase varies from instance to instance. We address these issues in our two systems, Augment-CLIP and Stable Diffusion Sampling (SD Sampling). Augment-CLIP augments the text prompt by generating sentences that contain the context phrase with the help of large language models (LLMs). We further explore CLIP models in other languages, as an ambiguous word may be translated into an unambiguous one in the other language. SD Sampling uses text-to-image Stable Diffusion to generate multiple images from the given phrase, increasing the likelihood that a subset of images match the one that paired with the text.

Introduction

The Visual Word Sense Disambiguation (VWSD) task presents a unique challenge at the intersection of computer vision and natural language processing. Given a context phrase (e.g., "breaking wheel") and a set of candidate images, a system must identify the single image that correctly represents the target word's meaning in that specific context. While OpenAI’s Contrastive Language-Image Pre-training (CLIP) has revolutionized zero-shot image classification, it faces significant hurdles in VWSD due to two primary "blind spots": compositionality and ambiguity.

Compositionality refers to the difficulty CLIP faces in understanding how word meanings shift when combined; for instance, "baby powder" is semantically distinct from the simple average of "baby" and "powder." Ambiguity arises because a single phrase can have multiple valid visual interpretations, and CLIP’s text encoder often fails to capture the specific descriptive focus required for niche or technical senses of a word. This research explores how Large Language Models (LLMs) and Diffusion models can be leveraged to mitigate these limitations.

Methodology

To address the shortcomings of baseline CLIP models, we propose two distinct systems designed to provide richer semantic and visual context.

1. Augment-CLIP

Augment-CLIP focuses on "de-biasing" the text representation through two primary techniques:

  • Key-to-Text (k2t) Augmentation: We utilize LLMs to generate descriptive sentences containing the target phrase. For example, for the phrase "administration prime minister," an LLM might generate: "The administration prime minister is the official title of the leader." This provides CLIP with structured semantic context, moving beyond the "bag-of-words" style processing often seen in its text encoder.
  • Cross-Lingual Embedding: We explore the intuition that ambiguity is often language-specific. By translating an English phrase into Chinese and utilizing a Chinese-specific CLIP model, we can often resolve ambiguity. A word with multiple meanings in English may translate to a highly specific, unambiguous term in another language, providing a clearer signal for image matching.

2. Stable Diffusion (SD) Sampling

Rather than relying exclusively on the provided candidate images, SD Sampling uses Stable Diffusion v1.4 to "hallucinate" reference images. We generate 50 synthetic images based on the input phrase, creating a visual representation of the model's internal knowledge. We then compare the ten candidate images to this "cloud" of 50 generated images using CLIP similarity. The candidate image with the highest similarity to the generated set is selected. This allows the system to leverage the pictorial diversity of a phrase as understood by a generative model trained on billions of images.

Experimental Setup

We evaluated our systems using the SemEval-2023 Task 1 dataset, which primarily focuses on English but includes Farsi and Italian components. The dataset consists of 12,869 training instances and a test set of 463 instances. Each instance includes a target word, a context phrase, and ten candidate images (one "gold" image and nine distractors). Performance was measured using Hit Rate (the frequency with which the top choice was correct) and Mean Reciprocal Rank (MRR).

Results and Discussion

Our experiments revealed that while individual augmentation strategies provided varied results, their combination through ensembling yielded the strongest performance.

Model Hit Rate (Test)
Base-CLIP (ViT-B/32) 57.88%
SD Sampling (Standalone) ~53.00%
Best Ensemble 63.93%

The Value of Complementarity

Interestingly, neither Augment-CLIP nor SD Sampling consistently outperformed the base CLIP model in isolation. However, they were "wrong in different ways." SD Sampling acts as a form of visual reasoning; if a generative model consistently renders a "breaking wheel" as a historical torture device, it helps filter out irrelevant distractors (like images of a music band) that might otherwise confuse a standard CLIP model. When ensembled, these systems provide complementary signals that significantly push the overall accuracy higher than any single model.

Limitations

Despite the improvements, several challenges remain. Stable Diffusion is susceptible to domain shift; for highly technical or historical terms, it may favor a modern popular interpretation over the correct context. Additionally, the cross-lingual approach is highly dependent on the quality of the translation engine. Finally, the choice of distance metric in the visual hallucination pipeline is critical, as $l_2$ distance between InceptionV3 features performed worse than CLIP-similarity.

Conclusion

Our findings suggest that the limitations of CLIP in handling compositionality and ambiguity can be effectively addressed through prompt augmentation and generative sampling. By using LLMs to provide "connective tissue" for natural language and Diffusion models to create visual references, we can move toward more robust zero-shot visual word sense disambiguation. Future work will focus on refining the distance metrics for synthetic image matching and improving the robustness of cross-lingual transfers.

Summary updated

Abstract

This paper describes our zero-shot approaches for the Visual Word Sense Disambiguation (VWSD) Task in English. Our preliminary study shows that the simple approach of matching candidate images with the phrase using CLIP suffers from the many-to-many nature of image-text pairs. We find that the CLIP text encoder may have limited abilities in capturing the compositionality in natural language. Conversely, the descriptive focus of the phrase varies from instance to instance. We address these issues in our two systems, Augment-CLIP and Stable Diffusion Sampling (SD Sampling). Augment-CLIP augments the text prompt by generating sentences that contain the context phrase with the help of large language models (LLMs). We further explore CLIP models in other languages, as the an ambiguous word may be translated into an unambiguous one in the other language. SD Sampling uses text-to-image Stable Diffusion to generate multiple images from the given phrase, increasing the likelihood that a subset of images match the one that paired with the text.
BibTeX
@inproceedings{li2023augmenters,
  author = {Jie S. Li and Yow-Ting Shiue and Yong-Siang Shih and Jonas Geiping},
  title = {{Augmenters at SemEval-2023 Task 1: Enhancing CLIP in Handling Compositionality and Ambiguity for Zero-Shot Visual WSD through Prompt Augmentation and Text-To-Image Diffusion}},
  booktitle = {SemEval @ ACL},
  year = {2023},
  url = {https://openreview.net/forum?id=GwZmJLmxxW},
}
Figure