Abstract
Visual Word Sense Disambiguation (VWSD) is the task of identifying the correct image for a polysemous word given its textual context. Current vision-language models (VLMs) often struggle with this task, as they tend to favor the most frequent sense encountered during pre-training. We propose an unsupervised framework that incorporates sense definitions (glosses) as a latent variable in a Bayesian inference model. Our approach decomposes VWSD into two sub-tasks: matching context to definitions (C2D) and matching definitions to images (D2I). To address out-of-vocabulary issues, we introduce Context-Aware Definition Generation (CADG) using large language models. Our method achieves state-of-the-art unsupervised performance on the SemEval-2023 Task 1 dataset, improving CLIP's Hits@1 from 73.00% to 83.08%.
Introduction
The ambiguity of language poses a significant challenge for computer vision. A single word, such as "Angora," can refer to a city, a goat, a cat, or a rabbit depending on the context. Standard VLMs like CLIP, while powerful, often fail to leverage subtle contextual clues to disambiguate these senses, frequently defaulting to the most common meaning.
This paper addresses two primary limitations in existing VWSD approaches: the lack of explicit reasoning about word senses and the difficulty of handling words missing from traditional lexical databases like WordNet.
Methodology
Our framework operates in a zero-shot, unsupervised manner, requiring no additional training of the underlying VLM.
Bayesian Inference with Gloss Information
We treat the word sense definition ($D$) as a latent variable. The probability of an image $v$ being the correct match for a target word $t$ in context $c$ is calculated by marginalizing over the possible definitions. This formulation allows us to break the problem into two components:
- Context-to-Definition (C2D): Estimating which definition $D_i$ best fits the context $c$.
- Definition-to-Image (D2I): Determining which image $v$ best represents the descriptive definition $D_i$.
Context-Aware Definition Generation (CADG)
To handle terms not found in WordNet, we utilize GPT-3 to generate definitions. Crucially, we employ Context-Aware Definition Generation, where the LLM is prompted with the specific context of the target word. This ensures that the generated "gloss" is relevant to the intended sense, preventing the model from being misled by irrelevant definitions.
Experimental Results
We evaluated our approach on the SemEval-2023 Task 1 VWSD dataset, which consists of 12,896 examples. Each task requires selecting the correct image from 10 candidates.
Using WordNet definitions alone, CLIP's performance improved significantly from a baseline of 73.00% to 81.98% Hits@1. By integrating CADG for cases where WordNet was insufficient, we achieved a peak performance of 83.08%. These results demonstrate that detailed definitions serve as better "visual anchors" for VLMs than single ambiguous words. The Bayesian framework effectively uses these anchors to guide the model toward the correct visual representation.
Conclusion
By using dictionary definitions as a bridge, we can significantly improve the disambiguation capabilities of off-the-shelf vision-language models. Our unsupervised Bayesian framework effectively marginalizes sense ambiguity, while context-aware generation addresses the limitations of static knowledge bases. Future work will focus on mitigating error propagation between the C2D and D2I stages and refining LLM prompting to reduce hallucinations.
Summary updated