Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information

Sunjae Kwon, Rishabh Garodia, Minhwa Lee, Zhichao Yang, Hong Yu

Abstract

Visual Word Sense Disambiguation (VWSD) is the task of identifying the correct image for a polysemous word given its textual context. Current vision-language models (VLMs) often struggle with this task, as they tend to favor the most frequent sense encountered during pre-training. We propose an unsupervised framework that incorporates sense definitions (glosses) as a latent variable in a Bayesian inference model. Our approach decomposes VWSD into two sub-tasks: matching context to definitions (C2D) and matching definitions to images (D2I). To address out-of-vocabulary issues, we introduce Context-Aware Definition Generation (CADG) using large language models. Our method achieves state-of-the-art unsupervised performance on the SemEval-2023 Task 1 dataset, improving CLIP's Hits@1 from 73.00% to 83.08%.

Introduction

The ambiguity of language poses a significant challenge for computer vision. A single word, such as "Angora," can refer to a city, a goat, a cat, or a rabbit depending on the context. Standard VLMs like CLIP, while powerful, often fail to leverage subtle contextual clues to disambiguate these senses, frequently defaulting to the most common meaning.

The ambiguity of the word "Angora" across different contexts and corresponding images.

This paper addresses two primary limitations in existing VWSD approaches: the lack of explicit reasoning about word senses and the difficulty of handling words missing from traditional lexical databases like WordNet.

Methodology

Our framework operates in a zero-shot, unsupervised manner, requiring no additional training of the underlying VLM.

Bayesian Inference with Gloss Information

We treat the word sense definition ($D$) as a latent variable. The probability of an image $v$ being the correct match for a target word $t$ in context $c$ is calculated by marginalizing over the possible definitions. This formulation allows us to break the problem into two components:

  1. Context-to-Definition (C2D): Estimating which definition $D_i$ best fits the context $c$.
  2. Definition-to-Image (D2I): Determining which image $v$ best represents the descriptive definition $D_i$.
The proposed unsupervised framework utilizing Bayesian inference to match context to images via sense definitions.

Context-Aware Definition Generation (CADG)

To handle terms not found in WordNet, we utilize GPT-3 to generate definitions. Crucially, we employ Context-Aware Definition Generation, where the LLM is prompted with the specific context of the target word. This ensures that the generated "gloss" is relevant to the intended sense, preventing the model from being misled by irrelevant definitions.

Experimental Results

We evaluated our approach on the SemEval-2023 Task 1 VWSD dataset, which consists of 12,896 examples. Each task requires selecting the correct image from 10 candidates.

Using WordNet definitions alone, CLIP's performance improved significantly from a baseline of 73.00% to 81.98% Hits@1. By integrating CADG for cases where WordNet was insufficient, we achieved a peak performance of 83.08%. These results demonstrate that detailed definitions serve as better "visual anchors" for VLMs than single ambiguous words. The Bayesian framework effectively uses these anchors to guide the model toward the correct visual representation.

Conclusion

By using dictionary definitions as a bridge, we can significantly improve the disambiguation capabilities of off-the-shelf vision-language models. Our unsupervised Bayesian framework effectively marginalizes sense ambiguity, while context-aware generation addresses the limitations of static knowledge bases. Future work will focus on mitigating error propagation between the C2D and D2I stages and refining LLM prompting to reduce hallucinations.

Summary updated

Abstract

Visual Word Sense Disambiguation (VWSD) is a task to find the image that most accurately depicts the correct sense of the target word for the given context. Previously, image-text matching models often suffered from recognizing polysemous words. This paper introduces an unsupervised VWSD approach that uses gloss information of an external lexical knowledge-base, especially the sense definitions. Specifically, we suggest employing Bayesian inference to incorporate the sense definitions when sense information of the answer is not provided. In addition, to ameliorate the out-of-dictionary (OOD) issue, we propose a context-aware definition generation with GPT-3. Experimental results show that the VWSD performance significantly increased with our Bayesian inference-based approach. In addition, our context-aware definition generation achieved prominent performance improvement in OOD examples exhibiting better performance than the existing definition generation method.
BibTeX
@inproceedings{kwon2023vision,
  author = {Sunjae Kwon and Rishabh Garodia and Minhwa Lee and Zhichao Yang and Hong Yu},
  title = {{Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=N7XHqSbtyzb},
}
Figure