TL;DR
This paper tackles the challenge of Visual Word Sense Disambiguation (VWSD) by using GPT-3 to generate more effective queries for CLIP, a multimodal model. By leveraging GPT-3's language understanding, they significantly improve CLIP's ability to select the correct image based on ambiguous word senses within a phrase.
What Problem Does This Paper Address?
The core problem is accurately retrieving an image that matches a textual description when that description contains words with multiple meanings (ambiguity). Traditional methods, even powerful ones like CLIP, struggle with this because they might focus on a common but incorrect sense of a word or misinterpret parts of the phrase. For instance, the phrase "Andromeda tree" could be misinterpreted as the constellation Andromeda or just a generic tree, rather than a specific plant species. This task, Visual Word Sense Disambiguation (VWSD), requires understanding the nuanced meaning of words within their context to select the correct image from a set of candidates.
Methods
The authors propose a two-stage approach. First, they use CLIP (Contrastive Language-Image Pre-training) as a baseline. CLIP creates a shared embedding space for images and text, allowing for similarity comparisons. They experiment with various "query templates" to feed text phrases into CLIP, aiming to better guide its understanding. These templates involve structuring the input phrase with placeholders for the ambiguous word and the rest of the phrase.
The key innovation is the second stage: using GPT-3 to enrich these queries. They leverage GPT-3's strong language generation capabilities to create more informative and contextually rich prompts. These prompts are designed to explicitly guide CLIP towards the intended meaning of the ambiguous word. Three types of GPT-3 generated prompts are explored: a direct query, a query with the phrase in double quotes, and an "explicit phrase" query that separates the ambiguous word from the rest of the phrase. These enriched queries are then used to finetune CLIP.
Datasets
The study uses data from the SemEval 2023 Task 1 (VWSD) dataset. This dataset includes images and phrases with ambiguous words. The official training data was split into a training set (11,869 examples) and a development set (500 examples). The test set contained 463 examples, distinct from the training data.
Key Insights and Ideas
- Human-like prompts matter: The study confirms previous findings that human-like prompts, especially those that explicitly highlight or quote ambiguous words, significantly improve performance compared to plain phrases.
- GPT-3 as a disambiguation aid: GPT-3, when prompted correctly, can generate contextual knowledge that effectively disambiguates word senses for CLIP, leading to better image retrieval.
- Finetuning is crucial: Simply using GPT-3 generated queries isn't enough; finetuning CLIP with these enriched queries is essential for realizing the performance gains.
- Explicit phrasing is powerful: The "explicit phrase" prompt type, which clearly separates the ambiguous word from the rest of the phrase, shows particularly strong results, suggesting that direct disambiguation is highly effective.
Key Results
The finetuned CLIP models consistently outperformed the baseline CLIP model (using only phrases).
- Baseline CLIP (phrase only): Achieved 71.50% accuracy on the Dev set and 58.53% on the Test set.
- CLIP finetuned with nine queries: Reached 90.20% on Dev and 55.29% on Test.
- CLIP finetuned with GPT-3 knowledge (explicit phrase): Achieved the highest accuracy on the Test set at 66.95%, demonstrating the benefit of GPT-3's generated knowledge. On the Dev set, this prompt type reached 92.20%.
- The paper also notes that while Dev set accuracies were generally high (often above 90%), Test set accuracies were considerably lower, suggesting a significant domain shift or difficulty in the Test set.
Limitations and Future Work
Limitations: * The study acknowledges that the Dev set performance was much higher than the Test set performance, potentially due to thematic differences between the datasets. * The "Direct" prompt type from GPT-3 sometimes led to worse performance than the baseline, suggesting that poorly generated knowledge can be detrimental. * Ensemble experiments showed improvement for most prompt types, but not for the "Explicit phrase" type, which was already performing very well.
Future Work: * The authors plan to explore other multimodal models and compare their performance. * They intend to investigate GPT-4 for generating knowledge, potentially incorporating both textual and visual cues for even better VWSD.
Summary updated