TL;DR
This paper tackles Visual Word Sense Disambiguation (V-WSD) by using ChatGPT to generate richer textual prompts for ambiguous words. These enhanced prompts, when fed into a CLIP model, significantly improve its ability to match words with their correct visual representations, especially in contexts where text alone is insufficient.
What Problem Does This Paper Address?
Natural language is full of words with multiple meanings (polysemy). Determining the correct meaning of a word based on its surrounding text is a classic Natural Language Processing (NLP) challenge known as Word Sense Disambiguation (WSD). Visual Word Sense Disambiguation (V-WSD) takes this a step further by incorporating visual information to help disambiguate word senses. This is particularly useful when textual context is limited or when the visual aspect of a word's meaning is crucial, such as in image captioning or retrieval. The challenge lies in effectively bridging the gap between textual descriptions and visual content, especially for words with subtle or context-dependent meanings.
Methods
The core innovation of this paper is the use of prompt engineering with a Large Language Model (LLM) to enhance the input for a multimodal model.
- Prompt Generation Module: The authors leverage ChatGPT to generate more descriptive and contextually rich prompts for ambiguous words. They frame the input as "What does the [WORD] mean in [CONTEXT]?" and use ChatGPT's response to add clarifying details. This aims to provide the multimodal model with a more robust textual signal.
- Multimodal Module: The CLIP (Contrastive Language-Image Pre-training) model is used as the backbone. CLIP is known for its ability to learn a shared embedding space for text and images, meaning similar text and image concepts are close in this space. The generated prompts and candidate images are encoded by CLIP, and their similarity is measured using cosine distance. The image with the highest similarity to the generated prompt is selected as the correct visual sense.
A key aspect is that this approach is designed to work in a zero-shot setting, meaning it doesn't require task-specific fine-tuning on the V-WSD dataset itself, making it more efficient.
Datasets
The paper evaluates its approach on a V-WSD task that includes data in English, Farsi, and Italian. For fine-tuning experiments, they utilize the MuWoSeD (Multimodal Word Sense Disambiguation) dataset.
Key Insights and Ideas
- LLMs as Prompt Enhancers: Using LLMs like ChatGPT to generate more detailed prompts is a powerful way to augment limited textual context for multimodal tasks. This goes beyond simply feeding raw text and allows the model to "understand" the nuances better.
- Zero-Shot Efficiency: The prompt generation approach allows for effective V-WSD without extensive task-specific fine-tuning, which is computationally cheaper and more adaptable to new domains.
- Multilingual Challenges: Translating contexts into English for Farsi and Italian inputs proved to be a more challenging setting, highlighting the complexities of cross-lingual V-WSD.
Key Results
The proposed zero-shot prompt generation approach achieved competitive results:
- English: Achieved a Mean Reciprocal Rank (MRR) of 0.79 in zero-shot settings. When fine-tuned on the task's training dataset, this improved to 0.80.
- Farsi: Achieved an MRR of 0.64.
- Italian: Achieved an MRR of 0.68.
- Overall Average: The model achieved an average MRR of 0.71 across all languages in zero-shot settings.
- Fine-tuning on the MuWoSeD dataset also showed improvements, with a pre-trained CLIP model on MuWoSeD achieving an MRR of 0.80 in zero-shot settings for English.
Limitations and Future Work
- ChatGPT Usage Limitations: The authors note that ChatGPT has limitations on the number of requests, which restricts the size of the input they can process.
- Translation Overhead: For Farsi and Italian, translating the textual contexts into English before feeding them to the CLIP model adds an extra step and potential for information loss.
- Future Work: The authors suggest exploring prompt generation in multilingual settings directly, rather than relying on translation, to further improve performance and efficiency.
Summary updated