TL;DR
This paper tackles Visual Word Sense Disambiguation (V-WSD), a task requiring models to select the correct image for a target word within a given context. The authors explore multimodal approaches, primarily leveraging pre-trained models like CLIP and image captioning frameworks, to bridge the gap between textual and visual semantics. Their findings highlight the effectiveness of CLIP-based methods for this task, outperforming traditional image captioning approaches.
What Problem Does This Paper Address?
The core challenge addressed is Visual Word Sense Disambiguation (V-WSD). Traditional Word Sense Disambiguation (WSD) focuses solely on text, aiming to identify the correct meaning of a word in a given context. V-WSD extends this by incorporating visual information, requiring a model to select an image that best represents the target word's sense within its textual context. This is particularly challenging because words can have multiple meanings (polysemy), and disambiguating them often requires understanding subtle contextual cues that can be visually represented. The SemEval-2023 Task 1 provides a benchmark for this, presenting samples with a target word, a short context, and ten candidate images, one of which is the "gold" image.
Methods
The authors explored two primary categories of methods:
- Image Captioning Method: This approach involves using a pre-trained image captioning model (specifically, the OFA framework) to generate a textual description for each candidate image. These generated captions are then compared to the definitions of the target word's senses (obtained from WordNet) using sentence embeddings (Sentence-BERT) and cosine similarity. The synset with the highest similarity score is chosen as the predicted sense, and the corresponding image is selected.
- CLIP-based Methods: This approach leverages the power of Contrastive Language-Image Pre-training (CLIP) models. CLIP is trained to align image and text embeddings in a shared vector space. The authors used two versions of CLIP (ViT-B/32 and ViT-L/14) to generate embeddings for both the context text (with a "a photo of" prefix) and the candidate images. Similarity was initially measured using cosine similarity between these embeddings. They further explored more sophisticated methods for assigning weights to the elements of the CLIP vectors, including linear regression and a custom neural network architecture, to improve similarity measurement.
Datasets
The paper utilizes the SemEval-2023 Task 1 dataset for Visual Word Sense Disambiguation. This dataset includes:
- A trial set
- A training set
- A testing set (containing samples in English, Italian, and Farsi)
The authors also mention the use of WordNet for word sense definitions and Microsoft COCO Captions as the dataset on which the OFA image captioning framework was pre-trained.
Key Insights and Ideas
- CLIP's Dominance: CLIP-based methods significantly outperform the image captioning approach for V-WSD. This suggests that directly learning joint vision-language embeddings is more effective than generating text descriptions and then comparing them.
- The Power of Pre-training: The success of CLIP highlights the immense value of large-scale, multimodal pre-training on diverse image-text pairs. These models learn robust representations that capture nuanced semantic relationships.
- Beyond Simple Cosine: While basic cosine similarity with CLIP embeddings is strong, employing linear regression or neural networks to re-weight or combine these embeddings can further boost performance. This indicates that the raw CLIP embeddings might benefit from task-specific fine-tuning or feature fusion.
- Limitations of Textual Definitions: WordNet definitions, while useful, often lack visual detail or describe abstract concepts, making them less effective for visual disambiguation when paired with image captions.
- Dataset Bias: The authors speculate that the strong performance on the training set compared to the test set might be due to the training data for CLIP (and potentially the V-WSD training set itself) being sourced from the internet, leading to a higher overlap with images found via search engines.
Key Results
- Image Captioning Method: Achieved a hit rate of 56.8% and an MRR of 70.6 on the training set.
- CLIP (ViT-B/32) - Cosine: Achieved a hit rate of 74.9% and an MRR of 83.7 on the training set.
- CLIP (ViT-B/32) - Linear Regression: Achieved a hit rate of 79.3% and an MRR of 86.9 on the training set.
- CLIP (ViT-L/14) - Cosine: Achieved a hit rate of 79.5% and an MRR of 86.5 on the training set.
- CLIP (ViT-L/14) - Linear Regression: Achieved a hit rate of 86.0% and an MRR of 91.2 on the training set.
- Submitted Results (English Test Set):
- Cosine CLIP: 58.3% hit rate, 72.1 MRR (62nd place).
- Linear Regression CLIP: 59.2% hit rate, 73.0 MRR (57th place).
- Organizer's Baseline (CLIP-based): 60.5% hit rate, 73.9 MRR (54th place).
The CLIP-based methods, especially with linear regression, consistently outperformed the image captioning method by a significant margin. The ViT-L/14 CLIP model generally showed better performance than ViT-B/32.
Limitations and Future Work
Limitations: * The image captioning method's performance was hampered by the simplicity of generated captions and the abstract nature of some WordNet definitions. * The authors note a discrepancy between training and test set performance, suggesting potential dataset biases or differences in image sourcing. * The submitted results were slightly below the organizer's baseline, indicating room for further improvement.
Future Work: * The authors propose exploring models similar to CLIP that can directly generate visual features for word senses, aiming to improve V-WSD performance.
Summary updated