TL;DR
This paper tackles Visual-Word Sense Disambiguation (V-WSD), a task requiring the selection of a clarifying image for an ambiguous word in a text. They leverage the powerful CLIP model, enhancing its performance through back-translation for text augmentation and exploring different text snippet inputs. Their approach achieved competitive results, particularly for Indo-European languages, highlighting CLIP's potential for multimodal understanding.
What Problem Does This Paper Address?
Visual-Word Sense Disambiguation (V-WSD) is a challenging task that sits at the intersection of natural language processing and computer vision. Imagine you have a sentence like "The bank is on the river." The word "bank" can refer to a financial institution or the edge of a river. In V-WSD, the goal is to pick the correct image from a set of candidates that clarifies which meaning of "bank" is intended. This is crucial for applications like multimodal machine translation, image captioning, and visual search. The core challenges are the inherent linguistic ambiguity of words and the variability of candidate images across different domains, making it difficult for models to consistently disambiguate and retrieve the right visual context.
Methods
The authors' approach centers around using the pre-trained CLIP (Contrastive Language-Image Pre-Training) model, known for its ability to map text and images into a shared embedding space. They explore three ways to present the text to CLIP:
- Full Phrase (FP): The entire sentence, including the ambiguous word and its context.
- Ambiguous Word (AW): Just the ambiguous word itself.
- Main Topic (MT): The sentence excluding the ambiguous word.
They found that using the Full Phrase yielded the best results, suggesting that the surrounding context is vital for disambiguation.
To further improve performance, they employed two key augmentation strategies:
- Text Augmentation via Back-Translation: This involves translating the original text into another language (e.g., Farsi, Italian, French, German) and then translating it back into English. This process introduces variations in phrasing and synonyms, enriching the training data and making the model more robust to different linguistic structures.
- Visual Augmentation: Standard image transformations like horizontal flipping, color jittering, and grey scaling were applied to the training images. This aims to make the visual encoder more resilient to variations in image appearance.
Additionally, they fine-tuned the CLIP model on the V-WSD dataset, specifically focusing on ambiguous terms, to better capture the nuances of the task.
Datasets
The V-WSD task is based on a bespoke dataset comprising 12,869 instances. Each instance includes a text snippet with an ambiguous word and 10 associated images. The dataset is multilingual, with English, Farsi, and Italian text snippets in the test set, although the training corpus is exclusively in English.
Key Insights and Ideas
- Context is King: Using the full phrase as input to CLIP significantly outperforms using only the ambiguous word or the main topic. This emphasizes the importance of surrounding text for disambiguation.
- Back-Translation Boosts Robustness: Augmenting text data through back-translation, especially with multiple languages, demonstrably improves performance, particularly for Indo-European languages.
- Fine-tuning Matters: Fine-tuning CLIP on the V-WSD dataset, especially on ambiguous terms, enhances its ability to discriminate between word senses.
- Multilingual Challenges: While the approach works well for Indo-European languages, performance dips for less represented language families like Indo-Iranian (e.g., Farsi), highlighting the need for more robust multilingual handling.
Key Results
The paper reports competitive results on the SemEval-2023 Task 1. Their best submission achieved:
- 40th rank out of 91 for the English test split.
- 28th rank out of 58 for the Farsi test split.
- 29th rank out of 64 for the Italian test split.
Specifically, using the full phrase as input and employing data augmentation (Submission #2) generally outperformed the baseline and Submission #1 (which used no augmentation). For instance, on the English test set, Submission #2 achieved a Hit Rate of 65.23% and an MRR of 78.30%, outperforming the baseline's 60.47% Hit Rate and 73.88% MRR.
Limitations and Future Work
The authors acknowledge that data augmentation, while generally beneficial, did not show significant improvement on the validation set and even caused a slight decrease in performance for Farsi. They attribute this to potential noise or redundancy introduced by augmentation, or the specific characteristics of the Farsi test set being dissimilar to the augmented English training data.
Future work includes:
- Exploring a truly multilingual CLIP model to avoid the need for translation during inference.
- Broadening the scope to include more Indo-Iranian and other underrepresented language families.
- Investigating the impact of individual back-translation languages and varying augmentation aggressiveness.
- Applying the system to other multimodal tasks like machine translation and visual question answering.
Summary updated