TL;DR
This paper tackles the challenge of Visual Word Sense Disambiguation (VWSD), where the goal is to match a word's correct meaning to an image from a set of options, using only minimal context. The authors explore various zero-shot approaches leveraging powerful pre-trained vision-language models (VLMs) and external knowledge sources like Wikipedia and Common Crawl, achieving competitive results without task-specific fine-tuning.
What Problem Does This Paper Address?
The core problem is disambiguating the meaning of a word when presented with an image and very limited textual context. Traditional Word Sense Disambiguation (WSD) relies on dictionaries or knowledge bases to resolve ambiguity within text. However, Visual Word Sense Disambiguation (VWSD) adds a visual dimension: given a target word and a short context (often just one or two words), the task is to select the correct image from a set of ten that illustrates the intended meaning. This is particularly challenging because the provided context is often minimal, sometimes consisting of rare or technical terms, and the images can be very similar, requiring nuanced understanding of both text and visuals. The SemEval-2023 Task 1 specifically targets this problem, requiring systems to perform effectively in a zero-shot manner, meaning without prior training on the specific task data.
Methods
The authors' central strategy is to augment the minimal provided context with information from external knowledge sources before feeding it into a pre-trained vision-language model (VLM). They primarily use CLIP (Contrastive Language-Image Pre-training) models, known for their strong zero-shot capabilities, to compute joint embeddings for text and images.
Their approaches include:
- Baseline (No External Knowledge): This method uses only the target word and context, experimenting with different "template strings" (e.g., "An image of a 'WORD' as in 'CONTEXT'") to form the textual input for CLIP.
- Wikipedia Summaries: They retrieve article summaries from Wikipedia for the target word and context, translate them to English if necessary, and use these enriched texts as input for CLIP.
- Common Crawl Sentences: This approach queries a large Common Crawl corpus for sentences containing the target word and context. They select the top-5 longest sentences (assuming more context) and use their embeddings.
- VisualSem: This method utilizes a multi-modal knowledge graph (KG) called VisualSem. They compute embeddings for KG nodes (based on images and glosses) and then use CLIP to find the best matching KG node for the VWSD sample's text or images. They explore different strategies for aggregating node embeddings (single vs. average).
A key aspect across all methods is the "zero-shot" nature, meaning no fine-tuning on the VWSD task data itself.
Datasets
- SemEval-2023 Task 1 Data: This includes a trial phase with 12,869 English samples and a final test phase with 463 English, 305 Italian, and 200 Farsi samples. Each sample consists of a target word, a context word, and ten images.
- Wikipedia: Used as an external knowledge source for text summaries.
- Common Crawl: A large-scale corpus (252B tokens) indexed via ElasticSearch, used for retrieving contextual sentences.
- VisualSem: A multi-modal knowledge graph with 90K nodes, 1.3M glosses, and 930K images.
Key Insights and Ideas
- External Knowledge is Crucial: The minimal context provided in VWSD samples necessitates external knowledge to achieve good performance, especially for non-English languages.
- Common Crawl Outperforms: Surprisingly, querying a large text corpus like Common Crawl for contextual sentences yielded the best results, even outperforming the multi-modal knowledge graph VisualSem. This suggests that simple, abundant textual context can be highly effective.
- Translation is Effective: Translating Italian and Farsi samples into English using Google Translate proved to be a viable strategy, enabling competitive performance across languages.
- Template Strings Matter: For the baseline, the choice of template string significantly impacts performance, with those including context information performing much better.
- Hit@3 vs. Hit@1: The significantly higher Hit@3 scores compared to Hit@1 suggest that multiple images in the set are often very similar, making disambiguation difficult and requiring expert-level judgment.
Key Results
- The Common Crawl approach was the best-performing method, achieving strong competitive results across all languages.
- The Baseline (No External Knowledge) showed strong performance on English data but a notable decrease for Italian and Farsi. Multi-lingual CLIP models significantly improved baseline performance on non-English data.
- Wikipedia Summaries provided substantial improvements for Italian and Farsi data, bringing them closer to English performance.
- VisualSem approaches did not perform well, yielding results worse than or equal to the baseline, which the authors found unexpected.
- The authors ranked 20th out of 56 systems in the SemEval competition, indicating their approaches were competitive.
Limitations and Future Work
- VisualSem Performance: The poor performance of VisualSem is a key limitation and requires further investigation. Potential causes include algorithmic flaws, the KG's relatively small size, or missed opportunities to leverage relational information within the graph.
- Wikipedia Usage: The authors suggest that using Wikipedia in its original languages (rather than translating to English) and incorporating visual information from Wikipedia could further improve results.
- Algorithmic Complexity: The VisualSem approach has many parameters and combinations, making detailed description challenging and suggesting a need for more streamlined implementations.
- Future Work: The authors aim to contribute meaningful insights to propel multi-modal machine learning tasks like VWSD. Further investigation into the VisualSem performance and exploring more sophisticated ways to integrate external knowledge are implied future directions.
Summary updated