TL;DR
This paper tackles the challenge of Visual Word Sense Disambiguation (VWSD) by proposing a novel multi-modular, zero-shot approach that leverages pre-trained vision-language models and knowledge graphs. Their method effectively disambiguates word meanings in images, outperforming baselines and offering a flexible framework for future VWSD research.
What Problem Does This Paper Address?
The core problem addressed is Visual Word Sense Disambiguation (VWSD). In essence, given a word and its surrounding textual context, the task is to select the correct image from a set of candidates that visually represents the intended meaning of that word. This is particularly challenging because candidate images often lack explicit textual descriptions, and the target word itself can have multiple meanings. The authors highlight that existing models, while good at image description, often lack the "image-related background knowledge" needed for deeper contextual understanding, especially for terms related to scientific domains like plants and animals.
Methods
The teamPN approach is a multi-modular, zero-shot system designed for VWSD. It consists of two primary modules:
- Module 1: BLIP Model: This module utilizes the BLIP (Bootstrapping Language-Image Pre-training) model, a powerful vision-language pre-trained model. BLIP is chosen for its ability to handle unified image-grounded text understanding and generation, and its strong generalization capabilities in a zero-shot setting. The core of this module involves using BLIP's Image-Text Matching (ITM) head to compute similarity scores between contextual text and candidate images.
- Term Expansion: To enrich the contextual information, the authors employ term expansion. They leverage Wikipedia APIs to find synonyms and hypernyms for the input text. A semantic filter, powered by a sentence transformer (all-MiniLM-L6-v2), is used to order and filter these expanded terms semantically, ensuring relevance.
- Module 2: Multi-Modal Knowledge Graph Grounding (MMKG): This module augments the image understanding by grounding it with the VisualSem Knowledge Graph. CLIP embeddings are used to extract central entities from images, which are then matched against MMKG node definitions. A semantic search model ("msmarco-distilbert-base-tas-b") is used to match image contexts with complex words and their associated keywords to find relevant documents within the knowledge graph.
- Aggregation: The final prediction is an aggregation of the confidence scores from Module 1 and Module 2. A weighted average is used, with a higher weight given to Module 2's confidence score, as it was found to perform better when confident. The formula is:
Confidence = 0.6 * module2 + 0.4 * module1.
Datasets
The primary dataset used is the SemEval-2023 Task 1 Visual Word Sense Disambiguation dataset. This dataset includes:
- An initial trial set with 16 samples.
- A final training dataset with 12,869 samples, each mapping a text/contextual text pair to a target image from a set of 10 candidate images.
- A total of 12,999 unique images in the training data pool.
The paper also mentions using Wikipedia APIs for term expansion and the MS MARCO retrieval dataset for training the semantic search model.
Key Insights and Ideas
- Knowledge Graphs Enhance VWSD: Integrating a multi-modal knowledge graph (MMKG) significantly aids in disambiguating word senses, particularly for scientific terms where context can be nuanced.
- Term Expansion is Crucial for Nuance: Expanding the textual context with synonyms and hypernyms, especially from sources like Wikipedia, helps models better grasp the intended meaning, leading to improved retrieval performance.
- Semantic Filtering is Key for Expansion: Simply expanding terms isn't enough; a semantic filter is essential to prune irrelevant or noisy expansions, ensuring that the augmented context is genuinely helpful.
- Modular Approach Offers Flexibility: The multi-modular design allows for independent development and improvement of different components (e.g., vision-language models, knowledge graph integration, term expansion), making the overall system adaptable and extensible.
- Zero-Shot Learning is Viable: The success of BLIP in a zero-shot setting demonstrates the power of large pre-trained models for tasks where labeled data is limited or specific fine-tuning is not feasible.
Key Results
The paper reports several key performance metrics on the test dataset:
- Baseline: Top 1 Accuracy of 60.47%, MRR of 73.87%.
- BLIP ITM (Module 1): Top 1 Accuracy of 61.1%, MRR of 75.4%.
- BLIP ITM + TE (Term Expansion): Top 1 Accuracy of 62.4%, MRR of 75.2%.
- BLIP ITM + FTE (Fusion Term Expansion): Top 1 Accuracy of 64.7%, MRR of 77.6%.
- MMKG (Module 2): Top 1 Accuracy of 49.0%, MRR of 65.5%.
- MMKG + BLIP ITM + FTE (Aggregated): Top 1 Accuracy of 66.9%, MRR of 78.6%.
The aggregated model, combining BLIP ITM with Fusion Term Expansion and MMKG, achieved the highest performance, demonstrating the synergy between the modules.
Limitations and Future Work
- Limitations:
- The paper notes that in cases where Module 2 was highly confident, Module 1 struggled, suggesting potential areas for better integration or confidence calibration.
- While term expansion improved Top 1 accuracy, it didn't always improve MRR, indicating that the order of retrieval might not always be perfectly optimized by expansion alone.
- The authors acknowledge that a significant portion of the dataset involves scientific terms, where their model performed better, implying potential challenges with more general or abstract concepts.
- Future Work:
- The authors intend to incorporate more modules for knowledge augmentation and improved sense disambiguation.
- They hope their work inspires innovative uses of large-scale trained visual language models in zero-shot settings.
Summary updated