SLT at SemEval-2023 Task 1: Enhancing Visual Word Sense Disambiguation through Image Text Retrieval using BLIP

Mohammadreza Molavi, Hossein ZeinaliACL 2023

TL;DR

This paper tackles the Visual Word Sense Disambiguation (VWSD) challenge by fine-tuning the powerful BLIP vision-language model. By leveraging BLIP's pre-trained capabilities and employing a candidate selection strategy, the authors achieve strong performance on the SemEval-2023 Task 1, demonstrating the effectiveness of pre-trained models for complex multimodal tasks.

What Problem Does This Paper Address?

Visual Word Sense Disambiguation (VWSD) is a crucial task in bridging the gap between text and images. It involves identifying the correct meaning of an ambiguous word within a given textual context and then selecting the corresponding image from a set of candidates. This is vital for applications like image captioning and visual question answering, where understanding the precise meaning of words in relation to visual information is paramount. The SemEval-2023 Task 1 specifically focuses on this challenge, requiring systems to accurately match textual descriptions with visual content.

Methods

The core of this work lies in adapting the Bootstrapping Language-Image Pre-training (BLIP) model for the VWSD task. BLIP is a state-of-the-art vision-language pre-training framework known for its ability to learn joint representations of images and text. The authors fine-tune BLIP using two key objectives:

  • Image-Text Contrastive Loss (ITC): This loss function aims to align the feature spaces of visual and textual representations. It encourages similar embeddings for matching image-text pairs and dissimilar embeddings for non-matching pairs.
  • Image-Text Matching Loss (ITM): This objective trains an image-grounded text encoder to develop a multimodal representation that captures fine-grained alignment. It's a binary classification task where the model predicts whether an image-text pair is a match or not.

To improve inference speed, the system incorporates a candidate selection approach. First, a subset of potential image candidates is selected based on their similarity in the shared embedding space. Then, these selected candidates are re-ranked using their pairwise ITM scores, leading to a more efficient and accurate selection process.

Datasets

The primary dataset used is the SemEval-2023 Task 1 VWSD dataset. This dataset comprises 12,869 ambiguous textual phrases and 12,999 associated images, totaling approximately 17GB. The dataset was divided into training, validation, and test sets, with the test set containing challenges in English, Persian (Farsi), and Italian languages. For non-English languages, Google Translate was used to convert the text into English for processing.

Key Insights and Ideas

  • Leveraging Pre-trained Power: The success of fine-tuning large-scale pre-trained vision-language models like BLIP is a significant takeaway. These models have already learned rich multimodal representations that can be effectively adapted to specific downstream tasks.
  • ITC and ITM Synergy: The combination of contrastive and matching losses proves effective in aligning image and text representations for the VWSD task, enabling the model to better understand semantic relationships.
  • Efficiency through Candidate Selection: The proposed candidate selection strategy significantly speeds up the inference process without a substantial loss in performance, making the system more practical for real-world applications.
  • Multilingual Handling: The paper demonstrates a practical approach to handling multilingual data in vision-language tasks by utilizing machine translation, showing that even with this intermediate step, good performance can be achieved.

Key Results

The proposed system achieved a Mean Reciprocal Rank (MRR) of 67.91% on average across all languages in the challenge test set. Notably, the system significantly outperformed the CLIP baseline, achieving 72.05% MRR in English compared to CLIP's 66.8%. The system also showed substantial improvements in Persian (63.14% vs. 38.9%) and Italian (68.55% vs. 37.5%). The evaluation of different batch sizes indicated that a batch size of 32 yielded the optimal MRR of 74.67%.

Limitations and Future Work

A key limitation acknowledged is the reliance on Google Translate for handling non-English languages. While this approach yielded good results, it introduces an additional layer of potential error and may not fully capture the nuances of each language. Future work could explore end-to-end multilingual vision-language models that are trained directly on diverse language data. The authors also suggest that their approach has potential applications in broader areas like image retrieval, recommendation systems, and virtual assistants, indicating avenues for further research and development.

Summary updated

BibTeX
@inproceedings{molavi2023slt,
  author = {Mohammadreza Molavi and Hossein Zeinali},
  title = {{SLT at SemEval-2023 Task 1: Enhancing Visual Word Sense Disambiguation through Image Text Retrieval using BLIP}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure