Rutgers Multimedia Image Processing Lab at SemEval-2023 Task-1: Text-Augmentation-based Approach for Visual Word Sense Disambiguation

Keyi Li, Sen Yang, Chenyang Gao, Ivan MarsicACL 2023

TL;DR

This paper tackles the challenge of Visual Word Sense Disambiguation (VWSD), where the goal is to match an ambiguous word in a given text to its correct corresponding image from a set of candidates. The authors propose a novel text-augmentation strategy, combining GPT-3's generative capabilities with the CLIP model, to significantly improve disambiguation accuracy.

What Problem Does This Paper Address?

Human language is rife with polysemy – words that have multiple meanings. While humans easily disambiguate these words based on context, machines struggle. Visual Word Sense Disambiguation (VWSD) extends this challenge to the multimodal domain: given a target word with limited textual context, the task is to select the correct image that represents the intended meaning of that word. For instance, distinguishing an image of the "Andromeda tree" from an image of the "Andromeda galaxy" requires understanding the specific sense of "Andromeda" in the context of "tree." Existing methods often rely on word embeddings, which can struggle to capture the nuances of polysemous words. This paper aims to improve VWSD by enriching the textual context provided to vision-language models.

Methods

The core of the proposed approach lies in text augmentation to enhance the disambiguation capabilities of vision-language pre-trained (VLP) models. They explore three main text augmentation techniques:

  • Prompting: Prepending phrases like "This is a picture of..." to the target word or phrase. This leverages the observation that VLP models often perform better with more descriptive text.
  • WordNet Synonyms Set: Utilizing WordNet, a lexical database, to retrieve synonyms and their descriptions for the target word. They then select the most relevant description based on cosine similarity with the target phrase, using various pre-trained language models (BERT, Transformer-XL, MiniLMv2, MP-Net) for this selection.
  • Text Generation: Employing GPT-3 (specifically, the text-davinci-003 model) to generate a concise, single-sentence description of the target phrase. This method aims to produce highly relevant and contextually rich descriptions.

These augmented texts, along with the candidate images, are then fed into VLP models. The image with the highest cosine similarity to the augmented text embedding is selected. The authors experiment with several VLP models, including CLIP, ALBEF, BLIP, and BLIP-2, focusing on their zero-shot learning capabilities.

Datasets

The primary dataset used for evaluation is the SemEval-2023 Task-1 test dataset. This dataset comprises 8100 images across three languages: English, Farsi, and Italian. Each sample includes a target word, a target phrase (providing limited context), and ten candidate images. The paper also mentions that VLP models like CLIP are pre-trained on large-scale image-text datasets, such as the 400 million image-text pairs collected from the internet by CLIP.

Key Insights and Ideas

  • Text augmentation is crucial for VWSD: Simply using the target word or phrase is insufficient. Enriching the text with context significantly boosts performance across various VLP models.
  • GPT-3 text generation is highly effective: The descriptions generated by GPT-3 provided the most significant performance gains, outperforming WordNet-based augmentations. This suggests that generative models can produce more accurate and contextually relevant descriptions than curated lexical resources for this task.
  • CLIP excels in VWSD, even with limited context: Despite other VLP models showing strong performance on image-text retrieval benchmarks like Flickr30k, CLIP demonstrated superior performance in the VWSD task. The authors attribute this to CLIP's larger pre-training dataset, which makes it more robust for zero-shot learning on novel images and contexts.
  • Prompts offer marginal benefits: While prompting can improve performance, its impact is less significant than more substantial text augmentation methods like GPT-3 generation. This suggests that VLP models are already adept at understanding sentence-level context.
  • Multilingual handling requires careful consideration: Translating Farsi and Italian target phrases into English and using English-based pre-trained models for description selection yielded better results than using multilingual models directly for this step. This highlights the current strengths of English-centric language models.

Key Results

The combination of GPT-3 text generation and the CLIP model achieved the best performance. Specifically, they reported a hit rate (HIT) of 51.11 and a mean reciprocal rank (MRR) of 65.69 on the multilingual test sets. This represents a substantial improvement over baseline approaches. For instance, using only the target word with CLIP yielded HIT/MRR scores of 19.56/38.92.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the multilingual aspect of the dataset was handled by translation, and direct multilingual understanding might offer further improvements.
  • The performance of different CLIP variants was similar when working with comprehensive text descriptions (WordNet, GPT-3), possibly due to the limited size of the test dataset.
  • Future Work: The authors suggest several promising avenues:
  • Image Augmentation: Generating synthetic images based on the target phrase using state-of-the-art image generative models.
  • Image Captioning: Using pre-trained image captioning models to describe candidate images and then comparing these descriptions to the target phrase.
  • Ensemble Models: Combining the predictions of multiple VLP models to potentially boost overall performance.

Summary updated

BibTeX
@inproceedings{li2023rutgers,
  author = {Keyi Li and Sen Yang and Chenyang Gao and Ivan Marsic},
  title = {{Rutgers Multimedia Image Processing Lab at SemEval-2023 Task-1: Text-Augmentation-based Approach for Visual Word Sense Disambiguation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure