GPL at SemEval-2023 Task 1: WordNet and CLIP to Disambiguate Images

Shibingfeng Zhang, Shantanu Nath, Davide MazzaccaraACL 2023

TL;DR

This paper tackles the Visual Word Sense Disambiguation (VWSD) task, aiming to select the correct image for a word based on its context. They propose a novel approach that augments textual context using WordNet and Wikipedia, then leverages multimodal models like CLIP and ViLT for image selection, achieving a strong 16th place in the SemEval-2023 competition.

What Problem Does This Paper Address?

Many common words have multiple meanings (polysemy). Traditional Word Sense Disambiguation (WSD) aims to identify the correct meaning of a word within a given text. Visual Word Sense Disambiguation (VWSD) extends this by incorporating visual information, requiring the selection of the correct image that corresponds to a word's sense in a specific context. The SemEval-2023 Task 1 presented a unique challenge: given a target word, its short textual context, and a set of candidate images, participants had to choose the image that best matched the word's intended meaning. This is particularly challenging because the provided textual context is often very brief, making it difficult to disambiguate the word's sense solely based on text.

Methods

The authors propose a two-step strategy:

  1. Text Augmentation: To overcome the limited textual context, they enrich it using external resources.
  2. WordNet: They leverage WordNet's synsets (sets of synonyms representing a concept) and glosses (definitions). To select the most relevant synset for augmentation, they encode the synsets and the given context using Sentence Transformers (sBERT) and choose the synset with the highest cosine similarity to the context.
  3. Wikipedia: For cases where WordNet might not provide sufficient disambiguation, especially for taxonomic terms (e.g., "genus," "species"), they use Wikipedia page summaries.
  4. Italian Dictionary: For the Italian dataset, they initially used an online Italian dictionary, falling back to Wikipedia if the word wasn't found.
  5. Image Retrieval: After augmenting the text, they use pre-trained language-vision models in a zero-shot setting to select the correct image.
  6. CLIP (Contrastive Language-Image Pre-Training): This model, trained on a large dataset of image-text pairs, learns to align visual and textual representations.
  7. ViLT (Vision-and-language Transformer): This is a single-stream multimodal transformer model designed to learn relationships between images and text.

They experimented with both models, comparing their performance with and without text augmentation.

Datasets

  • SemEval-2023 Task 1 Datasets: The authors evaluated their system on both English and Italian datasets provided for the task. These datasets consist of target words, short contexts, and 10 candidate images per instance.
  • WordNet: Used as a knowledge base for text augmentation.
  • Wikipedia: Used as a source for text augmentation.
  • Online Italian Dictionary: Used for text augmentation on the Italian dataset.

Key Insights and Ideas

  • Text augmentation is crucial for multimodal tasks with limited context: The significant performance boost observed with CLIP on the English dataset strongly suggests that enriching the textual input is vital when the original context is sparse.
  • CLIP's robustness to augmented text: CLIP's architecture, with separate encoders and a contrastive learning approach, seems to make it more resilient and effective at handling potentially noisy or varied textual inputs compared to ViLT.
  • WordNet's superiority over Wikipedia for augmentation (in this context): While Wikipedia provided useful information, the results indicated that WordNet's fine-grained synsets and glosses were more beneficial for improving the disambiguation accuracy with CLIP on the English task.
  • Model architecture matters: The paper speculates that CLIP's architecture might be better suited for capturing semantic relationships between words and images, leading to better performance than ViLT.
  • Augmentation isn't universally beneficial: The negative impact of augmentation on the Italian dataset highlights that the effectiveness of augmentation strategies can be dataset- and model-dependent.

Key Results

  • English Dataset:
    • The best performing model was the zero-shot CLIP with WordNet augmentation, achieving a hit rate of 68.47% and a mean reciprocal rank (mrr) of 79.17%.
  • Without augmentation, CLIP achieved a hit rate of 58.96% and mrr of 72.73%.
  • ViLT performed significantly worse, especially with augmentation.
  • Italian Dataset:
    • Surprisingly, text augmentation decreased performance for CLIP on the Italian dataset, with a hit rate of 32.46% and mrr of 51.23% (compared to 37.70% and 55.66% without augmentation).
  • Ranking: Their system was ranked 16th out of 50 submissions for the English language in SemEval-2023 Task 1.

Limitations and Future Work

  • Augmentation effectiveness varies: The paper acknowledges that text augmentation doesn't always lead to improved performance, as seen in the Italian dataset results. They attribute this to the different augmentation resources used (dictionary vs. WordNet/Wikipedia).
  • Fine-tuning did not help: The authors mention that fine-tuning the CLIP model did not improve its performance on this task.
  • Future work could explore:
    • Investigating why augmentation negatively impacted the Italian dataset and exploring alternative augmentation strategies for different languages or resource availability.
  • Further exploring the architectural differences between CLIP and ViLT and their implications for multimodal understanding.
  • Developing more robust methods for selecting the most beneficial augmentation strategy for a given task and model.

Summary updated

BibTeX
@inproceedings{zhang2023gpl,
  author = {Shibingfeng Zhang and Shantanu Nath and Davide Mazzaccara},
  title = {{GPL at SemEval-2023 Task 1: WordNet and CLIP to Disambiguate Images}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure