ML Mob at SemEval-2023 Task 1: Probing CLIP on Visual Word-Sense Disambiguation

Clifton Poth, Martin Hentschel, Tobias Werner, Hannah Sterz, Leonard BongardACL 2023

TL;DR

This paper explores how well CLIP, a powerful vision-language model, can perform Visual Word Sense Disambiguation (VWS) by matching ambiguous words to images representing their different meanings. They found CLIP to be surprisingly capable even in a zero-shot setting, and while fine-tuning with parameter-efficient methods like adapters further improved performance on in-distribution data, transferring knowledge to multilingual test sets proved challenging, highlighting a gap in current WSD approaches.

What Problem Does This Paper Address?

Understanding the precise meaning of a word based on its context, known as Word Sense Disambiguation (WSD), is a cornerstone of natural language understanding. Traditionally, WSD has focused solely on text. However, the SemEval-2023 Task 1 introduced a novel "Visual WSD" challenge, requiring systems to disambiguate a word within its textual context by selecting the most appropriate image from a set of candidates, each representing a different word sense. This multimodal task bridges the gap between vision and language, pushing the boundaries of WSD. The authors investigate how well large pre-trained vision-language models like CLIP can tackle this new challenge, exploring various transfer learning strategies to adapt CLIP for this task.

Methods

The core of this work involves leveraging pre-trained CLIP models. The authors explore several transfer learning techniques to adapt CLIP for the Visual WSD task:

  • Zero-Shot Transfer: Evaluating CLIP's performance directly on the task without any fine-tuning. This leverages CLIP's inherent ability to understand relationships between images and text.
  • Projection Layer Tuning: Fine-tuning only the final projection layers of CLIP's vision and text encoders, keeping the rest of the model frozen. This is a relatively lightweight fine-tuning approach.
  • Adapter Methods: Introducing small, trainable modules (adapters) into the pre-trained CLIP architecture. These adapters are trained while the original CLIP weights remain frozen, making it a parameter-efficient way to adapt the model. The paper specifically mentions "Bottleneck adapters" and "Low-Rank Adaptation (LoRA)".
  • Partial Fine-Tuning: Fine-tuning only the text encoder while keeping the vision encoder frozen.
  • Full Fine-Tuning: Updating all parameters of both the vision and text encoders.
  • Intermediate Task Transfer: A more complex approach where adapters are first trained on a text-only WSD task (using the SemCor dataset) and then transferred to the Visual WSD task. This investigates whether knowledge from a related but different task can benefit performance.

Datasets

  • Training Data: The SemEval-2023 Visual-WSD task provides a labeled English-language training dataset consisting of 12,869 (word, context) pairs, each associated with 10 candidate images representing different word senses.
  • Test Data: The test split includes samples in three languages: English (463 pairs), Italian (305 pairs), and Farsi (200 pairs). All language splits share a common set of 8100 test images.
  • Intermediate Task Data: The SemCor dataset was used for intermediate task transfer, providing (word, context) pairs annotated with sense keys.

Key Insights and Ideas

  • CLIP's Strong Zero-Shot WSD Capability: The paper demonstrates that even without any task-specific training, CLIP models exhibit a remarkable ability to perform Visual WSD, significantly outperforming random baselines. This suggests that the multimodal representations learned by CLIP are inherently well-suited for understanding word senses in a visual context.
  • Parameter-Efficient Fine-Tuning is Effective (for In-Distribution Data): For data similar to the training distribution, parameter-efficient methods like adapter tuning and projection layer tuning achieve performance comparable to full fine-tuning. This is a crucial finding, as it implies that adapting large models like CLIP for new tasks doesn't necessarily require computationally expensive full fine-tuning.
  • Multilingual Transfer is Hard: A significant challenge arises when transferring knowledge to the multilingual test set. While fine-tuning improves performance on the English validation set, the gains on the multilingual test set are minimal. This suggests that current methods struggle to bridge the gap between monolingual training and multilingual testing in this multimodal WSD setting.
  • Data Distribution Shift: The authors hypothesize that a key reason for the limited transferability to the test set is a distribution shift. They observe that the training set contains words with fewer distinct senses (synsets) compared to the test set, which has words with a much wider range of meanings. This difference in ambiguity levels likely makes generalization difficult.

Key Results

  • Zero-Shot Performance: OpenAI ViT-L/14-336 and LAION ViT-G/14 achieved the best zero-shot accuracy on the English validation set, around 81% and 80.7% respectively, significantly outperforming a random baseline (around 10.5%).
  • Fine-Tuning Gains: On the English validation set, fine-tuning methods (including adapters and full fine-tuning) consistently improved accuracy by about 12 percentage points over zero-shot performance for the LAION XLM-R Base model.
  • Best Overall Performance: The best performance was achieved by full fine-tuning of the LAION XLM-R Large model on the English validation set, reaching 91.91% accuracy and 95.37% MRR.
  • Multilingual Test Set Results: On the multilingual test set, fine-tuning generally showed only marginal improvements over zero-shot performance. For example, the LAION XLM-R Base model's average accuracy on the test set was around 52.6% in zero-shot, and fine-tuning methods only nudged this up slightly, with some even performing worse.
  • Intermediate Task Transfer: The SemCor adapter approach did not yield substantial benefits, performing similarly to other fine-tuning methods and even being outperformed by LoRA in some cases.

Limitations and Future Work

  • Limited Knowledge Transfer to Multilingual Test Sets: The paper explicitly acknowledges that knowledge transfer to the multilingual test set is "non-trivial." The improvements from fine-tuning on the training set do not translate well to the test set, especially across languages.
  • Data Distribution Shift: The authors identify the difference in the distribution of WordNet synsets between the training and test sets as a potential cause for the poor generalization. They suggest that systems fine-tuned on the specific characteristics of the training split may struggle with the increased ambiguity in the test split.
  • Future Work: The authors suggest that further investigation is needed to understand and overcome the challenges of knowledge transfer in multimodal WSD. This could involve exploring more sophisticated techniques beyond the simple ones tested, or developing methods that are more robust to data distribution shifts. They also leave the investigation of the observed data distribution effects to future work.

Summary updated

BibTeX
@inproceedings{poth2023ml,
  author = {Clifton Poth and Martin Hentschel and Tobias Werner and Hannah Sterz and Leonard Bongard},
  title = {{ML Mob at SemEval-2023 Task 1: Probing CLIP on Visual Word-Sense Disambiguation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure