PMCoders at SemEval-2023 Task 1: RAltCLIP: Use Relative AltCLIP Features to Rank

Mohammad Javad Pirhadi, Motahhare Mirzaei, Mohammad Reza Mohammadi, Sauleh EetemadiACL 2023

TL;DR

This paper introduces RAltCLIP, a novel approach for Visual Word Sense Disambiguation (VWSD) that leverages relative features from AltCLIP to improve image-text matching. By comparing image features in relation to the text, RAltCLIP significantly outperforms the official baseline, demonstrating the power of relative feature comparison for this task.

What Problem Does This Paper Address?

The Visual Word Sense Disambiguation (VWSD) task aims to identify the most relevant image for a given word within a specific textual context, chosen from a set of candidate images. This is essentially a specialized form of text-to-image retrieval. A key challenge in VWSD is accurately capturing the nuanced meaning of a word as it relates to visual information, especially when multiple images might share superficial similarities with the text. Existing methods often rely on direct feature comparisons, which can be susceptible to noise and less effective at highlighting subtle differences.

Methods

The core of the RAltCLIP system lies in its innovative use of relative features. Instead of directly comparing the feature vector of a phrase with the feature vectors of candidate images, RAltCLIP first extracts features using a pre-trained AltCLIP model (which itself enhances CLIP by using XL-R LM for better language understanding).

The novelty comes in the subsequent stage:

  1. AltCLIP Feature Extraction: Both the input phrase and the 10 candidate images are encoded into feature vectors using a frozen AltCLIP model.
  2. "Word Without Ambiguous Word" (WOAW) Feature: An additional feature vector is generated by removing the ambiguous word from the phrase and encoding the remaining text. This aims to provide a stronger contextual clue (e.g., "tree" instead of "Andromeda tree" to disambiguate between a tree and a galaxy).
  3. Relative Feature Vector (RFV) Generation: A 3-layer standard transformer encoder is employed to transform the independent AltCLIP features (phrase, WOAW, and image features) into relative feature vectors. This transformer learns to represent how each image feature relates to the phrase and WOAW features.
  4. Cosine Similarity for Ranking: The final relevance score for each image is calculated by computing the cosine similarity between the relative feature vector of the phrase and the relative feature vector of each image. This relative comparison helps the model focus on distinguishing characteristics.

The authors also explored different loss functions (cross-entropy and cosine embedding loss) and the impact of pretraining on a subset of the LAION-5B dataset to improve generalization.

Datasets

  • Official VWSD Dataset: This dataset consists of training, trial, and test splits for English, Farsi, and Italian. Each sample includes an ambiguous word, a phrase, and 10 candidate images.
  • LAION-5B Subset: A subset of the massive LAION-5B dataset was used for pretraining. This subset was filtered for caption length, word count, and image dimensions, resulting in approximately 45,000 image-caption pairs per language. This pretraining aims to enhance the model's generalization capabilities.

Key Insights and Ideas

  • Relative Features Matter: The most significant insight is that comparing image features relative to the text (phrase and WOAW) is more effective than direct comparison. This allows the model to better discern subtle differences, like color variations in otherwise identical objects.
  • WOAW Feature's Dual Nature: The "Word Without Ambiguous Word" (WOAW) feature can be highly beneficial, providing strong disambiguation cues. However, it also carries a risk of overfitting, where the model might over-rely on this simplified context, as illustrated by the "Jungle lion" example.
  • Pretraining for Generalization: Pretraining on a large dataset like LAION-5B significantly improves performance, especially for languages like Farsi and Italian where the original dataset might have less coverage.
  • Translated Data Nuances: While translated data can help models learn new languages, it introduces noise that can degrade performance, particularly when combined with pretraining that has already established strong language representations.

Key Results

The RAltCLIP system achieved state-of-the-art results on the SemEval-2023 Task 1 VWSD. The proposed system beat the official baseline by 2.41% in macro-averaged hit rate and 2.39% in macro-averaged MRR. Specifically, their best configuration reached 39.61% hit rate and 56.78% MRR, outperforming the official baseline's 37.20% hit rate and 54.39% MRR. The inclusion of the 12th feature vector (WOAW) generally improved performance, though careful handling is needed to mitigate overfitting.

Limitations and Future Work

  • Overfitting with WOAW: The authors acknowledge that the WOAW feature can lead to overfitting. They suggest penalizing its usage to ensure it's only employed when genuinely helpful.
  • Pretraining Data Impact: While pretraining improved performance, the authors believe that using even more data for pretraining could further enhance generalization and potentially address the slight decrease in English performance observed after pretraining.
  • Future Directions:
    • Adapting the system for image-to-text retrieval by pretraining on massive datasets.
  • Exploring alternative methods for selecting pretraining samples, perhaps based on caption feature vector similarity rather than image similarity.
  • Developing more sophisticated strategies for word dropping during pretraining, beyond random selection.

Summary updated

BibTeX
@inproceedings{pirhadi2023pmcoders,
  author = {Mohammad Javad Pirhadi and Motahhare Mirzaei and Mohammad Reza Mohammadi and Sauleh Eetemadi},
  title = {{PMCoders at SemEval-2023 Task 1: RAltCLIP: Use Relative AltCLIP Features to Rank}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure