Rahul Patil at SemEval-2023 Task 1: V-WSD: Visual Word Sense Disambiguation

Rahul Patil, Pinal Patel, Charin Patel, Mangal VermaACL 2023

TL;DR

This paper tackles Visual Word Sense Disambiguation (V-WSD), a task that requires matching an ambiguous word to the correct image from a set of candidates, considering its textual context. The authors propose an ensemble of two neural network systems, achieving a strong 4th overall rank in SemEval-2023, with particularly impressive results for English and Italian.

What Problem Does This Paper Address?

Understanding the meaning of words is a fundamental challenge in natural language processing (NLP). However, many words have multiple meanings (polysemy), and disambiguating them based solely on text can be difficult. Visual Word Sense Disambiguation (V-WSD) takes this a step further by incorporating visual information. The goal is to identify the correct image that represents a word's meaning within a given textual context. For instance, the word "bank" could refer to a financial institution or the side of a river. V-WSD aims to pick the right image based on context like "bank erosion." This task is crucial for improving multimodal applications like image captioning and visual question answering.

Methods

The authors developed an ensemble of two main approaches:

  • Fine-tuned CLIP: They took OpenAI's CLIP (Contrastive Language-Image Pre-Training) model and further trained it on additional data. CLIP is designed to learn relationships between text and images by mapping them into a shared feature space.
  • Joint Embedding: This approach maps image and text features into a common vector space, allowing for direct comparison. They used a ResNet101 for image encoding and a multilingual SBERT model for text encoding, both reduced to 512-dimensional vectors. Cosine similarity was used to compare embeddings, with cross-entropy as the loss function.

For multilingual support, they utilized translation models to convert Italian and Farsi text into English. They also augmented text data by generating multiple sentence variations for each word-context pair.

Datasets

The primary datasets used were:

  • SemEval-2023 Task 1 (V-WSD) datasets: These included trial and train sets with English language data, and a test set with English, Italian, and Farsi.
  • MS-COCO dataset: Used for its English captions.
  • Wikipedia Image Text (WIT) dataset: A large multimodal and multilingual dataset sourced from Wikipedia, providing English, Farsi, and Italian data.

Key Insights and Ideas

  • Ensembling for Robustness: Combining different approaches (fine-tuned CLIP and joint embedding) proved effective, leading to a strong overall performance.
  • Multilingual Challenges: The system struggled with Farsi, highlighting the difficulties in cross-lingual transfer and the need for more sophisticated multilingual models or data for such languages.
  • Text Augmentation: Generating diverse textual descriptions for word-context pairs helped improve the model's understanding.
  • Overfitting for Performance: In the case of fine-tuning CLIP, the authors intentionally overfitted the model to the training data, which surprisingly led to enhanced performance on the test set, despite an increase in validation loss. This suggests that for this specific task and model, a degree of overfitting might be beneficial.

Key Results

The proposed system achieved:

  • 4th overall rank in the SemEval-2023 V-WSD competition with a Hit Rate (HR) of 69.805 and Mean Reciprocal Rank (MRR) of 78.230.
  • 2nd rank in English with HR of 83.153 and MRR of 88.802.
  • 1st rank in Italian with HR of 84.262 and MRR of 89.048.
  • 16th rank in Farsi with HR of 42.000 and MRR of 56.841, indicating the aforementioned challenges with this language.

Limitations and Future Work

The authors acknowledge that their system struggled with Farsi language understanding and Farsi-to-English conversion. They suggest that improving Farsi performance might require more sophisticated multilingual models or additional data. They also believe their solution has zero-shot learning capabilities and can generalize to unseen data. Future work could explore more advanced multilingual models and investigate techniques to better handle cross-lingual transfer.

Summary updated

BibTeX
@inproceedings{patil2023rahul,
  author = {Rahul Patil and Pinal Patel and Charin Patel and Mangal Verma},
  title = {{Rahul Patil at SemEval-2023 Task 1: V-WSD: Visual Word Sense Disambiguation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure