TL;DR
This paper tackles the challenge of Visual Word Sense Disambiguation (VWSD) by proposing three novel approaches that learn joint word-sense and image embeddings. Their supervised and self-supervised methods, leveraging Siamese networks and Bayesian Personalized Ranking, significantly outperform unsupervised baselines, demonstrating the effectiveness of learning rich representations for associating ambiguous words with their correct visual context.
What Problem Does This Paper Address?
Natural language is rife with ambiguity, where a single word can have multiple meanings depending on its context. This phenomenon, known as Word Sense Disambiguation (WSD), is a long-standing problem in Natural Language Processing (NLP). While WSD has been extensively studied, applying it to downstream tasks like image retrieval presents unique challenges. SemEval-2023 Task 1 specifically addresses this by framing WSD as a problem of selecting the correct image for a given word and its textual context. The core challenge lies in effectively disambiguating the word's meaning and then mapping that disambiguated sense to the appropriate visual representation. This requires models that can understand both linguistic nuances and visual semantics, bridging the gap between text and images.
Methods
The authors propose three distinct approaches to tackle the Visual Word Sense Disambiguation task:
- Sense-Caption Similarity Approach (Unsupervised): This method leverages WordNet to identify potential senses for ambiguous words within a context. It then generates image captions using a pre-trained Neural Image Caption (NIC) model. The similarity between the word senses and the generated image captions is computed using cosine similarity, and the image with the highest similarity is selected. This approach aims to align word senses with visual descriptions without explicit labeled data for the WSD task itself.
- Binary Classification Approach (Supervised): This approach frames the problem as a binary classification task. For each word-image pair, the model predicts whether the image corresponds to the intended meaning of the word (positive) or not (negative). It utilizes a Siamese neural network architecture. Word senses are obtained using WordNet and then embedded using the pre-trained Sentence-BERT model. Image embeddings are extracted using methods like ORB, KAZE, or the VGG16 model. These word-sense and image embeddings are projected into a shared latent space and then fed into the Siamese network to learn their similarity.
- Contrastive Learning Approach (Self-Supervised): This method also learns joint word-sense and image embeddings but employs a self-supervised contrastive learning strategy based on the Bayesian Personalized Ranking (BPR) framework. Given a word, a positive image (correctly corresponding to its intended meaning) and negative images are used. The model learns to rank the positive image higher than the negative images. This approach aims to learn discriminative embeddings by pushing embeddings of related word-sense and image pairs closer together in the latent space while pushing unrelated pairs further apart.
Datasets
The paper utilizes the dataset provided for SemEval-2023 Task 1 for training and evaluation. This dataset consists of ambiguous words, their textual contexts, and a set of candidate images. The authors split the provided training set into an 80% training set, a 10% validation set, and a 10% test set for their experiments. They also mention using a trial set provided during the trial phase and a test set during the evaluation phase.
Key Insights and Ideas
- The power of joint embeddings: The success of the supervised and self-supervised approaches highlights the importance of learning unified representations for both word senses and images. This allows for a direct comparison and association between linguistic meaning and visual content.
- Supervised and self-supervised methods excel: The paper clearly demonstrates that explicitly learning from labeled data (supervised) or using clever self-supervision strategies (contrastive learning) significantly outperforms unsupervised methods in this visual WSD task.
- Image features matter: The choice of image feature extraction method (ORB, KAZE, VGG16) has a notable impact on performance, with VGG16 generally yielding better results for the supervised and self-supervised approaches on validation and test sets.
- Negative sampling is crucial: The experiments on varying negative sample sizes reveal that using all available negative images might not always be optimal. A carefully chosen subset of negative samples can lead to better performance, suggesting that imbalanced data can indeed degrade model accuracy.
Key Results
The results, summarized in Table 1, show that the supervised (Siamese) and self-supervised (BPR) approaches, particularly when using VGG16 for image features, significantly outperformed the unsupervised Sense-caption approach and the baseline MLP model on the validation and test sets. For instance, Siamese using VGG16 achieved an HR of 48.02% and MRR of 66.52% on the split test set, while BPR using VGG16 achieved 48.17% HR and 66.04% MRR. The Sense-caption approach struggled, achieving only 12.14% HR on the trial set, attributed to limitations in its knowledge base and spaCy's word recognition. The paper also found that using a negative sample size of 5 generally yielded better overall performance for the Siamese model compared to using more negative samples.
Limitations and Future Work
The authors acknowledge that the unsupervised Sense-caption approach underperformed due to limitations in the adopted knowledge base (WordNet) and the reliance on spaCy's English model, which failed to recognize a significant percentage of words. They also noted that the trial set, with only 16 samples, might not fully represent actual model performance. The paper suggests that the accuracy of the Siamese model decreased when using all negative images, indicating a potential imbalanced data problem that could be further investigated. Future work could explore more robust knowledge bases, advanced NLP techniques for word sense disambiguation, and more sophisticated methods for handling imbalanced data in visual WSD tasks.
Summary updated