TL;DR
This paper explores using "quasi-symbolic" representations derived from multi-modal models like CLIP to improve Visual Word Sense Disambiguation (V-WSD). While these representations boosted performance on a development set, they surprisingly degraded results on the actual test set, suggesting a significant distribution shift.
What Problem Does This Paper Address?
Visual Word Sense Disambiguation (V-WSD) is the task of selecting the correct image from a set of candidates that best matches a given text query, especially when the query contains an ambiguous word. Traditional approaches often rely on image retrieval techniques. Modern multi-modal models like CLIP excel at zero-shot image classification and can map text and images into a shared latent space. However, these continuous, dense representations are often opaque ("lack translucency"). The authors' previous work showed that applying sparse coding to text-only word embeddings could yield more interpretable, "quasi-symbolic" representations that improved performance. This paper investigates whether this benefit extends to the multi-modal domain, specifically for V-WSD, by applying similar quasi-symbolic techniques to CLIP's latent representations.
Methods
The core idea is to transform the dense, continuous latent representations from a CLIP model into sparse, quasi-symbolic representations. This is achieved in two main steps:
- Dictionary Learning: The authors first learn a dictionary matrix ($D$) using the latent representations of images from the training set. This process involves solving a LASSO optimization to find sparse coefficients ($a_i$) that reconstruct the image embeddings ($h_i$). Crucially, they found that using only image embeddings for dictionary learning yielded better results than including text embeddings, suggesting that specialized latent dimensions for each modality are beneficial.
- Sparse Coding: Once the dictionary ($D$) is learned, they obtain the quasi-symbolic representation for any input (text or image) by solving another LASSO optimization. This maps the input's dense embedding ($h_j$) to a sparse vector ($a_j$) using the learned dictionary. The non-zero coefficients in this sparse vector indicate the presence of specific "quasi-symbolic" features. These features can be interpreted as discrete semantic categories.
For the V-WSD task, they then construct feature vectors for ranking candidate images. These vectors include:
- Cosine similarity between original CLIP embeddings.
- Elementwise product of original CLIP embeddings.
- Sum of elementwise products of sparse embeddings.
- Jaccard similarity between sets of quasi-symbolic features.
- Binary features indicating overlap or difference in quasi-symbolic dimensions between text and image representations.
These features are then fed into a logistic regression classifier to rank the candidate images.
Datasets
- Training Data: The paper uses the official SemEval-2023 Task 1 training dataset, consisting of 12,869 text-image pairs. For development, they split this into a reduced training set of 10,000 instances and a development set of 2,869 instances.
- Test Data: The official test set contained instances in English, Farsi, and Italian. The paper primarily focuses on the English test set for comparison with their development set.
Key Insights and Ideas
- Interpretability Boost: The quasi-symbolic representations offer a degree of human interpretability, allowing researchers to associate latent dimensions with semantic concepts.
- Modality Specialization: Learning a dictionary from image embeddings alone proved more effective than using both image and text embeddings, suggesting that distinct latent dimensions for different modalities are crucial for effective sparse representation.
- Distribution Shift Impact: The most striking finding is that features that significantly improved performance on the development set (drawn from the training data distribution) were detrimental to performance on the unseen test set. This strongly indicates a substantial distribution shift between the training/development data and the test data.
- Divergent Semantic Importance: Correlation analysis between models trained on different datasets revealed that the important quasi-symbolic features for the test set differed significantly from those for the training set, further supporting the distribution shift hypothesis.
Key Results
- Development Set Gains: The quasi-symbolic features provided a "substantial performance gain" on the development set, improving metrics like hit rate and MRR compared to baseline approaches (cosine similarity and elementwise products of dense embeddings).
- Test Set Degradation: Conversely, on the official test set, the addition of quasi-symbolic features led to a "substantial drop" in performance compared to the baselines. For example, the baseline hit rate on the test set was 0.702, while the extended baseline (including coordinatewise products) was 0.702, but the paper implies the quasi-symbolic features performed worse.
- Final Submission: The final submission, an ensemble of models trained on all official training instances (not just the 10K subset), achieved a hit rate of 0.678 and an MRR of 0.800 on the English test set.
Limitations and Future Work
- Distribution Shift: The primary limitation acknowledged is the performance degradation on the test set due to a distribution shift. The authors suggest that using more diverse training data, potentially from sources like MS COCO or GCC, could mitigate this issue.
- Hyperparameter Sensitivity: The performance of the quasi-symbolic representations is dependent on hyperparameters like $k$ (number of symbols) and $\lambda$ (regularization strength), which require careful tuning.
- Future Directions: The authors propose exploring alternative sources for training data to address the distribution shift and potentially investigating other methods for inducing interpretable representations.
Summary updated