Visual word sense disambiguation asks a system to pick, from ten candidate images, the one that shows the intended meaning of an ambiguous word given only a tiny context such as "passenger coach" or "music keyboard." The current recipe is to expand that short phrase into a longer description, using a dictionary, Wikipedia, web text or a large language model, and then rank images by how closely they match it in CLIP, a model that places images and text in one shared embedding space. This recipe has a blind spot. The description is written without looking at the candidates. The benchmark, however, deliberately fills the candidate set with images from the same domain as the answer: pianos and violins next to the synthesizer. Models therefore fall into what the task organizers call a "domain trap" (SemEval-2023 Task 1). A definition that says "an electronic musical instrument" is equally compatible with every distractor.
We propose to first work out which competing concepts are actually present among the ten images. We then gather, from several heterogeneous knowledge sources, the visual attributes that separate the intended sense from those specific competitors, and rank images by those attributes rather than by a generic definition. Put in two sentences: label the distractors, then ask the knowledge sources what visibly distinguishes the target from them. Attribute statistics from visual corpora, property edges from commonsense graphs, dictionary glosses and language-model descriptions each contribute candidate attributes, and contrast against the present competitors decides which ones count. The method needs no training, which matters because every reported attempt to fine-tune on this task's automatically generated training data hurt the human-curated test set.
Core Claim
The claim is that the useful unit of external knowledge for this task is not a sense definition but a contrastive attribute. A contrastive attribute is a visually checkable property that holds for the target sense and fails for the competitors that are actually in the candidate set. Two things follow if this is right. Errors on same-domain distractors should drop more than errors on random distractors. And it should explain a contradiction in the literature. GPL, which added WordNet glosses to CLIP, raised English hit rate from 58.96% to 68.47%. UAlberta, which added BabelNet glosses, found they decreased performance. Our hypothesis is that glosses help only when they happen to contain visually discriminative content against the present distractors, and add noise otherwise. That hypothesis is measurable.
Relation to Prior Work
Three groups of work frame the proposal.
The first group expands the query text before CLIP scoring, and it motivates the proposal. GPL picks a WordNet gloss by sentence-embedding similarity, with Wikipedia as a fallback. LT tests Wikipedia summaries, Common Crawl sentences and the VisualSem multimodal graph, but only one at a time; VisualSem did no better than no knowledge at all. Language Models as Knowledge Bases asks Vicuna-13B or GPT-3.5 for a definition and reports over 72% accuracy with CLIP-LAION. It also adds a penalty for "majority bias," meaning generic images that rank high for many queries. All three generate text blind to the candidates. Our method differs on exactly that point: the knowledge we retrieve is conditioned on which competitors appear among the ten images.
The second group supplies the method. EDAM combines WordNet/Wiktionary definitions, VisualGenome sensory attributes and ConceptNet into sparse attribute vectors. It decides whether a feature separates a pivot concept from a comparison concept, and shows that the visual source recovers properties such as shape and colour that text omits because people rarely write them down. We reuse its triangulation idea, but apply it to continuous CLIP scores over real images rather than symbolic concept pairs. Iterative multi-source retrieval for CommonsenseQA found dictionary definitions were the strongest single source (+5.5%, against +2.3% for ConceptNet and +0.9% for Wikipedia), which is why glosses stay in our pool.
The third group constrains the design. G-SAP reports that adding more graph-derived prompt content saturates and then hurts. That warns against dumping every retrieved fact into the query, so we weight attributes by contrast instead of concatenating them. Silver-to-gold distribution shift wrecked fine-tuning in UAlberta, GPL and ML Mob. Dictionary learning on CLIP features in SzegedAI fell from a 0.702 baseline to 0.678. Together these rule out a trained scorer and push us to a training-free one with at most two tuned scalars.
The closest connecting work is FCLL, the task winner. It builds a BabelNet-derived knowledge base and trains contrastive encoders on it. It uses knowledge as training data, not as per-instance evidence about the distractors.
Research Program
The method runs in three stages.
The first stage builds a competitor inventory. For each instance, we translate non-English queries to English for lookup only, following UAlberta's finding that translation beats native-language models. We then collect the other senses of the target word from WordNet and BabelNet, and the domain siblings of the intended sense, meaning WordNet co-hyponyms and ConceptNet nodes that share an IsA parent. Each of the ten candidate images is then soft-labelled over the intended sense and this inventory, using CLIP similarity to each concept's gloss. We keep the competitors that hold some image's top label. This produces the list of concepts the method actually has to beat.
The second stage collects and filters attributes. For the intended sense and each present competitor, we pool candidate attributes from four sources: VisualGenome attribute and part co-occurrence, ConceptNet HasProperty, PartOf and MadeOf edges, noun phrases in the gloss, and short attribute lists generated by a language model. Each attribute $a$ receives a contrast weight $w_a$. This weight is the difference between its source-aggregated support for the target and its maximum support among present competitors, clipped at zero. Attributes that the target shares with its distractors therefore get no weight.
The third stage scores the images. Each image $i$ is scored as
$$\mathrm{score}(i) = s(q, i) + \lambda \sum_{a} w_a \big( s(a, i) - \tfrac{1}{10}\textstyle\sum_{j} s(a, j) \big),$$
where $s(\cdot,\cdot)$ is CLIP cosine similarity, $q$ is the gloss-expanded query, $j$ ranges over the ten candidates, and $\lambda$ is one scalar. Centring each attribute score over the candidate set removes attributes that every image matches. Only $\lambda$ and the number of retained attributes are tuned, on the trial split.
Alongside the method we run a diagnostic study. We hand-label the wrong image chosen by each baseline on the English test set as alternative-sense, same-domain or unrelated. The Ukrainian benchmark already provides these distractor categories for its negatives. We also score each gloss for "visual checkability," defined as the fraction of its content words that appear as attributes with non-zero contrast weight. This lets us test whether checkability predicts when gloss augmentation helps, which is the GPL versus UAlberta contradiction.
Baselines
The naive reference point is zero-shot CLIP on the raw phrase. Augmenters reports a 60.48% hit rate and 73.88% MRR (mean reciprocal rank of the correct image) for CLIP ViT-L/14-336 on English test. The task paper reports 37.2% average HIT@1 for its CLIP baseline across all three languages. We rerun this with the same backbone used for every other system.
The gloss-augmentation baseline is GPL's WordNet-plus-Wikipedia pipeline, which reports 68.47% hit rate and 79.17% MRR on English. On Italian, the same pipeline drops from 37.70% to 32.46%. We reimplement it on our backbone so that the only difference from our method is the contrast stage.
The strongest zero-shot competitor is LM-as-KB phrase enhancement, reported at over 72% accuracy with Vicuna-13B and CLIP-LAION. We run it with the same language model, the same CLIP-LAION backbone and its best "meaning_of" template, with and without its majority-bias penalty.
The strongest overall system is FCLL, at 72.56% average H@1 and 82.22% MRR across languages. The task paper reports English and Italian peaks above 84% and a Farsi peak of 64.0%. FCLL is trained, so we compare against its reported numbers rather than rerunning it, and we state that asymmetry openly.
Beating the field would mean three things. First, exceeding LM-as-KB under an identical backbone and language model. Second, a reduction in same-domain errors that is larger than the reduction in unrelated-distractor errors. Third, not degrading Italian and Farsi, which is where GPL's augmentation failed. Matching FCLL's average without any training would be a strong but secondary result.
Four ablations isolate the key ingredient. The first, candidate-blind contrast, contrasts against every competitor in the inventory regardless of whether it appears in the images. The second, no-contrast, sets all $w_a$ equal. The third is language-model-attributes-only. The fourth is external-sources-only.
Why not just…?
Why not just ask the language model for a definition and feed it to CLIP? This is the incumbent, and it is strong. But the definition is produced without knowledge of the distractors, so it cannot tell the synthesizer from the piano when both fit "electronic or acoustic keyboard instrument." In Augmenters, standalone language-model augmentation only tied base CLIP on trial data (56.25% for both). Our error-type analysis tests directly whether definitions leave same-domain errors untouched.
Why not just ask the language model for contrastive visual attributes against the other senses? This is the real threat, because it might capture most of the gain with no knowledge bases at all. That is why the language-model-only and external-only ablations are central. If language-model attributes alone match the full pool, we would report candidate-aware contrast as the contribution and the heterogeneous sources as unnecessary in English. We expect the external sources to matter in two places: Italian and Farsi, where language-model and lexical knowledge are weaker, and properties subject to reporting bias, which EDAM found only VisualGenome covers.
Why not just caption the ten images and let a language model choose? Language Models as Knowledge Bases tried BLIP-L and GiT-L captions with chain-of-thought selection, and it underperformed embedding retrieval because the captions drop the discriminating detail. Our method checks attributes against the image embedding directly, so no caption sits in the path. A multimodal model that views all ten images is also a natural alternative, but on Ukrainian the best of eight scored 42.78%, below the English CLIP baseline.
Why not just fine-tune CLIP on the training data? Four teams report that it hurt or did not help on gold data because of silver-to-gold shift (UAlberta, GPL, ML Mob, SzegedAI).
Proposed Experiments
All experiments use the SemEval-2023 Task 1 gold test sets: 463 English, 305 Italian and 200 Farsi instances, scored by hit@1 and MRR. We additionally run the 87-entry Ukrainian benchmark as a low-resource stress test with labelled distractor types. No reported result was found on Lacuna for zero-shot CLIP on the Ukrainian set itself; the 60.48% figure that paper cites is the English baseline.
Three checks decide the outcome. The first is whether candidate-aware contrast beats candidate-blind contrast and no-contrast weighting at equal attribute budgets. If it does not, the core claim fails. The second is whether the gain concentrates on same-domain errors in the hand-labelled English subset and on the similar-domain distractor category in Ukrainian. The third is whether gloss visual checkability predicts the per-instance gain from GPL-style augmentation, which would explain why glosses help in one system and hurt in another.
Secondary questions are whether external sources add accuracy over language-model-only attributes in Italian and Farsi, and whether VisualSem, which failed as a direct query expansion in LT, becomes useful once it is used as an attribute and competitor source instead.
References
- SemEval-2023 Task 1: Visual Word Sense Disambiguation
- GPL at SemEval-2023 Task 1: WordNet and CLIP to Disambiguate Images
- UAlberta at SemEval-2023 Task 1: Context Augmentation and Translation for Multilingual Visual Word Sense Disambiguation
- LT at SemEval-2023 Task 1: Effective Zero-Shot Visual Word Sense Disambiguation Approaches using External Knowledge Sources
- Language Models as Knowledge Bases for Visual Word Sense Disambiguation
- Identifying and Explaining Discriminative Attributes
- Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources
- G-SAP: Graph-based Structure-Aware Prompt Learning over Heterogeneous Knowledge for Commonsense Reasoning
- ML Mob at SemEval-2023 Task 1: Probing CLIP on Visual Word-Sense Disambiguation
- SzegedAI at SemEval-2023 Task 1: Applying Quasi-Symbolic Representations in Visual Word Sense Disambiguation
- TAM of SCNU at SemEval-2023 Task 1: FCLL: A Fine-grained Contrastive Language-Image Learning Model for Cross-language Visual Word Sense Disambiguation
- Augmenters at SemEval-2023 Task 1: Enhancing CLIP in Handling Compositionality and Ambiguity for Zero-Shot Visual WSD through Prompt Augmentation and Text-To-Image Diffusion