The Modality Gap Biases Sense Selection Across Modalities

Many words have several meanings. "Crane" can be a bird or a machine. A long tradition in lexical semantics handles this by comparing meanings rather than words: the similarity of two words is the similarity of their closest pair of senses. Vision-language models such as CLIP (an image encoder and a text encoder trained so that matching images and captions land near each other) could adopt the same trick. To decide whether a photo shows "crane", one would compare the photo against the dictionary definition of each sense and keep the best match.

This note argues that this cross-modal version of the closest-sense rule fails in a specific, fixable way. In CLIP-style models, images and texts sit in two separate regions of the shared space, offset from each other. Most current applications are immune to that offset, because they rank many images against one text. The closest-sense rule is not immune, because it compares one image against many different texts. When it does, the offset adds a score to each sense that does not depend on the image at all. That fixed score behaves like a hidden prior favouring some senses over others. We propose to measure this per-sense bias, test whether it explains part of CLIP's known habit of picking the dominant meaning of ambiguous words, and remove it with a one-vector correction applied before the closest-sense comparison.

Core Claim

Resnik's heuristic was formalized in Using Information Content to Evaluate Semantic Similarity in a Taxonomy, which scores two words by their most similar pair of taxonomy senses. It has been carried into distributional spaces by Distributional measures of concept-distance and into hashtag clustering by Sense-Level Semantic Clustering of Hashtags in Social Media. WordNet Is All You Need uses it inside an unsupervised entailment method that reaches $\rho=0.744$ on HyperLex. In every one of these settings, all senses live in one space.

Across modalities they do not. Mind the Gap shows that image and text embeddings occupy separate narrow cones, and that this separation exists even at random initialization. Closing the Modality Gap Aligns Group-Wise Semantics separates two kinds of task. Rank-based tasks, such as ranking images for one query, are insensitive to the gap. Distance-based tasks, such as clustering, are sensitive to it.

Our claim is that cross-modal sense selection belongs to the gap-sensitive kind, for a precise reason. Write an image embedding as $v=\mu_v+\tilde v$ and a sense-gloss embedding as $s_j=\mu_t+\tilde s_j$. Here $\mu_v$ and $\mu_t$ are the mean image and mean text embeddings over a reference pool, and the tilde terms are deviations from those means. Expanding the dot product gives

$$\langle v,s_j\rangle=\langle\mu_v,\mu_t\rangle+\langle\tilde v,\mu_t\rangle+\underbrace{\langle\mu_v,\tilde s_j\rangle}_{b_j}+\langle\tilde v,\tilde s_j\rangle .$$

For a fixed image, the first two terms are the same for every sense. The term $b_j$ differs across senses but does not depend on the image. When we take the maximum over senses $j$, $b_j$ therefore acts as a geometric prior: a sense whose gloss happens to point toward the image cone wins regardless of what the photo shows.

From this we draw two hypotheses. First, $b_j$ correlates with sense frequency, such as WordNet sense order, and so supplies one geometric mechanism for the dominant-sense failure documented by Vision Meets Definitions. Second, subtracting the modality means and renormalizing before taking the maximum removes the bias. Gains from this correction should concentrate on items whose correct sense is not the first-listed one.

The resulting score for an image $v$ given a target word $w$ and optional context $c$ is

$$S(v\mid w,c)=\max_{j\in\mathrm{Senses}(w)}\Big[\lambda\log p(s_j\mid c)+\cos(\tilde v,\tilde s_j)\Big].$$

Here $p(s_j\mid c)$ is a text-only estimate of which sense the context points to. $\lambda$ weights that context term, and $\lambda=0$ recovers Resnik's context-free rule.

Relation to Closest Work

Two prior systems are closest. Vision Meets Definitions treats WordNet glosses as a latent variable. It sums over them, matching context to definition in text and then definition to image with frozen CLIP. Its English results are 73.00% Hits@1 for plain CLIP, 81.98% with WordNet glosses, and 83.08% with glosses generated by GPT-3. GPL at SemEval-2023 Task 1 commits to one gloss using sentence embeddings before querying CLIP.

Neither system examines CLIP's geometry. Both compare scores across different gloss texts, which is exactly where $b_j$ enters. Our method reuses their gloss inventory and context model. It differs in three ways: it centers both modalities before comparing, it pools by maximum rather than by summing, and above all it diagnoses $b_j$ as the reason sense-level CLIP scoring is biased.

Concept Visualization (ConVis) ranks a patch's similarity to one synset against all other WordNet definitions. That ranking is an empirical correction for image-independent text bias, applied for interpretability. We give the correction an explicit cause and apply it to polysemy.

Closing the Modality Gap for Mixed Modality Search (GR-CLIP) supplies the centering technique, which subtracts per-modality means estimated on a calibration set, and validated it on mixed-modality search. Applied to a single word prompt, it cannot separate senses. DiCA addresses ambiguity among annotation labels rather than lexical senses, so it is related only in spirit.

Two papers constrain the proposal. ModGap, from Closing the Modality Gap Aligns Group-Wise Semantics, collapses the gap by training with a loss that pulls true pairs together and a loss that spreads concept centroids evenly. Mind the Gap reports that a nonzero gap can help zero-shot accuracy and fairness. We therefore treat collapse as a competitor to be measured, not assumed beneficial. We also test whether ModGap's single-centroid assumption harms the minority senses of polysemous words.

Research Program

The work has three stages, all using frozen CLIP backbones.

The first stage is measurement. For every polysemous target in SemEval-2023 Task 1 (SemEval-2023 Task 1: Visual Word Sense Disambiguation), we encode each WordNet gloss and compute $b_j$. The image mean $\mu_v$ comes from a large reference pool of MSCOCO images that is disjoint from the task images. We then test three things: whether the correlation between $b_j$ and WordNet sense rank is nonzero; whether $b_j$ predicts which wrong sense uncorrected CLIP chooses; and whether $b_j$ is stable across reference pools and backbones. If $b_j$ is near-constant across senses, or unrelated to errors, the core claim is wrong and we will say so.

The second stage is the corrected operator. We implement $S(v\mid w,c)$ with the context prior $p(s_j\mid c)$ taken from Kwon et al.'s context-to-definition step. We compare maximum pooling with summation and with top-$k$ soft pooling. The comparison targets the known weakness of a hard maximum, which can latch onto rare, spurious senses.

The third stage is the context-free probe and the collapse question. We build a word-to-image sense-coverage probe from OpenImage-V7 classes whose bare names map to several WordNet senses. Given only the word, the probe asks whether retrieval returns images of every sense or only the dominant one. On this probe we compare uncorrected CLIP, GR-CLIP centering, our operator, and a ModGap-retrained model. This tests whether collapsing the gap during training, as ModGap does, helps or hurts minority senses.

Baselines

The naive reference point is zero-shot CLIP scoring the bare target word, or the word plus context, against the candidate images. The task organizers report an average HIT@1 of 37.2% for this baseline across languages, and GPL reports 58.96% Hits@1 on English. Kwon et al. report 73.00% for their CLIP baseline. That number is not comparable to GPL's, presumably because the backbones or prompts differ. For this reason we rerun every method on one shared CLIP checkpoint and do not compare against published figures.

The strongest existing unsupervised method is Kwon et al.'s gloss marginalization, which reports 81.98% English Hits@1 with WordNet glosses and 83.08% with generated definitions. Its gloss inventory and context model are the closest to ours. We run it with the same glosses, backbone, and context prior. Beating it means a gain of at least 2 Hits@1 points on English, confirmed by paired bootstrap over the 463 gold instances. The gain should be concentrated on items whose correct sense is not the first WordNet sense.

The supervised ceiling is the task winner, TAM of SCNU, which reports 72.56% average HIT@1 and above 84% on English and Italian. We report it as context only, since our method is training-free.

GR-CLIP centering applied to the plain word prompt isolates the effect of centering when no sense inventory is used. The file reports up to 26 NDCG@10 points of improvement on MixBench, but no reported result on Lacuna was found for GR-CLIP on SemEval-2023 VWSD. GPL's sentence-embedding pre-disambiguation, at 68.47% English Hits@1 in its own setup, represents committing to a sense in text first.

The key ablations hold everything else fixed. They are maximum pooling without centering, which is the bias-exposed operator; Kwon-style summation with centering; and centering that subtracts a random vector of the same norm instead of the modality mean. Together they isolate the contribution of the geometric correction from that of the pooling rule.

Why not just…?

Why not just subtract modality means from CLIP's word embedding, as GR-CLIP does? Centering removes the constant offset, but the text vector for "crane" is still one blended point. Nothing about centering creates separate bird and machine targets. Our sense-coverage probe directly measures whether centered single-prompt retrieval recovers minority senses. The prediction is that it recovers them less often than centered maximum over senses, because the minority sense has no vector of its own to win with.

Why not just ask a language model for a better definition? In Kwon et al.'s results, generated definitions added 1.10 points on top of WordNet glosses, while WordNet glosses alone added 8.98 over CLIP. The authors also flag hallucinated definitions. Generated definitions are compatible with our method, since they simply enlarge the sense set. The bias $b_j$ applies to generated glosses exactly as it does to WordNet ones, so we test centered maximum pooling with generated glosses too.

Why not just disambiguate in text first and send one gloss to CLIP? GPL's text-first pipeline lowered Italian Hits@1 from 37.70% to 32.46%, and Cross-lingual Visual Verb Sense Disambiguation shows that images are often needed to resolve senses that text leaves ambiguous. A hard text-only choice discards that visual evidence. We nonetheless run GPL's approach on the shared backbone, so the comparison is fair.

Why not just apply centering to Kwon et al.'s summation? This is the real threat. If centered summation matches centered maximum pooling, then maximum pooling contributes nothing beyond the correction. The contribution would shrink to two parts: the diagnosis that sense-level CLIP scoring is biased by the gap, and the one-line fix for it. The ablation is designed to settle this question, and we will report the outcome either way.

Why not just retrain to close the gap, as ModGap does, or fine-tune CLIP? GPL reports that fine-tuning CLIP did not help on this task. ModGap reaches a clustering V-Measure of 23.63 versus about 13.0 for CLIP on MSCOCO, but it assumes each pair maps to one concept centroid. That assumption may pull a polysemous word toward a single sense. The sense-coverage probe compares a ModGap-retrained model with training-free centering on minority-sense recall.

Decisive Checks

The idea stands or falls on four results. First, $b_j$ must vary meaningfully across a word's senses and correlate with sense rank and with uncorrected CLIP's errors. Second, centering must remove most of the improvement that would otherwise require a learned prior, and the random-vector control must yield no gain. Third, the gains must be concentrated on items whose correct sense is not the dominant one, evaluated with SemEval-2023 Hits@1 and Mean Reciprocal Rank (MRR, the average inverse rank of the correct image) on English, plus sense-coverage recall on the OpenImage-V7 probe. Fourth, the conclusions must hold across at least two CLIP backbones and two reference pools for estimating $\mu_v$.

If the first check fails, the geometric account is wrong. If only the fourth fails, the correction is an artifact of a particular backbone or pool rather than a general fix.