Where the Modality Gap Lands in Language-Conditioned Heatmaps

Many systems now locate things in images by letting a sentence "look" at the picture and paint a heatmap, a grid of scores whose peak marks where the described object is. The sentence is turned into a query vector, every image patch into a key vector, and the heatmap is essentially their dot products passed through a softmax over locations. Vision-language models are also known to keep text vectors and image vectors in two separate regions of their shared space, with a roughly constant offset between them called the modality gap. This note makes a simple observation about what that offset does inside a heatmap. Any offset shared by all image patches adds the same score to every location and cancels in the softmax. The offset shared by all text queries does not cancel. Dotted with each patch, it paints a fixed map that gets added to every heatmap no matter what the sentence says. That map is a hidden prior, and we hypothesize it is a main driver of "right kind of thing, wrong instance" errors in referring and pointing tasks.

The proposal is to measure this sentence-independent map inside trained heatmap decoders and to test whether it explains distractor errors. The fix we test is architectural. We subtract the mean query inside the decoder and learn an explicit, separately regularized prior map to take its place. We expect the effect to be largest exactly where language is thinnest: embodied reference, where utterances average 3.73 words (YouRefIt), and low-data fine-tuning, where the decoder has little signal with which to learn to ignore the offset.

Core Claim

Write the decoder's projected query for a referring expression as $q = \mu_q + \tilde q$, where $\mu_q$ is the mean query over the dataset and $\tilde q$ is the expression-specific residual. Write patch $i$'s key as $k_i = \mu_k + \tilde k_i$, with $\mu_k$ the mean key. The heatmap logit at location $i$ then splits into three parts:

$$ q^\top k_i = \underbrace{\tilde q^\top \tilde k_i}_{\text{referent evidence}} + \underbrace{\mu_q^\top \tilde k_i}_{\text{sentence-independent map}} + \underbrace{q^\top \mu_k}_{\text{constant over } i}. $$

The third term is the same at every location, so a spatial softmax removes it. Image-side gap corrections should therefore do almost nothing for softmax heatmaps. The second term, the map $b_i = \mu_q^\top \tilde k_i$, is identical for every expression about the same image.

Our claim has three parts. First, in trained language-conditioned heatmap decoders, $b$ carries a measurable share of the peak mass. Second, $b$ peaks on large, salient, frequently mentioned objects. This is the spatial form of the object bias that Two Effects, One Trigger ties to the same cause as the modality gap. Third, when the model picks the wrong object, the wrong pick sits disproportionately at peaks of $b$.

A sentence-independent prior is not always bad, because referents do sit in predictable places. So the fix does not simply delete $b$. It centers the query and adds back a learned prior map $p_i$ whose strength is set by an explicit penalty rather than by accident. Gaussian heatmap targets then force the referent-specific term to produce the peak. That term is the only one that changes with the words.

Relation to Prior Work

Three works motivate the setting. Lumen has a language model emit a location token, and a small cross-attention module turns that token and the image features into a heatmap that feeds box, mask and keypoint decoders. BridgeVLA predicts heatmaps over orthographic views of a robot's workspace in place of coordinate or text outputs. CAPE encodes pointing gestures as Gaussian ray heatmaps and re-ranks candidates with CLIP similarity. Notably, CAPE switches strategies by object size, a hand-made patch for what may be a size-dependent bias. All three treat the heatmap head as a plain output layer trained with pixel losses. None asks what the geometry of the language query contributes to the map. We study that question inside the same kind of decoder rather than proposing a new decoder.

Four works on the gap constrain the design. Mind the Gap shows the gap exists from initialization and that shifting embeddings post hoc changes zero-shot accuracy and fairness in both directions. Two Effects, One Trigger shows that thinner captions widen the gap and that closing it post hoc hurts, because the two modalities' local neighborhoods differ. Closing the Modality Gap Aligns Group-Wise Semantics shows the opposite for distance-based tasks: pulling pairs together while spreading class centroids nearly doubles clustering quality. FSAlign argues that only multi-scale neighborhood matching aligns the modalities without distortion.

These results warn us not to close the gap globally. Our intervention does not move any embedding in the shared space. It changes only how the text-side mean enters one dot product inside a spatial decoder. We also address the dispute in these papers head-on. Heatmap localization is a softmax over positions, which is neither purely rank-based nor distance-based, and our decomposition predicts exactly which half of the gap matters for it.

Two works supply comparison methods. AMLRIS builds patch-to-word similarity maps with a random projection and masks weakly aligned pixels during training. Tuning LayerNorm in Attention treats the gap as a domain shift fixable by tuning normalization parameters alone. Both are plausible routes by which a trained model could absorb or sidestep $b$ without our change, so both appear as baselines.

Research Program

The first stage is diagnosis. It needs no new training beyond reproducing standard decoders. We build two heatmap decoders on the same data. One is a Lumen-style module in which a location-token embedding from a LLaVA-class model cross-attends to image features. The other is a lightweight cross-attention head on a frozen CLIP ViT, which gives a low-capacity regime. For each trained model we estimate $\mu_q$ from the training expressions and compute $b$ for every test image. We then measure three things. The first is the share of the heatmap's peak logit explained by $b$. The second is the rank correlation of $b$ with object area and with how often each category is mentioned. The third is the "prior-hit rate": among localization failures, the fraction whose predicted peak falls in the top-$b$ object, compared against the same rate for correct predictions.

The second stage is the intervention. We subtract a running mean of projected queries inside the decoder and add a learned prior map $p$ from a single small convolutional layer over the keys, with an L2 penalty on $p$'s logits. We compare this against the plain decoder and the alternatives described in the Baselines section.

The third stage is a controlled test of the thin-language prediction. On RefCOCOg we truncate expressions to 75%, 50% and 25% of their descriptive tokens, following the caption-thinning protocol that Two Effects, One Trigger applied to CC12M. We also build a multi-object canvas version of their Morpho-MNIST MAD benchmark. Each canvas holds several digits, and the caption names one of them with a controlled number of attributes, so we know exactly how many same-class distractors exist. The prediction is that $b$'s share of the peak, and the benefit of centering, both grow as descriptions shrink.

Baselines

The naive reference point is the unmodified decoder of each architecture, trained with the same Gaussian-target loss, data, schedule and seeds. Every intervention must beat this model under an identical tuning budget. We use one hyperparameter sweep of equal size per method.

The strongest existing method on referring localization in the file is Lumen, which reports 65.1 cIoU on RefCOCOg against 61.2 for LAVT. Our Lumen-style reimplementation is the main comparison, and we will report how close it gets to that figure. On embodied reference, the strongest competitor is CAPE, which reports 75.0 mAP at 0.25 IoU on YouRefIt. We run CAPE's pipeline with our centered decoder in place of its spatial branch, with and without CAPE's size-based CLIP ensemble. The claim to test is that centering removes most of the ensemble's gain. Beating these references means a gain in point-in-box accuracy and cIoU across three seeds on RefCOCO, RefCOCO+ and RefCOCOg, together with a lower prior-hit rate. On YouRefIt it means matching or exceeding 75.0 [email protected] without the size switch.

Four further baselines test whether other routes absorb or sidestep $b$. The first is post-hoc gap subtraction, as studied in Mind the Gap, applied to the frozen encoder outputs. The second is a ModGap-style training loss (pair pull plus centroid uniformity) from Closing the Modality Gap, applied to decoder queries and keys. The third is LayerNorm-only tuning from Tuning LayerNorm in Attention. The fourth is AMLRIS masking.

The key ablations isolate the mechanism. Image-side centering alone should do nothing under a softmax head. Query centering without a learned prior tests whether the prior is needed. A learned prior without centering tests whether the prior alone explains the gain. We repeat everything with sigmoid/MSE heatmaps, where the constant term no longer cancels, to confirm that the softmax-invariance prediction really depends on the softmax.

Why not just…?

Why not just trust the decoder's learned projections to remove the offset? A trained query projection can in principle map $\mu_q$ toward zero, and this is the most serious threat to the idea. The diagnosis stage settles it directly. If $b$'s share of the peak is negligible in the Lumen-style decoder trained on full RefCOCO, the claim fails for high-data regimes, and we will report that. We still expect the effect to survive with frozen CLIP heads, on YouRefIt's 4,195 clips, and under truncated expressions. There, far less gradient signal is available to learn the cancellation.

Why not just shift the embeddings to close the gap? Both Two Effects, One Trigger and Mind the Gap report that post-hoc shifting damages local neighborhoods. Global shifting also treats both sides alike, whereas our decomposition says only the text side matters for softmax heatmaps. The post-hoc baseline and the image-side-only ablation test this. We predict that image-side shifting does nothing and that full shifting helps less than decoder-internal centering.

Why not just tune LayerNorm, which already treats the gap as a domain shift? LayerNorm tuning (Tuning LayerNorm in Attention) re-centers features per token. It does not subtract a dataset-level query mean, so the sentence-independent map could survive it. This is a real competitor. If it matches our method, we will say so and measure $b$ in the LayerNorm-tuned model to see whether it removed the map by the same route.

Why not just re-rank candidates with CLIP, as CAPE does? Re-ranking fixes errors after the heatmap is formed. It needs box proposals, and CAPE had to switch strategies by hand according to object size. If the size dependence comes from $b$, centering should make the switch unnecessary, and the CAPE-with-and-without-ensemble comparison tests exactly that.

Proposed Experiments and Decisive Checks

The benchmarks are RefCOCO, RefCOCO+ and RefCOCOg for heatmap-peak point accuracy and SAM-decoded cIoU, with the RefCOCO perturbation setting AMLRIS used for robustness. We add YouRefIt for embodied reference at IoU 0.25, and truncated-RefCOCOg plus the multi-object MAD canvases for controlled language thinning. If time allows, we will run a BridgeVLA-style heatmap head on RLBench, where BridgeVLA reports 88.2% average success. There we test the same-class distractor tasks in the low-demonstration regime.

The idea stands or falls on three checks. The first is whether failures land on peaks of $b$ significantly more often than successes do, across both architectures. The second is whether query centering beats both post-hoc shifting and LayerNorm tuning while image-side centering does nothing. The third is whether the gain grows monotonically as expressions are thinned. If the first check fails, the offset is harmless in practice and the proposal ends as a clean negative result about where the modality gap does and does not matter for dense prediction.