Vision-language models (systems that read an image and a question and answer in text) are poor at simple spatial questions such as "is the cup left of the plate?" One tempting remedy is to give them extra help in the prompt, for example object boxes or depth estimates produced by an off-the-shelf detector. This works only up to a point. When More Is Less found that one well-chosen spatial hint raises Visual Spatial Reasoning accuracy by up to 9.6%. Five or more hints push accuracy below the single-hint level, and step-by-step reasoning prompts lower it by up to 2.0%. That study treated the models as black boxes, so it offers no explanation for why more help hurts.
Our proposal is to explain this collapse in terms of where attention goes, and then to fix it at inference time without retraining. The suspected mechanism comes from the literature on attention sinks: a few tokens that soak up large amounts of attention while carrying little meaning. Sinks exist because softmax attention must assign a total weight of one to something, even when nothing in the context is relevant. Prior work shows that in middle layers these models correctly point their attention at the objects the question mentions. In late layers that attention drifts back onto background and sink tokens. We hypothesize that extra prompt context speeds up this late-layer drift. The intervention follows directly: take part of the late-layer sink attention and move it back onto the object locations the model itself found earlier.
Core Claim
We claim that the overload curve in spatial reasoning, where accuracy peaks at one hint and falls as hints are added, is caused at least in part by late-layer attention leaving the queried objects. We further claim that this loss can be reversed with no training by a rule we call grounding carry-forward. For each attention head that mainly reads the image, in the late layers, let $a_j$ be the attention weight on token $j$. Let $S$ be the set of sink tokens. These are the first text token plus visual tokens with unusually large hidden-state norm, identified as in See What You Are Told. Let $s=\sum_{j\in S}a_j$ be the total attention on sinks. We rewrite the weights as
$$a'_j = (1-p)\,a_j \ \ (j\in S), \qquad a'_j = a_j + p\, s\, g_j \ \ (j\notin S),$$
where $p$ is the fraction of sink attention we move, and $g$ is a distribution over non-sink tokens that sums to one. The new weights still sum to one.
The contribution lies in the choice of $g$. In Visual Attention Redistribution (VAR), the method of See What You Are Told, $g_j$ is proportional to the head's current weight $a_j$. When the context is noisy, that choice also amplifies the distractor tokens. We instead set $g$ to the attention that the question's object words (the two objects being related) placed on image patches and hint tokens in the middle layer where that attention was most concentrated. This is the "early locking" stage described in Visuals Lie, Consistency Speaks. In short, the method takes the model's own earlier, correct grounding and enforces it where the model has let it go.
Relation to Prior Work
Three pieces of prior work motivate the proposal. When More Is Less establishes the overload curve, but it does not analyze model internals. RAR shows that late layers "re-sink" visual attention onto background patches. It attributes this to sparse gradients on visual tokens during text-only supervision, and it fixes the problem by pruning late-layer sink heads. Visuals Lie, Consistency Speaks shows that grounding is correct early and becomes detached from the output late.
None of these three tests spatial-relation benchmarks or varies the amount of prompt context. Our proposal does both. It also changes the intervention from removing heads (RAR) or redistributing attention in proportion to existing weights (VAR) to redistributing toward a target taken from an earlier layer.
Two results constrain the design. The first is Mann et al.'s finding that spatial attention statistics correlate with correctness at $R\approx0.001$. This bears directly on our claim, and we take it head on. Their measures were spread and cluster counts of attention, compared across different examples. We instead measure, within each example, how much attention lands on the queried objects as context grows, and we test whether that attention matters causally by intervening on it. A correlation across examples can be near zero even when the effect within each example is strong. If our patching experiment (below) shows no causal effect either, the hypothesis is refuted.
The second constraint comes from To Sink or Not to Sink. That paper finds that sinks inherited from the image encoder act as summaries of the whole scene, and that routing them separately improves geometry on MathVista by up to 7.5 points. Spatial relations depend on a frame of reference, and that frame is plausibly stored in exactly these summary tokens. We therefore treat "which sinks to drain" as a variable to test, not an assumption. SinkPruner supplies the high-norm detection procedure. It also warns that merging the top 1% of high-norm tokens may erase the spatial landmarks we need.
The architectural fixes, softmax-1 in From Attention to Activation and Affine-Scaled Attention, let attention weights sum to less than one. That removes the reason sinks form. However, both require pretraining or distillation, and both were validated only on text models of 3B parameters or fewer. We therefore treat them as conceptual motivation rather than practical competitors.
Research Program
The program has three stages: diagnosis, causal test, and intervention.
For diagnosis, we reproduce the context-load protocol of Akasaka and Han on Qwen2-VL and LLaVA-Next-1.6, and add LLaVA-1.5-7B and Qwen2.5-VL-3B to allow comparison with VAR and RAR. We inject 0, 1, 3, or 5 spatial hints generated by Grounding DINO and Depth Anything v2, in both raw-number and verbalized form. We add a separate condition with irrelevant distractor sentences in the style of the SciAux auxiliary-information resource. For each layer and each image-reading head, we log three quantities: attention mass on the queried object patches, attention mass on sinks (the first text token, LLM-level visual sinks, and encoder high-norm sinks, each tracked separately), and attention mass on the hint tokens.
The prediction to test is a choice between two accounts. Under the absorption account, sink mass grows with the number of hints $K$. Under the dilution account, hint tokens take mass directly from the objects. The intervention we design depends on which account the data support.
For the causal test, we copy late-layer attention patterns from the run with one hint into the run with five hints on the same example. This gives an upper bound on how much of the accuracy loss attention can explain.
For the intervention, we implement grounding carry-forward. The free parameters are $p$, the late-layer band, and the choice of early layer, with the early layer picked per example by maximum concentration of object attention. All of these are tuned only on a held-out VSR development split.
Baselines
The naive reference point is the unmodified model at each hint count $K$, together with the best single hint as reported by Akasaka and Han. The relevant reported figure there is the up-to-9.6% VSR gain from one hint. All gains are measured against that curve.
The strongest existing training-free competitor is VAR from See What You Are Told. It is the closest in form, since it redistributes sink attention in image-reading heads. Its reported results on LLaVA-1.5-7B are a POPE F1 of 85.90 rising to 86.53 and MMVP accuracy of 3.33% rising to 9.33%. We found no reported VAR result on VSR, What'sUp, or EmbSpatial in Lacuna, so we will run it ourselves on identical prompts, models, and hint conditions. It will receive the same tuning budget over its redistribution fraction and head-selection threshold on the same development split.
The second competitor is RAR's sink-head sparsification from RAR. It reports Qwen2.5-VL-3B POPE moving from 86.1% to 87.4% and OVDEval from 39.5% to 43.8%. We found no reported RAR result on spatial-relation benchmarks in Lacuna. We run it on the same footing.
The ablation that isolates our key ingredient keeps every setting fixed and changes only $g$. We compare three choices: our early-layer object grounding, VAR's proportional rule, and a uniform distribution over the object boxes from Grounding DINO, which serves as an oracle with an external detector. A second ablation varies which sinks are drained: the text first token only, LLM-level visual sinks, or also the encoder summary sinks. This tests the tension raised by To Sink or Not to Sink.
We count the method as beating the baselines only if three conditions hold on VSR and What'sUp: it beats VAR and RAR at five hints, its accuracy drop from one hint to five hints is smaller than both, and it stays within one point of the base model on POPE and MMVP.
Why not just…?
Why not just give the model one well-chosen hint? Akasaka and Han show that a single hint is the best prompt-level operating point. If a cheap reranker can reliably pick that hint, an internal intervention adds little. This is a real threat. We will build such a reranker, which scores hints by text similarity to the question's object names. If reranked single-hint prompting matches grounding carry-forward at five hints, we will report that the problem is best solved upstream. Our case rests on two points. First, the right hint is often unknowable in advance, for example whether depth or a bounding box decides the relation. Second, the distractor-text condition, where no clean hint exists to select, tests robustness that selection cannot provide.
Why not just run VAR or RAR on spatial benchmarks? That experiment is itself a gap, and it appears here as a baseline. The difference we test is whether proportional redistribution amplifies the injected noise. If VAR flattens the overload curve as well as our method does, then using early-layer grounding is unnecessary. The $g$-ablation answers this directly.
Why not just use step-by-step reasoning or a spatial scratchpad? Plain step-by-step prompting lowers VSR accuracy by up to 2.0% according to Akasaka and Han, because it builds reasoning on misread grounding. SpatialThinker trains a structured scene-graph scratchpad with reinforcement learning and reports a +12.1% gain over GPT-4o on 3DSRBench. That approach needs training, and it has not been tested under contextual noise. We include SpatialThinker-7B under our hint conditions to see whether long scratchpads themselves suffer re-sinking. We treat it as a complementary point of reference, not a same-cost competitor.
Why not just scale the model? Scaling barely moves perspective-taking according to Vision Language Models See What You Want but not What You See. Also, What's "up" shows spatial prepositions appear in fewer than 0.2% of web captions. We nonetheless report the 3B-to-7B trend for each method to show whether the overload gap closes with size.
Proposed Experiments and Decisive Checks
The benchmarks are VSR, where humans score 95.4%, leading models about 70%, and zero-shot CLIP 56% according to Liu et al.. We also use What'sUp, where the best model scores 56% against 99% for humans, EmbSpatial for cluttered scenes, and 3DSRBench. POPE and MMVP serve as checks that general performance does not regress.
The primary metric is accuracy at each hint count $K$. We report two summaries: the overload gap, defined as accuracy at $K=5$ minus the best accuracy at $K=1$, and the area under the accuracy-versus-$K$ curve.
Four results would settle the question. First, if copying attention from the one-hint run into the five-hint run recovers less than a third of the lost accuracy, attention is not the bottleneck, consistent with Mann et al., and we stop. Second, within each example, the drop in late-layer attention mass on the queried objects should predict that example's accuracy loss across hint counts. Third, grounding carry-forward should beat VAR in the $g$-ablation. Fourth, draining the encoder summary sinks should hurt What'sUp and other frame-of-reference questions while helping cluttered EmbSpatial scenes. Such a split would resolve the dispute over whether sinks are trash bins or summaries by showing that their role depends on what the question needs.
References
- When More Is Less: A Systematic Analysis of Spatial and Commonsense Information for Visual Spatial Reasoning
- See What You Are Told: Visual Attention Sink in Large Multimodal Models
- Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models
- RAR: Reversing Visual Attention Re-Sinking for Unlocking Potential in Multimodal Large Language Models
- To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models
- SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
- From Attention to Activation: Unraveling the Enigmas of Large Language Models
- Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention
- SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards
- Vision Language Models See What You Want but not What You See
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- Visual Spatial Reasoning