Keeping the Diagram in View: Step-Aware, Parameter-Free Visual Re-Anchoring for Multimodal Math

Vision-language models can read every label and line in a geometry figure and still get the problem wrong. A common reading of this "reasoning gap" is that the language half of the model cannot deduce. This note tests a different and more mechanical explanation. As a model writes out a long step-by-step solution, its attention to the image may fade. Each new reasoning token looks mostly at earlier text and hardly at the diagram, so later steps run on a verbal summary of the figure instead of the figure. If that is right, it would explain an odd finding: on competition-level visual math, asking for step-by-step reasoning sometimes lowers accuracy (Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset). A longer solution gives the image more time to drop out.

The proposal has two parts. The first measures how much attention the image receives at each generated reasoning step and tests whether that fading predicts errors. The second is a fix that adds no weights and needs no training. At every decoding step, the model gets a fresh read of the image through a parameter-free attention branch: attention computed directly from dot products between raw hidden states, with no learned query, key or value matrices. The strength of this read grows as the model's own attention to the image falls. The idea is to restore visual access during reasoning, which Position: Reasoning After Perception Means Reasoning Without Vision argues is missing, and to do it with no retraining, no extra prompt tokens, and no second inference pass.

Core Claim

The claim has two parts, and either can fail on its own. The diagnostic claim is that, in open vision-language models, the share of attention going to image tokens falls as the reasoning chain grows. It should fall faster on problems the model gets wrong, and the drop should be concentrated in visually heavy subjects such as transformation geometry, where GPT-4o scores roughly 18% versus nearly 50% on arithmetic (MATH-Vision). The intervention claim is that restoring visual access in proportion to this measured deficit improves accuracy on diagram-heavy math more than two alternatives: attention fixes that ignore where the model is in its reasoning, and simply re-showing the image to the model.

Where This Sits

The proposal draws on three groups of prior work.

Training-free attention repair in vision-language decoders is the closest group. See What You Are Told: Visual Attention Sink in Large Multimodal Models (VAR) finds that large amounts of attention land on uninformative background image tokens and moves that attention to informative ones. RAR: Reversing Visual Attention Re-Sinking shows that intermediate layers understand the image better than the final layers, because late-layer attention sinks back onto background patches, and it prunes the useless heads. Make LVLMs Focus (CAMA) adjusts attention scores using the example demonstrations included in the prompt.

All three reshape attention that already exists. All three apply the same rule at every token regardless of reasoning progress. None was evaluated on multi-step math: VAR and RAR report results on POPE, MMVP, and grounding tasks, and CAMA requires in-prompt demonstrations. The proposal differs on two points. The correction is conditioned on reasoning progress, and it adds a new read of the image instead of redistributing attention among the existing learned heads.

The mechanism comes from the parameter-free attention literature. CALIP shows that attention built from raw feature dot products, with no projection matrices, can usefully mix modalities in a frozen dual-encoder model, though only for zero-shot classification. RealFormer shows that residual attention paths add no parameters and stabilize deep models. Understanding Multimodal Hallucination with Parameter-Free Representation Alignment finds that the quality of the visual representation levels off, and can drop, once it enters the language model. Together with RAR, this motivates reading image features from a middle layer rather than a late one.

Two papers constrain the design. Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch? (IP-Merging) argues that math reasoning lives in the deep feed-forward layers, the MLPs, and that attention is largely irrelevant to it. That directly contradicts the premise here, and one experiment below is designed to test it. The SimAM and CALIP results also show that "parameter-free" methods still hide tuned constants (SimAM). The proposal therefore limits itself to three constants, all tuned on a held-out development split.

Research Program

Let $a^{(l)}_{t,j}$ be the attention weight from generated token $t$ to image token $j$ at layer $l$, averaged over image-centric heads. These heads are selected with VAR's rule, which keeps heads whose attention mostly falls on non-sink image tokens. The visual attention mass at step $t$ is

$$m_t = \frac{1}{|L|}\sum_{l \in L}\sum_{j \in \mathcal{V}} a^{(l)}_{t,j},$$

where $\mathcal{V}$ is the set of image tokens excluding the sink tokens identified by VAR, and $L$ is a set of middle-to-late layers. The deficit at step $t$ is $d_t = \max(0,\, 1 - m_t / m_0)$, where $m_0$ is the mass at the first reasoning token. The deficit is 0 when the model looks at the image as much as it did at the start, and approaches 1 when it has stopped looking entirely.

The re-anchoring branch reads the image using the current hidden state $h_t$ at a late layer as the query. The keys and values are the image tokens' hidden states $v_j$ taken from a middle reference layer $l_0$, before the late-layer collapse that RAR describes:

$$r_t = \sum_{j\in\mathcal{V}} \mathrm{softmax}_j\!\left(\frac{h_t^\top v_j}{\tau\sqrt{D}}\right) v_j, \qquad h_t \leftarrow h_t + \alpha\, d_t\, \frac{\|h_t\|}{\|r_t\|}\, r_t,$$

Here $D$ is the hidden width, $\tau$ is a temperature, and $\alpha$ is a global strength. Rescaling $r_t$ to the norm of $h_t$ keeps the added vector at the same scale as the hidden state it is added to. The only constants are $\tau$, $\alpha$, and $l_0$. Hidden states at $l_0$ and at the injection layer may not share a geometry, so the reading layer and injection layer are swept jointly on the development split rather than assumed.

The program has three stages. The first logs $m_t$ along greedy reasoning chains on MATH-Vision and on GAOKAO-MM mathematics and physics. It then asks whether the slope of $m_t$ separates correct from incorrect solutions, and whether this holds more strongly on the vision-recognition errors in MATH-Vision's error taxonomy. The second stage applies re-anchoring to two open model families that VAR and RAR already support, LLaVA-1.5 and Qwen2.5-VL, so that those methods run exactly as their authors intended. The third stage tests the claim that attention does not matter for math reasoning by combining re-anchoring with IP-Merging on LLaVA-1.5-7B, the setting where IP-Merging reports results. If attention is truly passive, the gains should not add up.

Baselines

The naive reference is the unmodified model with standard step-by-step prompting and greedy decoding. The same model with a direct-answer prompt is also reported, since MATH-Vision finds that step-by-step prompting can hurt. Published references anchor the difficulty: on MATH-Vision, GPT-4o reaches 30.39% and InternVL-Chat-V1-2-Plus 16.97%, against 68.82% for humans (MATH-Vision). Those systems will not be rerun. They are there to calibrate difficulty, while every compared method runs on the same open base models.

The strongest competitors are VAR and RAR, because they are the only training-free attention interventions shown to help open vision-language decoders. No reported result was found on Lacuna for VAR or RAR on MATH-Vision, GAOKAO-MM, or ChartQA. Their published gains are on perception benchmarks, for example LLaVA-1.5-7B POPE F1 rising from 85.90 to 86.53 with VAR, and Qwen2.5-VL-3B OVDEval rising from 39.5% to 43.8% with RAR. Both will be rerun on the math benchmarks using their released selection rules. They get the same development split and the same number of hyperparameter trials as the proposal: an equal grid of about 30 configurations per method.

IP-Merging is the strongest intervention on the weights instead of on attention. It reports +4.0% on MathVerse for LLaVA-1.5-7B and +3.7% for Qwen-2-VL. No reported MATH-Vision number for it was found on Lacuna, so it will be rerun.

Two ablations isolate the key ingredient. The first is a constant-strength version that fixes $d_t$ to its dataset average. It tests whether step-conditioning matters or whether any added visual read helps. The second reads image features from the final layer instead of $l_0$, which tests RAR's re-sinking account.

Beating the field means a gain over the best of VAR, RAR, and constant-strength re-anchoring on MATH-Vision that exceeds the spread across three prompt paraphrases. The gain must be concentrated in geometry-type subjects, and latency must rise by no more than a few percent.

Why not just…?

Why not just apply VAR or RAR to the math benchmarks? This is the most serious threat, and it is also the gap the file identifies. If moving attention away from sink tokens already fixes visual fading, re-anchoring adds nothing. VAR and RAR rearrange attention inside the existing learned heads. If a head has largely stopped looking at the image, redistributing its small visual share among image tokens cannot restore the missing total. The deficit-dependent term targets exactly that case. The experiment settles the question directly: if VAR or RAR matches re-anchoring on the problems with the largest measured drops in $m_t$, the proposal's mechanism is redundant, and the paper becomes a diagnostic study.

Why not just show the image again partway through the reasoning? Re-inserting the image tokens gives genuine persistent access and may work as well. It is run as a baseline, re-feeding the image every $k$ reasoning steps with $k$ tuned on the same development split. It costs hundreds of extra tokens per insertion and breaks the key-value cache that stores past attention inputs. If it matches re-anchoring at equal accuracy, the proposal's contribution shrinks to a cheaper approximation, and the note will say so. If it matches while re-anchoring fails, the fading must be something a hidden-state read cannot repair.

Why not just rely on the MLPs, merging in math skill as IP-Merging does, or translate the figure into text? IP-Merging's claim that reasoning lives in the MLPs is tested by combining the two methods. Additive gains would mean both mechanisms matter, and gains that cancel would support IP-Merging. Translating the figure into text runs into the problem that even perfect human-written diagram descriptions leave text models near chance on GeoSQA, 23% to 26.22% (GeoSQA). Chart-to-table conversion also loses information such as color (Chart-based Reasoning). A caption-then-reason baseline is still included on MATH-Vision geometry so the comparison is measured rather than assumed.

Decisive Experiments

The main evaluation uses MATH-Vision, run through the NeMo-Skills pipeline (NeMo-Skills). Results are reported per subject and per difficulty level, and errors are relabeled into vision-recognition and reasoning categories following MATH-Vision's taxonomy. GAOKAO-MM mathematics and physics serve as a transfer test. OlympiadBench (OlympiadBench) serves as a harder held-out set.

Two probes test mechanism rather than headline accuracy. The first is Chart101's grid-line perturbation, on which PaLI-3 falls from 29.5% to 21.3% (Plot Twist). Re-anchoring should shrink that drop only if the model genuinely reads the image rather than relying on title text. The second is the Turing Eye Test, used as a negative control. Its tasks fail at the vision encoder itself, so no gain is expected, and a gain there would indicate a measurement artifact.

The first decisive check comes before any intervention. If $m_t$ does not decline along reasoning chains, or its slope does not predict errors (AUC near 0.5 on MATH-Vision), the premise is false and the program stops. The second check is the constant-strength ablation: if it matches the step-conditioned version, the "step-aware" claim is dropped. The third check is that gains should be larger on long solutions than on short ones. A uniform gain would suggest a generic visual boost rather than a repair of fading. All results use three seeds of prompt paraphrase, report latency and memory, and use a fixed hyperparameter budget shared by all methods.