Seeing Again on Demand: Probe-Gated Visual Re-Anchoring for Multimodal Reasoning

Vision-language models that "think out loud" have a known failure: the longer they reason, the less they look. As a chain of thought grows, the model's attention shifts from the image toward its own earlier text. By the last steps it states what it expects the figure to show, not what the figure shows. We propose two pieces. The first is a cheap readout of the model's hidden states that tells, step by step, when the model is making a claim about the image without actually consulting it. The second is an intervention that fires only at those moments: it sends attention back to the specific image regions the model itself judged relevant when it first read the question. Fixes that act on every step or every head are blunt. This one is targeted twice: it acts only when the model is drifting, and only on the regions the model itself chose.

The idea borrows a trick from personalized image generation. Tuning-free personalization systems avoid per-subject fine-tuning by storing the keys and values (the internal vectors that attention reads from) of a reference image and making new generations attend to them. We apply the same "attend to a stored snapshot" move inside a reasoning model. The stored reference is the model's own early, well-grounded attention over the image, kept from the single forward pass that reads the prompt. It costs no extra optimization, no extra forward passes per token, and no external detector. If it works, it should cut hallucinated visual claims in long reasoning chains without hurting the steps that legitimately reason in text. It should also do this at almost no latency cost, which separates it from contrastive decoding.

Core Claim

Hallucination in long multimodal reasoning is a problem of timing and targeting, not of overall visual attention mass. Two things are needed to fix it. One is a step-level detector of ungrounded visual claims, built from internal states. The other is a re-anchoring bias toward the model's own early evidence map, applied only when the detector fires. Together they should beat static, always-on attention rescaling at the same total amount of visual attention. Each ingredient should matter separately: firing at every step should hurt pure-text reasoning steps, and boosting all visual tokens equally should amplify background noise.

Where This Sits

Three papers motivate the proposal. More Thinking, Less Seeing? diagnoses the problem: attention drifts from visual tokens to generated text as reasoning lengthens. It introduces RH-Bench and the RH-AUC metric for the reasoning-versus-hallucination trade-off, but it offers no intervention. Why LVLMs Are More Prone to Hallucinations in Longer Responses traces the same drift to a pull toward narrative coherence. MATH-Vision shows that chain-of-thought can lower accuracy for some multimodal models and that reasoning errors outnumber recognition errors (42.2% vs 31.9% of GPT-4V errors). Together these imply that the fix must keep the model's textual reasoning intact rather than force it to stare at the image.

The closest existing method is Functional Attention Control. It classifies heads as "perception" or "reasoning" by how much each attends to image versus text, then applies a fixed gain (1.1 to 1.4) to every token. It reports an average 4.2-point accuracy gain across MathVista, MATH-Vision, HallusionBench, MMStar and SEED-Bench. Our proposal differs in two testable ways. Its gain is always on, whereas ours is gated per step by a learned readout. Its gain scales all visual attention in a head, whereas ours redistributes attention toward specific image tokens.

Prompt-level steering methods also constrain the design. SHARP, DMAS, AFTER and per-instance disentangled subspaces each add a correction computed once per prompt or instance, and each was evaluated on captioning or yes/no object polling rather than multi-step reasoning. Two details from them matter here. DMAS needs an external object detector to build corrupted images, and the subspace method needs an extra forward pass on a heavily masked image. Both are awkward on diagrams and charts. DSCR smooths the visual key-value cache by depth before the question is read, so it is query-agnostic by construction. Our anchor, by contrast, is defined by the question.

Attention Hijackers supplies a constraint. Instruction tokens can route attention to high-norm background outlier tokens, which accounts for up to 46.7% of LLaVA-1.5 hallucinations. A naive "boost visual attention" intervention would amplify exactly those tokens, so our anchor removes them explicitly.

The method comes from two sources. The re-anchoring move reuses the constructive key-value attention of KV Inversion and CustAny. In those systems, new content attends to stored reference states instead of the model being optimized per subject. The detector reuses probing results showing that models internally signal ungrounded output before emitting it: LLMs Know More Than They Show, On Early Detection of Hallucinations, and Do Androids Know They're Only Dreaming. SHARP adds that faithful and hallucinated multimodal states separate linearly in middle layers (10 to 15).

Research Program

Labels: an operational definition of an ungrounded visual claim. Following the "hidden hallucination" observation of Villa et al., we call a reasoning step ungrounded when it asserts something about the figure but its tokens barely depend on the figure. For a step whose content tokens run from position $t$ onward, we measure image dependence as

$$\Delta_t = \mathrm{KL}\big(p(\cdot \mid x_{<t}, I)\,\|\,p(\cdot \mid x_{<t}, \varnothing)\big),$$

averaged over the step's content tokens. Here $p$ is the model's next-token distribution, $x_{<t}$ is the prefix, $I$ is the image, and $\varnothing$ is a blank image. A separate language-model judge marks which steps make visual assertions. A step is labeled ungrounded if it is a visual assertion and $\Delta_t$ falls below a threshold set on a small hand-checked sample. The blank-image pass is used only to build training labels, never at inference.

We first validate these labels against ground truth. On scientific figures from VeriSciQA, whose answers are auto-verified, we check that ungrounded-labeled steps contradict the verified figure facts more often than grounded steps.

Detector. A linear probe reads the middle-layer hidden state at the first token of each step and predicts the ungrounded label. It is trained once per model on traces generated from VeriSciQA and a held-out portion of chart and diagram data. The detector costs one dot product per step.

Anchor. During the single forward pass that reads the prompt, we take the attention over visual tokens from the perception heads (identified by the modality-attention ratio of Functional Attention Control) at the final question token. We remove high-norm outlier tokens as identified in Attention Hijackers and renormalize, giving an anchor distribution $a^0$. It is stored per image, much like a reference subject's keys and values.

Re-anchoring. When the probe fires at step $t$, so that the gate $g_t = 1$, we modify the attention scores of those perception heads for every visual token $j$ throughout the step:

$$\tilde{s}_{t,j} = s_{t,j} + \lambda\, g_t \log(\epsilon + a^0_j).$$

Here $s_{t,j}$ is the original pre-softmax attention score, $\lambda$ is the bias strength, and $\epsilon$ is a small constant that keeps tokens outside the anchor from being zeroed out. The bias tilts attention toward regions the model judged relevant to the question. It does not inflate attention to the whole image.

Analyses. We report attention-to-image curves over reasoning length, as in More Thinking, Less Seeing, with and without the intervention. We plot gate firing rates against step type: visual assertion versus algebraic manipulation. We also run a dose-response sweep over $\lambda$ to locate the region where accuracy improves and fluency survives.

Baselines

The naive reference point is the unmodified reasoning model under identical decoding and token budget. For Ocean-R1-7B, More Thinking, Less Seeing reports an RH-AUC of 0.63, the highest among the models it evaluates. Any intervention must at least raise this number on the same model and protocol to count as progress on the reasoning-versus-hallucination trade-off.

The strongest existing competitor on this exact question is Functional Attention Control. It is the only cited method that intervenes inside reasoning models during multi-step chains, and it reports an average 4.2-point accuracy gain across MathVista, MATH-Vision, HallusionBench, MMStar and SEED-Bench, with gains up to 7% on some tasks. No RH-Bench result for it was found on Lacuna. We will reimplement it on the same models (Ocean-R1-7B and R1-Onevision-7B, both evaluated in that paper) with the same head classification, and give its gain factor the same tuning budget as our $\lambda$. Both will be tuned on a held-out VeriSciQA split, never on evaluation benchmarks. Beating it means a higher RH-AUC and higher MathVista and MATH-Vision accuracy at no greater latency, with confidence intervals from multiple sampling seeds that do not overlap.

For hallucination-specific comparison on standard settings, we run SHARP on LLaVA-1.5. SHARP reports CHAIR_S falling from 52.8 to 34.8, a sentence-level object-hallucination rate in captions. This baseline checks that our method is not merely a reasoning trick: on short captioning it should roughly match prompt-level steering. Matching SHARP there while beating it on long reasoning would support the timing claim. We also report a contrastive decoding baseline (VCD) to anchor latency. SHARP documents VCD at 13.97 s per sample versus 7.04 s for SHARP on LLaVA-Next.

Three ablations isolate the key ingredients. Always-on re-anchoring sets $g_t = 1$ everywhere and tests whether gating matters. Uniform boosting replaces $a^0$ with a flat distribution over visual tokens, with $\lambda$ adjusted so the total extra visual attention mass matches the anchored version. This tests whether targeting matters beyond magnitude. A random gate fires at the same rate as the probe and tests whether the probe carries information about when to intervene.

Why not just…?

Why not just re-show the image or prompt "look at the figure again" midway through the chain? This is the most serious threat, because it attacks the same drift with no white-box access. Its costs are extra context (hundreds of visual tokens per re-insertion) and the uncertainty about when to re-insert. We will run it at three fixed intervals and also triggered by our own probe, matched on total generated tokens. If probe-triggered re-insertion matches our method, the honest conclusion is that the contribution is the detector rather than the attention bias. The note would then be reframed around step-level grounding detection.

Why not just use Functional Attention Control as is? It already gains 4.2 points with no learned components. Our concern is that a fixed gain applied to every token conflicts with that paper's own finding that deep reasoning heads should attend to text. Uniform visual boosting also risks amplifying the background outliers identified in Attention Hijackers. The always-on and uniform-boost ablations test both concerns directly. If neither ablation loses to the full method, Functional Attention Control is sufficient and our extra machinery is unjustified.

Why not just shorten the reasoning chain? If hallucination grows with length, a smaller token budget might suffice. But RH-AUC exists precisely because cutting length trades away reasoning accuracy, and MATH-Vision shows that reasoning errors are the dominant failure. We report full accuracy-versus-budget curves, and our method must shift the whole curve upward, not just move along it.

Why not just use contrastive decoding? It needs no probe, but it runs extra forward passes per token. The cited papers report it as roughly 2x slower than activation steering, and up to 6.6x slower for some variants, per Functional Attention Control. We include VCD in both accuracy and wall-clock comparisons. Our intervention adds one dot product per step plus a bias on existing attention scores.

Proposed Experiments

The main experiment evaluates Ocean-R1-7B and R1-Onevision-7B on RH-Bench (RH-AUC), MathVista, MATH-Vision and HallusionBench. We use LLaVA-1.5 on POPE and CHAIR as a short-form sanity check. This yields a direct comparison against the base model, Functional Attention Control, VCD, and the three ablations.

The decisive checks are four. First, the probe's step-level labels must predict contradictions with VeriSciQA ground truth better than chance. Second, gated anchoring must beat both always-on anchoring and matched-mass uniform boosting. Third, the attention-to-image curve must stay flatter over long chains under our method than under the baseline, while accuracy on algebra-only MATH-Vision subjects does not fall. Fourth, latency overhead must stay in the single-digit-percent range that Functional Attention Control reports for itself.

A negative result is also informative. If gating helps but targeting does not, drift is a magnitude problem and scalar rescaling is enough. If neither helps over Functional Attention Control, step-level timing is not the bottleneck in these models.