Counterfactual Image Checks for Each Step of Multimodal Reasoning

When a vision-language model reasons step by step about an image, it builds a growing pile of its own text. That text gradually takes over. The model attends less to the picture and more to what it has already written, and a step that claims to read something off the image ("the bar for 2019 is taller") may really be a guess drawn from language habits and from the earlier steps. Once such a step is written, every later step treats it as fact. We propose a cheap, training-free test that runs during generation. For each step that claims to describe the image, we re-score the same step with the image blanked out. If the step is almost as likely without the picture, it was not read from the picture, and we regenerate only that step, from a shortened context that removes the accumulated trace.

The contribution is a trigger that is conditional on the kind of step. Blanket "look at the image more" interventions conflict with a finding in the literature that disengaging from the image is harmless, or even helpful, once the needed numbers have been extracted and the model is doing pure symbol manipulation. Our test fires only on steps that assert visual content, so it should repair the harmful form of this disengagement without taxing the benign form. The same move also treats the model's own earlier reasoning as a source of distracting context. It borrows the pruning idea from retrieval-augmented question answering and applies it inside a single multimodal reasoning trace.

Core Claim

For a reasoning step $s_t$ produced from image $I$, question $Q$, and prior steps $h_{<t}$, define its visual dependence as the per-token log-likelihood gain from seeing the real image rather than a blanked or heavily noised image $I_0$:

$$V_t = \frac{1}{|s_t|}\Big[\log p(s_t \mid I, Q, h_{<t}) - \log p(s_t \mid I_0, Q, h_{<t})\Big].$$

We claim three things. First, among steps that assert visual content, low $V_t$ predicts that the step is hallucinated better than the signals now in use. Those signals are output entropy and attention mass on image tokens. Second, regenerating only the flagged steps, with the image kept and the prior trace compressed, improves final accuracy and step-level faithfulness on image-reference tasks. Third, it does this without the losses on symbolic-execution tasks that an always-on intervention causes. A fourth, secondary claim is that the gain grows when irrelevant text is added to the prompt, because distractor text and accumulated trace text drive the same failure.

Where This Sits in Prior Work

Two papers motivate the proposal by documenting the failure. Don't Blink: Evidence Collapse during Multimodal Reasoning measures attention to image tokens and to the relevant bounding box across reasoning steps, and finds that it decays in 83.5% to 100% of samples. It then proposes a monitor that vetoes or rejects answers. We differ in two ways: we intervene inside the trace instead of rejecting the output, and we use a counterfactual likelihood rather than attention mass. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows the same attention shift toward self-generated text in long free-form descriptions and suppresses it with contrastive decoding against context. We reuse contrastive decoding as one repair operator, but only on flagged steps of logical or mathematical derivations, which that paper does not study.

The same Don't Blink paper also constrains our design. It reports that the confident-but-blind penalty is "absent or inverted on tasks like MathVista," where looking back at the image after extraction can add noise. This is the reason our trigger is gated by step type rather than always on. It is also why MathVista serves as a test of whether we avoid harm, not only as a place to look for gains.

Three lines of work supply the machinery. SSR: Socratic Self-Refine shows that locating and refining the single weakest step beats holistic self-refinement. It finds that GPT-5 on HLE math gains 1.63% with SSR, while plain self-refine loses 1.41%. SSR, however, is text-only and estimates confidence by re-solving each step $M$ times. Our score costs one extra teacher-forced pass per step and tests grounding directly, instead of using self-consistency, which cannot catch a prior the model repeats every time. Superposition Prompting and CompAct show that forking or compressing context before answering helps retrieval QA. We apply the same principle to the model's own trace when regenerating a flagged step. SHARP shows that faithful and hallucinated states are linearly separable in middle layers. That result motivates a learned probe as a competing trigger in our ablations.

The closest joint work takes different routes. Sherlock trains self-correction with visual perturbation and corrects the whole response in a second pass. Its authors note that step-wise correction during the first pass "remains a significant challenge," and that is exactly the regime we target. NoisyGRPO also injects image noise, but at training time during reinforcement-learning rollouts. We use the noised image at inference as a counterfactual probe, which requires no training. Grounded Chain-of-Thought requires bounding boxes before answers and has no recovery path when a box is wrong. Visual Para-Thinker argues for reasoning in width over depth. Our method stays in depth but cuts the depth-induced trap at the step where it bites.

Research Program

The system has three parts, all running on open-weight models where teacher-forced likelihoods are available. The models are LLaVA-v1.5, LLaVA-Next, Qwen2.5-VL at 3B and 7B, and LLaVA-ov-7B.

The first part is a step tagger. It splits the trace at step boundaries and labels each step as a visual assertion or as symbolic or derived. We will compare two taggers: the model prompting itself for a one-token tag, and a small classifier trained on a few hundred annotated steps. The second part is the score $V_t$ from the Core Claim, computed with each of three counterfactual images: a blank image, Gaussian-noised pixels, and a patch-shuffled image. We also compute a companion trace-dependence score $C_t$. It is the likelihood gain from conditioning on the full prior trace rather than on a compressed version of it: the question plus a one-line statement of the current subgoal. A visual assertion with low $V_t$ and high $C_t$ is the signature of the contextual trap.

The third part is the repair operator. When a visual-assertion step falls below a threshold $\tau$, which is tuned on a held-out split, we regenerate that step from the image, the question, and the compressed subgoal rather than the full trace. Decoding contrasts against $I_0$. We accept the new step only if its $V_t$ rises, and we allow at most $K$ repairs per trace to bound cost. Generation then continues from the repaired step. No step is ever regenerated without the image, so the method cannot strip away visual evidence the way blanket pruning might.

The analytic program has two goals. We will measure, on annotated traces, how well $V_t$ separates supported from unsupported visual steps. We will also check whether the MathVista "inversion" disappears once steps are split by type. The prediction is that symbolic steps have low $V_t$ for good reason, while visual steps that are wrong still have low $V_t$.

Baselines

The naive reference is standard greedy chain-of-thought from the same base model, with the same prompt and answer extraction. Every other method is reported as a difference from it.

The strongest existing in-trace competitor is always-on contrastive decoding against context, as in HalTrapper from Why LVLMs Are More Prone to Hallucinations in Longer Responses, applied at every token of the reasoning trace. It is the strongest because it attacks the same attention shift with no gating. If gating adds nothing, this baseline will match us. It will use the same models, the same counterfactual image, and a contrast-strength grid of the same size as our $\tau$ grid. On detection, that paper reports AUROC gains of 5–12% on its own descriptive setting. No reported result found on Lacuna for it on multi-step reasoning benchmarks. Beating it means higher accuracy on image-reference subsets, statistically indistinguishable or better accuracy on MathVista, and fewer decoding passes.

The strongest correction method is Sherlock. Pairing it with a verifier raised MathVista accuracy from 54.0 to 55.9, and it moved its eight-benchmark average from 64.1 to 65.4. Because Sherlock requires preference training, we will run the released models where available. We compare at matched inference token budgets and report our method both alone and applied on top of Sherlock's first pass. Beating it means exceeding its MathVista gain at equal or lower token cost, or adding to it when stacked.

The strongest step-level verifier is an SSR-style multimodal port of SSR. It re-solves each step $M$ times with the image and refines the lowest-consistency step. $M$ is set so that its compute matches ours and also at its native $M$. Its reported results are text-only, for example 62.00% versus 37.00% on AIME 2025. No reported result found on Lacuna for it on multimodal benchmarks.

For detection alone, we compare our trigger against three signals: output entropy; attention mass on image and bounding-box tokens, as in Don't Blink, with no numerical accuracy value given in the file; and a SHARP-style linear probe at a middle layer from SHARP.

Two ablations isolate the key ingredient. One removes the step-type gate, so the same score and repair apply to all steps. The other keeps the gate but replaces the counterfactual score with attention mass or with a random trigger at a matched firing rate.

Why not just…?

Why not just run contrastive decoding against a blank image on every token? This is the most serious threat, and it may well capture most of the gain on hallucination-heavy benchmarks. The evidence against it as a complete answer is the MathVista inversion reported in Don't Blink: forcing visual reliance during symbolic execution can hurt. Our "no gate" ablation and the always-on baseline settle the question directly. If always-on decoding matches gated repair on both the image-reference subsets and MathVista, the gate is unnecessary. We would then report the method as a cheaper version of existing contrastive decoding, not as a new mechanism.

Why not just re-inject the image or prompt "look at the image again" every few steps? This spends compute on every step regardless of need. Sherlock reports that models rarely repair their logic from a noticed error without specialized training. SSR shows that unfocused self-refinement drifts, with the -1.41% result on HLE math. We include periodic re-prompting at matched firing rate as a baseline. If it equals targeted repair, the value lies in re-reading, not in the trigger.

Why not just filter by output entropy or reject low-confidence answers? Don't Blink finds that entropy fails to detect evidence collapse, because ungrounded steps are written confidently. Separately, Improving Zero-shot Reader by Reducing Distractions finds distracted readers assign confidence above 0.95 to hallucinated answers. Rejection also yields no answer. Our step-level detection comparison against entropy reports this directly.

Why not just crop regions with an external detector first? TFAR needed conformal calibration to keep detector errors from misleading the model. MATA shows that fixed tool pipelines cannot recover when an upstream tool fails. Detectors also do not address errors that enter through the trace itself. We treat TFAR as a complementary preprocessing step and report our method with and without it.

Proposed Experiments

The first experiment tests detection. We sample traces from each base model on MathVista, HallusionBench, and MM-GCoT, and annotate about 1,500 visual-assertion steps as supported or unsupported. HallusionBench's paired original and edited images give a partial automatic label: a step whose claim is unchanged across a pair that should flip it is unsupported. MM-GCoT's boxes let us check whether a step's claim refers to the annotated region. Metric: AUROC of $V_t$ against entropy, attention mass, and the linear probe, reported separately by step type.

The second experiment tests end-task accuracy. We report MathVista accuracy, with image-reference and symbolic-execution subsets reported separately. We report HallusionBench question-pair accuracy, where GPT-4V reached 31.42% and all other tested models stayed below 16% in Hallusionbench. We report MM-GCoT answer-grounding consistency, where the GCoT-trained LLaVA-13B reached 61.8% in Grounded Chain-of-Thought, and MMStar. For each, we also report extra forward passes per question. As a sanity check that single-step object questions are not harmed, we run POPE adversarial, where SHARP reports 82.8% on LLaVA-Next.

The third experiment tests contextual noise. We build noise-augmented versions of MathVista and HallusionBench by prepending plausible but irrelevant text, such as captions of other images or chart descriptions from other items, at several lengths. We then measure how the accuracy gap between our method and each baseline changes as the distractor grows. The file reports no prior numbers for this setting, so standard chain-of-thought on the same augmented data is the reference.

The decisive checks are three. The method must beat always-on contrastive decoding on image-reference items while not falling below plain chain-of-thought on MathVista's symbolic items. The counterfactual score must beat attention mass as a trigger at matched firing rate. The advantage must widen as distractor text lengthens. If the first check fails, the step-type gate has no support. If the second fails, the counterfactual score has no support. If the third fails, the link to contextual noise is unsupported, and we would report the work as a hallucination method only.