Grounding-Contrast Look-Ahead for Multimodal Reasoning

Vision-language models often reason worst at exactly the moments that matter. Midway through a geometry or chart problem, a model writes a step that sounds right, such as "the maximum value on the y-axis is 50," then builds every later step on it. Often that claim came from the chart's title or from the wording of the question, not from the pixels. The usual fix is to have a second model, a "judge," score candidate reasoning paths and keep the best one. But judges share the same weakness. They reward fluent, coherent-sounding text and miss whether it is visually true. When a search procedure optimizes hard against such a judge, it can learn to produce more convincing hallucinations.

We propose a different scoring signal for look-ahead decoding. Look-ahead decoding means that, before committing to the next reasoning step, the decoder briefly drafts how each candidate step would continue and scores those short futures. Instead of asking a judge whether a drafted step is good, we ask the reasoning model itself how much more likely that step is when it sees the real image than when it sees a version with the data removed. Steps that are equally likely without the visual evidence rely on language priors, and the decoder penalizes them. The signal comes from likelihoods, not from a judge's written verdict, so a verbal rationale cannot quietly invent a justification for it.

Core Claim

Three findings from the literature motivate this. The first is that multimodal chain-of-thought can lower accuracy, because intermediate steps introduce hallucinated visual details that the model then treats as premises. Both MATH-Vision and MLLM-as-a-Judge report this. The second is that models lean on textual shortcuts in images. Plot Twist shows Chart101 accuracy reaching 79% when the title gives away the answer and falling to the 30% range when it does not. The third is that judges are swayed by surface cues they never admit to. The Silent Judge finds a 0% rate of judges acknowledging the cue that drove their verdict. Local Coherence or Global Validity? finds GPT-4o detecting only 57.8% of logical errors in coherent-looking math traces.

Our claim is that a step-level look-ahead decoder, scored by an image-ablation likelihood contrast, prevents hallucinated visual premises from entering the reasoning chain. It should do so more effectively per unit of compute than judge-scored best-of-N or judge-guided look-ahead. It should also reduce the model's reliance on textual shortcuts, where judge-guided search is expected to amplify that reliance.

The contrast we use is

$$S(s) = \bar{\ell}(s \circ r \mid q, I) + \lambda\,\big[\bar{\ell}(s \circ r \mid q, I) - \bar{\ell}(s \circ r \mid q, \tilde{I})\big]$$

where the symbols are as follows:

  • $s$ is a candidate next step and $r$ is its greedy continuation to the end of the following step.
  • $q$ is the question, $I$ is the image, and $\tilde{I}$ is an ablated image.
  • $\bar{\ell}$ is the per-token average log-likelihood under the reasoning model.
  • $\lambda$ trades fluency against visual dependence.

We average per token so that length alone cannot raise the score, which is the verbosity failure documented for multimodal judges.

The choice of $\tilde{I}$ is the key design decision. We use a content-erased image that keeps all text on the image (titles, axis labels, vertex letters) but removes the plotted marks or the figure's lines. This aims the contrast at the specific shortcut Chart101 exposes: a step that could be written from the title alone gains nothing from the real image. A blank image serves as a coarser second ablation.

The scope is narrower than "closing the multimodal reasoning gap," and we state that up front. MATH-Vision attributes 42.2% of GPT-4V's errors to pure reasoning failures and 31.9% to visual recognition. GeoSQA shows models failing even when given perfect text descriptions of the diagrams. A grounding signal cannot fix a wrong deduction from correct premises. We therefore predict gains concentrated in the perception and hallucinated-premise portion of the errors, and the experiments check that this is where any gains appear.

Relation to Prior Look-Ahead and Judge Work

The method borrows its mechanics from text-only look-ahead decoders. TWEAK rolls each candidate token forward to the end of the sentence and scores the result with an external verification model against source facts. It improved FactKB by 2.43 points on WebNLG and by up to 7.17 points out of distribution. We take its forward-hypothesis idea but replace the external verifier with a contrast computed inside the model. We also apply it to images rather than knowledge triples.

HALLUCANA supplies the idea of triggering look-ahead only at risky moments and reading internal model signals rather than querying a second large model. It raised GQI from 0.68 to 1.69 on FActScore with over 6x less compute than SelfCheckGPT. We keep the selective triggering but need no trained classifier, which avoids having to collect labels of multimodal hallucinations.

Adaptive Tree Search supports our use of a single greedy rollout instead of Monte Carlo sampling. AutoJudge also places a judge inside decoding, but its judge decides which draft mismatches are safe to accept for speed, not which reasoning steps are visually grounded.

On the judge side, MR. Judge trains a multimodal judge to reason before choosing. It reports beating GPT-4o by 9.9% on VL-RewardBench and improving MM-Vet best-of-N by up to 7.7%. Its authors note it remains fragile on dense diagrams. MR. Judge scores finished responses with long verbal reasoning, while we score partial steps with no verbal judgment at all. VL-RewardBench itself finds that generative reward models fail more on visual existence and recognition than on abstract reasoning. That finding motivates the proposal, because it suggests a judge is weakest exactly where the multimodal CoT paradox arises.

DCoT attacks grounding at the prompt level by having the model box the relevant region before answering. DCoT cannot recover if the box is wrong, whereas look-ahead can reject a bad step before it is committed.

Research Program

We build the decoder on open, white-box vision-language models, because the contrast needs token likelihoods under two images. InternVL-Chat-V1-2-Plus has a reported MATH-Vision number, and LLaVA-1.5-13B has reported MM-Vet and DCoT numbers.

Reasoning is segmented into steps at line breaks. At each step boundary the decoder does the following:

  1. It samples $K$ candidate steps (default 4).
  2. It rolls each candidate greedily to the end of the next step.
  3. It scores each with $S$, commits the best step, and repeats.

The image-ablation forward pass reuses the cached question encoding, so each candidate costs roughly two scoring passes plus one rollout. We then make triggering selective: look-ahead fires only at boundaries where the first-token entropy is high, or where the step contains a numeric or relational claim about the figure. We report the compute saved and the accuracy lost relative to triggering at every step.

The content-erased images are produced in different ways per benchmark:

  • Chart101: the synthetic plots can be re-rendered without their marks.
  • ChartQA-derived real charts: data regions are inpainted while the OCR'd text is preserved.
  • MATH-Vision diagrams: we use the blank-image ablation plus a version with the lines erased and the labels kept, validated on a hand-checked sample of 200 images.

The second strand of work is a bias audit of the obvious alternative, judge-guided look-ahead, in which an MLLM judge scores the same rollouts. We measure three things as search pressure increases with $K$:

  • whether average step length grows;
  • whether title reliance grows, measured as the Chart101 accuracy gap between title-present and title-removed conditions;
  • whether the judge's verdicts shift when cues are swapped, using the verdict-shift-rate protocol from The Silent Judge transplanted to multimodal pairs.

This audit tests the claim that searching against a biased judge amplifies its bias, independently of whether our method wins.

Baselines

The naive reference point is plain greedy chain-of-thought decoding with the same model and prompt, with no search. A second naive point is direct answering without chain-of-thought, since MATH-Vision reports that chain-of-thought sometimes hurts. For context only, MATH-Vision reports 16.97% accuracy for InternVL-Chat-V1-2-Plus, 30.39% for GPT-4o, and 68.82% for humans. We reproduce the open-model number under our own harness before comparing, and treat any discrepancy as a harness difference rather than a gain.

The strongest existing competitor is judge-scored best-of-N. The policy samples N complete solutions and a multimodal judge picks one. We use MR. Judge-7B if its weights are available, and otherwise a prompted judge from the same model family, because MR. Judge is the strongest multimodal judge reported in this file. The relevant reported figures are a 9.9% margin over GPT-4o on VL-RewardBench and up to 7.7% best-of-N gains on MM-Vet, both from MR. Judge. For MATH-Vision and Chart101 we found no reported result on Lacuna for judge-scored best-of-N. For fairness, N is set so that the total number of forward tokens, including judge tokens, matches our decoder's measured cost per problem, and both methods get the same budget for tuning hyperparameters on a held-out split. Beating this baseline means higher final-answer accuracy at equal compute on MATH-Vision and Chart101, with non-overlapping bootstrap confidence intervals.

The third baseline is judge-guided look-ahead: our exact search procedure, with $S$ replaced by a prompted MLLM judge's score on each rollout. It isolates the question of which scorer to use while holding the search fixed.

Two ablations isolate the key ingredient. Setting $\lambda = 0$ gives likelihood-only look-ahead, which tests whether search alone helps. Setting the horizon to zero, so the contrast is applied to the candidate step without its rollout, tests whether the look-ahead adds anything beyond a per-step contrast. A further ablation compares the blank image to the content-erased image, to test whether targeting text shortcuts specifically matters.

We also compare to DCoT prompting on MM-Vet, where the reported LLaVA-1.5-13B accuracy is 38.4% against a 36.1% base, from the DCoT paper.

Why not just…?

Why not just use best-of-N with a strong judge such as MR. Judge? This is the most serious threat. VL-RewardBench reports that its benchmark accuracy correlates with best-of-N gains at r > 0.9, so a good judge does translate into better answers. Our argument against it is that the judge sees only finished solutions, by which point a hallucinated premise has already shaped every later step. Judges are also weakest at checking visual existence, which is precisely the error we target. The matched-compute comparison settles the question. If best-of-N equals or beats our decoder on the hallucinated-premise errors in MATH-Vision, the case for step-level grounding fails and we would report that.

Why not just describe the image in text and let a text model reason? MLLM-as-a-Judge found judges do better with text descriptions than with raw images, which makes this tempting. But GeoSQA shows reasoning still fails given perfect descriptions, and a model-written description inherits the same shortcut, since a caption can echo the title. We include describe-then-reason as an extra baseline on Chart101's title-removed split. Our prediction is that its accuracy will drop on that split just as direct decoding does.

Why not just apply visual contrastive decoding token by token, with no look-ahead? A per-token contrast between the real and ablated image is cheaper and could capture most of the benefit. The difficulty is that a visual claim is often decided across several tokens: "50" only becomes wrong once it is attached to "maximum." TWEAK's finding that look-ahead matters most in the later half of sentences points the same way. The horizon-zero ablation and a token-level contrast baseline at equal compute test this directly. If either matches the full method, we would drop the look-ahead and report the simpler method.

Why not just fine-tune the policy on grounded reasoning traces? Fine-tuning is complementary rather than competing. It needs labeled or synthesized grounded traces, as in Synthesize Step-by-Step, while our method works at inference time on any white-box model. We do not claim to beat fine-tuning. We test only whether the decoder still adds gains on top of an already-tuned chart model.

Proposed Experiments

The main evaluation runs on MATH-Vision (3,040 problems), Chart101 (both synthetic and real plots, at 5% tolerance), and MM-Vet. For each method we report final accuracy at matched compute.

The first decisive check is error attribution on MATH-Vision. On a sample of 300 failures per method, annotators label each failure as a vision-recognition error or a reasoning error, following the MATH-Vision taxonomy, and also mark hallucinated visual premises in intermediate steps. The core claim predicts that our gains come almost entirely from the recognition and hallucinated-premise categories. If gains appeared mainly in the reasoning category, that would suggest the method helps for some other reason, such as plain search, and the $\lambda = 0$ ablation would show it.

The second decisive check is shortcut reliance on Chart101. For each method we measure the accuracy gap between the title-present and title-removed conditions, and accuracy under the irrelevant visual perturbations from Plot Twist: grid lines, changed markers, and a switched plotting library. We predict our decoder narrows the title gap. We predict judge-guided look-ahead widens it as $K$ grows, with step length growing alongside.

The third check evaluates the contrast as a judge. We score VL-RewardBench preference pairs using the grounding contrast from an open 7B model, and compare against the reported ~62.5% for GPT-4o and Gemini-1.5-Pro and near-chance for open 7B judges. A score well above chance would show that likelihood contrast is itself a less biased judging signal. A score at chance would confine the contribution to decoding.

Two known risks are measured directly. The search might learn to restate irrelevant visual facts simply because they carry high visual dependence. We measure this as the fraction of committed steps that are never used later in the solution, and cap $\lambda$ if that fraction grows. The content-erased ablation might itself leak information. We verify it on the hand-checked sample before running any experiments.