Checking the Picture First: Prerequisite-Ordered Verification for Cheap, Anchoring-Resistant Multimodal Judges

When a vision-language model is used as a judge, meaning a model that grades another model's answer to a question about an image, it tends to be fooled by fluent reasoning. If a candidate answer misreads a bar in a chart and then reasons carefully from the wrong number, the judge usually approves it. The judge reads the confident argument first and lets it override what the image shows. The usual fix is to make the judge think longer before grading. That is expensive, and on images it can backfire, because the judge may invent visual details mid-rationale and then trust them. We propose the opposite. Break the candidate answer into a small dependency graph of the kind education researchers use for prerequisite maps. The leaves are claims about the image ("the y-axis value for 2019 is 12"), and the inner nodes are deductions that use those claims. Each image claim is checked alone against the image, with the rest of the answer hidden, using a one-token yes/no query. The judge only spends expensive reasoning on deductions whose image prerequisites have passed.

The two-sentence version is this. A multimodal judge should verify the perceptual prerequisites of an answer in isolation before it ever sees the argument built on them, which removes the anchoring effect. Because every downstream step depends on those prerequisites, a single failed load-bearing percept ends the evaluation early, so accuracy on perceptual errors and cost should improve together rather than trade off.

Core Claim

Our claim is specific and falsifiable. Response anchoring in multimodal judges is largely a context-ordering failure, not a capacity failure. Hiding the downstream rationale while verifying each image claim should recover most of the perceptual-error detection that currently requires reinforcement-learning fine-tuning on perturbed data. The dependency structure then does two further jobs that a flat list of claims cannot. First, it separates load-bearing percepts, those that are ancestors of the final answer, from incidental ones, so harmless misreadings do not sink a correct response. Second, it gives a natural early-exit order, so token cost scales with the number of load-bearing percepts rather than with rationale length.

Several findings in the file motivate this claim. Mitigating Perceptual Judgment Bias documents the anchoring effect: judges reward fluent chains built on hallucinated visual features. Its fix is to train a 7B judge with group-relative policy optimization on its Perceptually Perturbed Judgment Dataset. We instead test whether restructuring inference alone gets most of that gain without training. MLLM-as-a-Judge reports that free-form chain-of-thought can lower multimodal judging accuracy because judges hallucinate premises mid-trace. Our design avoids that failure by never asking the judge to write a free rationale about the image.

Flex-Judge and MR. Judge seem to contradict us, since both find that structured <think> reasoning makes better judges. We read these results as compatible with ours. Reasoning helps on deductions, which is where we still spend it. What we reject is spending it on re-describing the image.

The technique we reuse comes from AgentEval. It grades each step of a text agent trace against its upstream parents in a directed acyclic graph and raises failure-detection recall from 0.41 to 0.89. That work is text-only and makes one GPT-4o call per node. We carry its structure into a regime it did not address, where the parents are image claims, and we make the node checks cheap.

Research Program

The system has three parts.

Parser. A parser model rewrites a candidate response into atomic steps. Each step carries a type (percept or deduction) and explicit parent pointers. The file does not report a result for the specific parser we plan to use, so parser reliability is itself something we measure. We will measure it against human-built graphs on a subset. We will also test robustness by randomly deleting edges and mislabeling node types.

Percept verifier. The verifier sees only the image, the question, and one claim, and outputs a yes/no token. The image prefix is encoded once and its key-value cache is reused across all claims for that image. This matters because re-encoding image tokens per claim would otherwise dominate the cost.

We also test a second, generation-free verifier. It is a linear probe on the frozen hidden states of the same model at the claim's final token, following INSPECTOR. INSPECTOR showed that 1B–1.7B text models encode "this is wrong" in mid-to-upper layers and reach 80–90% binary filtering accuracy. Whether such probes survive the gap between continuous image features and discrete text features is an open question. Learning Robust Visual-Semantic Embeddings and FindRec both describe these features as poorly integrated clusters. Our experiment answers the question directly by probing the claim token after it has attended to the image.

Deduction checker. This component checks each deduction given only its parents' verified content. That context is text, so a cheaper text model can do this step. Arithmetic steps can be executed symbolically, in the spirit of the chart-to-code decomposition in Synthesize Step-by-Step.

Scoring rule. A response is accepted only if every ancestor of its final answer verifies:

$$s(r) = \prod_{v \in \mathrm{Anc}(a_r)} p_v,$$

Here $a_r$ is the final-answer node of response $r$, $\mathrm{Anc}(a_r)$ is the set of steps it depends on, and $p_v$ is the verifier's probability that step $v$ is correct. Pairwise preferences compare $s$ across candidates.

Evaluation order and routing. Percept nodes are evaluated first, in order of how many descendants they have. Evaluation stops as soon as the running product falls below a threshold. This is a routing policy, but it is driven by the answer's own dependency structure rather than by a learned difficulty estimate as in RACER.

Educational use. The same graph yields a diagnosis of whether an answer failed at seeing or at reasoning. This is the perceptual-versus-deductive distinction that curriculum-grounded feedback systems such as LearnLens lack for diagrams.

Baselines

All learned and prompted baselines use Qwen2.5-VL-7B as the backbone, the base model of MR. Judge and Perception-Judge, unless stated otherwise. Every method sees the same images, candidates, and splits. Any threshold or prompt tuning gets the same development set and the same number of configurations. Cost is reported as generated tokens, prefill tokens, and wall-clock time per judgment on the same hardware.

Naive reference. Base Qwen2.5-VL-7B prompted directly for a verdict with no rationale. Park et al. report 69.5% perceptual-error identification for it on PPJD. On VL-RewardBench, the VL-RewardBench paper reports that open 7B judges score near chance.

Strongest trained competitor on perceptual errors. Perception-Judge-Flex-7B from Mitigating Perceptual Judgment Bias, which reports 85.7% on PPJD perceptual-error identification. It is the strongest competitor because it was built for exactly this failure, using training we do not require. Our untrained graph judge on the same backbone counts as beating it if it matches or exceeds 85.7% on PPJD categories held out from any tuning, at lower generated-token cost. If it falls short on PPJD but wins on out-of-distribution data (see below), we will report that honestly as a partial result.

Strongest general multimodal judge. MR. Judge-7B, which reports 75.5% overall and 83.2% on the hallucination subset of VL-RewardBench, against 65.8% for GPT-4o. Beating it means matching 75.5% overall while generating substantially fewer tokens, or exceeding 83.2% on the hallucination subset at equal cost.

Cost-routing baseline. A RACER-style router that chooses between a no-reasoning and a full-reasoning judge on the same backbone. Its authors report halving token cost at all-reasoning accuracy on text benchmarks. We found no reported result on Lacuna for RACER on multimodal inputs, so we will run it ourselves with the same development budget.

Ablations isolating the key ingredients. "Flat" verification checks every extracted claim in isolation but ignores dependencies: all claims count, and there is no early exit. "Anchored" verification is identical to our method except that each percept check sees the full candidate response in context. "Probe-only" replaces generative percept checks with the hidden-state probe.

Why not just…?

Why not just prompt the judge to describe the image first and then grade? Prompting a strong model to describe first is exactly the free-form multimodal rationale that MLLM-as-a-Judge found can reduce accuracy, because invented details become premises. It also leaves the candidate's fluent argument in context during verification, which is the condition under which Park et al. observe anchoring. The "anchored" ablation tests this directly: same claims, same model, only the context differs. If anchored verification is as good as isolated verification, our central hypothesis is wrong.

Why not just extract claims and check each one, without any graph? This is the real threat, and we treat it as the main ablation rather than argue it away. A flat checker keeps the isolation benefit. It loses two things. It cannot tell a harmless misread of an irrelevant tick label from a load-bearing one, so it should over-reject correct answers. It also cannot stop early in a principled order. We expect flat and graph to be close on PPJD, where perturbations are load-bearing by construction. We expect the graph to win on false-rejection rate and cost on naturally occurring responses to MathVista and ChartQA. If flat matches the graph on both, we will drop the graph claim and report claim isolation alone as the contribution.

Why not just caption the image and use a cheap text judge? GeoSQA shows that even human transcriptions of diagrams do not rescue reasoning. More to the point for judging, a caption fixes one reading of the image before the judge knows which details the candidate relies on, so it will often omit the value in dispute. We include caption-then-text-judge as a baseline. Our method differs because it queries the image once per specific claim, so we expect caption-then-judge to lose mainly on chart and OCR categories.

Why not just fine-tune on perturbed data, as Park et al. did? That is a legitimate alternative and is our strongest baseline. The open question is transfer. A judge trained on PPJD's 11 categories may learn perturbation styles rather than verification. We therefore also report both methods on MLLM-as-a-Judge, where GPT-4V reaches about 79% pairwise agreement with humans, and on VL-RewardBench, neither of which was used for training.

Proposed Experiments

Main comparison. We evaluate on four benchmarks with the baselines above:

  • PPJD, measuring perceptual-error identification accuracy.
  • VL-RewardBench, measuring pairwise accuracy overall and on the hallucination subset.
  • The MLLM-as-a-Judge pairwise and HARD subsets, measuring human agreement.
  • A new attribution set: about 500 candidate solutions to MathVista, ChartQA, and ScienceQA image questions, sampled from open models. Annotators mark the first wrong step and label it as perceptual or deductive.

Metrics are accuracy, false-rejection rate on correct responses, generated and prefill tokens per judgment, and, on the attribution set, root-cause accuracy scored as in AgentEval, which reports 72% against an 81% human ceiling on text traces.

Four decisive checks.

  1. Anchoring. The gap between isolated and anchored verification should be large on PPJD's visually perturbed pairs and near zero on fully corrupted ones.
  2. Cost scaling. Cost should grow with the number of load-bearing percepts and stay flat as rationale length grows. We will test this by padding candidate rationales with correct but irrelevant reasoning. A chain-of-thought judge's cost should rise under padding while ours should not.
  3. Probe transfer across modalities. If the probe verifier reaches within a few points of generative percept checks, INSPECTOR-style evaluation carries over to multimodal judges. If it collapses on chart and OCR claims, that locates where the modality gap bites.
  4. Parser robustness. We plot accuracy against injected graph errors. The method is only worth adopting if moderate parse noise degrades it more gracefully than the flat checker's over-rejection penalty.