Ask Before You Look: Blind, Region-Grounded Claim Checking for Multimodal Judges

Multimodal language models are increasingly used as automatic graders: given an image, a question, and one or more candidate answers, a "multimodal LLM-as-a-judge" decides which answer is better. These judges have a specific failure. When a candidate answer states something false about the image in fluent, confident prose, the judge tends to believe it, even though the same judge can see the correct detail when asked about it directly. This effect is called response anchoring, and it has been documented in Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge for Mathematical and Scientific Reasoning Evaluation. We propose to separate looking from being persuaded. The judge first breaks each candidate into atomic visual claims. Each claim is then rewritten as a neutral question that does not reveal the candidate's answer: "how many red cubes are left of the cylinder?" rather than "check that there are three red cubes left of the cylinder." An open-vocabulary detector, meaning a model that draws boxes around objects named by arbitrary text, finds the image regions the question refers to. The judge answers the neutral question from enlarged crops of those regions, without seeing the candidate text. Only after that does a text-only step compare the judge's own answers to each candidate's claims and produce a score or a preference.

The bet is that two separate weaknesses of current judges have two separate fixes, and that they combine. Anchoring is fixed by blinding: the judge never sees the candidate's wording while it perceives. Blurry, low-resolution perception of small details is fixed by grounding: the detector tells the judge where to zoom. The design also has to handle a third problem that the closest prior work leaves open. Detectors miss small and unusual objects. When a detector finds nothing, our protocol treats that as "unknown" and falls back to a blind full-image question. It never treats a miss as evidence that the claim is false.

What exists and how this differs

The closest work is OVFact. It extracts entities from a long caption with an LLM and checks whether open-vocabulary detectors and segmenters find them. This yields a pointwise precision and recall score that was good enough to filter ShareGPT4v training data, and humans preferred the resulting model's outputs 68.2% of the time. OVFact supplies our claim-extraction step, but it differs from this proposal in three ways. Its verdict is the detector's verdict, so detector misses become false hallucination flags, which the paper itself acknowledges. It covers object and attribute presence, not spatial relations or counts. And it scores one caption at a time rather than comparing candidates.

Connecting the Dots (GroundingAgent) runs the pipeline in the opposite direction: a reasoning LLM picks which detector box matches a query. Its analysis matters for our design. Accuracy rises from 65.1% to about 90% when the regional captions are correct, which shows that routing perception through text captions is the bottleneck. This constrains our design: our judge answers questions directly from pixel crops, not from captions of crops.

MG-LLaVA shows that feeding detector-located high-resolution object crops into a multimodal model improves fine-grained perception. This supplies the zoom mechanism. MG-LLaVA, however, is an assistant and is never used to grade other answers.

Two papers motivate the proposal. MLLM-as-a-Judge finds that step-by-step reasoning in visual judging can hurt, because the judge invents visual premises along the way. It also finds that judges agree with humans more when given a text description than the raw image. Perception-Judge addresses anchoring by training a 7B judge with reinforcement learning (GRPO, which rewards the model for ranking a group of candidates in the right order) on ordered triplets of correct, perceptually perturbed, and fully corrupted answers. That work uses no explicit regions, and its authors report that the model still struggles with spatial layouts.

Our contrast with Perception-Judge is the central scientific question. Can a structural change to what the judge sees and when match or exceed parametric training, and does it transfer outside the training domain?

Core Claim

A judge that answers neutral, region-grounded questions before it sees any candidate's claims will show a much smaller anchoring gap than an unconstrained judge with the same backbone. We define the anchoring gap as the drop in a judge's accuracy on a visual fact when a false version of that fact is embedded in fluent candidate text. We claim this gap shrinks most on counts and spatial relations. We further claim that blinding and grounding contribute separately: blinding mostly removes anchoring, and grounding mostly fixes small and fine-grained details.

Research Program

The system has four stages, all built on an open-weight judge backbone (Qwen2.5-VL-7B, so that we can compare directly with Perception-Judge).

In the first stage, a text model splits each candidate into claims and labels each by type: existence, attribute, count, spatial relation, or reading text or values from a chart. It then rewrites each claim as a neutral question with an open answer slot. Claims that share a question across candidates are merged. In pairwise judging, this means both answers are checked against one perceptual answer, rather than each being judged in its own anchoring context.

In the second stage, the noun phrases in each question prompt an open-vocabulary detector to return boxes. For relation claims, we answer the relation two ways. One is from box geometry, where it applies (left of, above, inside). The other is by asking the judge about a crop that contains both objects. When the two disagree, we record the disagreement.

In the third stage, the judge answers each neutral question from the enlarged crops plus the full image. If no box is found, the claim is marked ungrounded and answered from the full image alone, with no zoom.

In the fourth stage, a text-only comparison step aligns the judge's answers with each candidate's claims. It then produces one of three outputs: a pointwise score, a pairwise preference decided by the claims where the candidates differ, or a batch ranking. Every verdict comes with an audit trail linking claim, question, region, and answer. This trail lets us attribute each error to one stage, in the spirit of AgentEval's step-level evaluation: a missed box, a wrong crop answer, or a wrong comparison.

Baselines

The naive reference point is the backbone judge prompted directly to score or compare the candidates, with no reasoning, under the same prompts and image resolution. Alongside it we report trivial heuristics, "always prefer the longer answer" and "always give a high score." These expose the verbosity and high-score biases that MLLM-as-a-Judge documents. Any gain must clear both.

The second baseline is the same backbone with analyze-then-judge chain-of-thought, and a frontier model such as GPT-4o in the same setting. This is what a practitioner would use by default. On MLLM-as-a-Judge, GPT-4V reaches a Pearson correlation of 0.490 with human scores and roughly 79% agreement on pair comparison. On JudgeAnything, Gemini 1.5 Pro reaches 70.6% pairwise agreement on understanding tasks (Judge Anything). Beating these means higher human agreement on the same splits.

The third baseline is the "vision expert" setting. A captioner describes the image densely once, and a text-only LLM judges against that description. This is the cheapest way to break anchoring, since the judge never sees pixels, so it competes directly with our blinding idea.

The fourth baseline is an OVFact-style detector-only judge: a claim is true if and only if the detector grounds it. It isolates whether the multimodal model's crop answers add anything beyond detector presence.

The strongest existing method is Perception-Judge-Flex-7B. It reports 85.7% accuracy at identifying perceptual errors on PPJD, against 69.5% for the Qwen2.5-VL-7B base. We run it with its released weights, at the same resolution, on the same test items. Because it was trained on PPJD and our method is training-free, a fair comparison has three parts. We compare on PPJD categories held out from its training where possible. We compare on out-of-domain sets (MLLM-as-a-Judge, JudgeAnything, and our spatial set below). And we run our protocol on top of Perception-Judge's weights to test whether the two gains add. Beating it means at least matching 85.7% on PPJD while clearly exceeding it on spatial and out-of-domain items.

The ablations isolate the key ingredient. One runs blind neutral questions on the full image with no detector (blinding without grounding). Another gives the judge grounded crops but also shows it the candidate's claims while answering (grounding without blinding). A third uses ground-truth boxes instead of detector boxes, to measure the cost of detector misses. A fourth removes the "ungrounded means unknown" fallback, which reproduces OVFact's failure mode. All variants share the backbone, decoding settings, and a fixed, equal prompt-tuning budget on a small development split.

Why not just…?

Why not just use a frontier multimodal model with chain-of-thought as the judge? The evidence in the file runs against this. MLLM-as-a-Judge reports that visual chain-of-thought can lower judging accuracy because the model invents visual premises and then relies on them. Perception-Judge shows that anchoring persists in strong models. Still, frontier models improve quickly, so we include GPT-4o as a baseline and measure its anchoring gap directly rather than assuming it is large. If its gap is already near zero, the blinding component is unnecessary for frontier judges, and we would say so.

Why not just caption the image and let a text-only judge decide? This also avoids anchoring, and MLLM-as-a-Judge found that descriptions sometimes beat raw pixels. But a fixed caption is written before anyone knows which details the candidates will dispute. GroundingAgent's 65.1% versus about 90% gap shows how much caption omissions cost. Our questions are asked after the claims are known, so they target exactly the disputed details. The vision-expert baseline tests this head-on, and stratifying results by claim type will show whether captions lose mainly on counts and relations, as we expect.

Why not just ask neutral questions on the full image and skip the detector? This is the most serious threat. Blinding may account for most of the gain, with grounding adding latency for little benefit. The blind-full-image ablation settles it. We predict grounding matters mainly for small objects and dense scenes, the same regime where MG-LLaVA's crops helped. If its benefit is confined to that stratum, we will recommend running the detector only on demand, for example when the judge's answer has low confidence.

Why not just fine-tune the judge as Perception-Judge does? Fine-tuning works in-domain, as the 85.7% shows. But it needs perturbation data generated with GPT-4o, and its authors report persistent spatial weaknesses. Our protocol needs no training and produces auditable evidence. The stacked experiment tells us whether the two approaches are substitutes or complements. If fine-tuning alone matches the protocol out of domain, the protocol's main value is interpretability, and we will report that honestly.

Proposed Experiments

The decisive experiment is a controlled anchoring test. We build a spatial-and-count perturbation set from RefCOCO-family and OV-VG images, whose ground-truth boxes let us generate both correct and minimally perturbed statements without human labels. Example perturbations swap "left of" for "right of," change a count, or change an attribute of a named object. OV-VG's object-size labels let us split results by small and large targets, where small-target zero-shot grounding is near zero. For each fact we measure judge accuracy in two conditions: asked neutrally, and embedded false in a fluent answer. The difference is the anchoring gap. The core claim predicts four things. Blinding shrinks the gap on all claim types. Grounding raises neutral accuracy mainly on small targets. The "unknown" fallback prevents the small-target collapse that detector-only judging shows. And ground-truth boxes bound how much better detection could help.

We then test transfer. On PPJD we report accuracy at identifying perceptual errors, and on MathVista-derived items within it we check whether the gains extend to charts and diagrams, where the detector is least reliable. On MLLM-as-a-Judge (HQ and HARD splits) and JudgeAnything we report Pearson correlation with human scores, pairwise agreement, and batch-ranking agreement. This lets us check whether the comparative mode avoids JudgeAnything's checklist paradox, in which checklists helped scoring but hurt pairwise comparison. Our explanation for that paradox is that checklist items were invented without reference to the image. In our protocol, every item is a claim that some candidate actually made and that has been answered from the image, so we expect pairwise judging to improve rather than degrade.

Finally, the stage-level audit trails let us count what fraction of residual errors come from detection, crop perception, or comparison. This tells future work which component to improve next. We also report wall-clock cost per judgment against each baseline, since the multi-stage design must justify its latency.