Does the Judge Get Lost Too? Long-Context Perceptual Bias in Multimodal LLM Judges

Researchers now routinely grade a vision-language model's free-form reasoning by asking another, stronger multimodal model to act as the judge: read the image and the answer, then decide whether the reasoning is right. On single diagrams this already goes wrong in a known way. Judges reward fluent explanations that misread the picture. We ask what happens when the picture the judge must check is one of thirty images in a long multi-page document.

We already know that models answering questions over long multi-image inputs lose track of material in the middle. The judge is the same kind of model, so it should fail the same way. A judge that cannot find the right image should fall back even harder on how convincing the text sounds. Long context also creates an error that short-context hallucination taxonomies, the existing catalogues of error types, do not contain: a reasoning step that reports a real visual fact, but one taken from the wrong image. We propose to measure how judge reliability decays with context length and evidence position, error type by error type, and to test one targeted fix. The fix is to have the judge check each claim against the image the question actually requires, not against whichever image best matches the claim.

Core Claim

We make two claims, and each can be falsified on its own.

The first is diagnostic. Judges detect perceptual errors less well as distractor images are added, and less well again when the relevant image sits mid-sequence. The drop should be larger for perceptual errors than for pure logic errors, because logic errors can be caught from the text alone. A new category, cross-image misattribution, should be the hardest of all for long-context judges.

The second is about method. The obvious repair is to let the judge retrieve the images most similar to each reasoning step. That repair is confirmation-biased against exactly this new category: a claim like "the second chart shows 7 units" will retrieve the distractor chart that does show 7. Requirement-anchored verification avoids this. The judge first identifies, from the question alone, which images the problem depends on. It then verifies each step only against those images, and it flags any step whose content is supported only by an image outside that set.

Relation to Prior Work

Three papers motivate the diagnosis.

Mitigating Perceptual Judgment Bias names the "plausibility trap" and builds the Perceptually Perturbed Judgment Dataset (PPJD). Each item is a set of ranked responses: a correct one, one with a perceptual slip, and one with a slip plus corrupted reasoning. The paper trains a judge on these items, but only on single-diagram tasks.

MileBench and Multimodal Needle in a Haystack show that answer models lose mid-sequence images and collapse in dense image collections. Neither studies models acting as judges, and neither breaks results down by error type.

Lost in the Middle establishes the positional effect in text.

Four papers supply the error categories and step-level judging methods we reuse.

MIRAGE isolates reasoning-induced hallucinations, where the image is described correctly but the conclusion is wrong, and scores reasoning chains with an LLM judge. Its items are short and single-image.

GM-PRM critiques each reasoning step for intent, image alignment and logic. We borrow that three-way step check but feed it different evidence.

ReMI and VCBench decompose multi-image errors into perception and logic. They stop at 2 to 6 images, and they disagree about whether combining images into one helps.

LongHalQA is the closest long-context hallucination benchmark, and it constrains our design. It deliberately uses multiple-choice questions to avoid needing an LLM judge. It also finds that chain-of-thought hurts smaller models in long contexts. That warns us that any fix which makes the judge write longer rationales may backfire, so we measure judge output length as a cost.

Research Program

We first build a controlled stress test. Each base item comes from PPJD or MIRAGE and contains:

  • a question,
  • its required diagram(s),
  • a correct response,
  • a perturbed response with a known, labelled error.

We place the required diagram among $N \in \{0, 4, 16, 32, 64\}$ distractor images drawn from the MileBench and MMNeedle image pools, at relative depth $d \in \{0, 0.25, 0.5, 0.75, 1\}$. Multi-image items from VCBench and ReMI are padded the same way.

We then add a fourth perturbation family that does not exist in current data. A value or attribute in the correct response is replaced by one that is true of a distractor image. This tests cross-image misattribution. It requires choosing distractors that share a type with the target, for example a second bar chart, which we do by layout and type matching. Because every error is planted, labels are exact and need no human annotation. A 300-item human audit checks that the perturbations are unambiguous, which addresses the silent-failure risk raised by Auditing the Audit.

Next we measure judge accuracy as a function of $N$, $d$ and error category for several judges:

  • open judges: Qwen2.5-VL-7B and the PPJD-trained Perception-Judge,
  • a step-level critic in the style of GM-PRM,
  • a closed frontier judge with a long context window.

We report both detection accuracy and PPJD-style ranking accuracy. A verdict flip is a case where the judge correctly penalizes a perturbed response at $N=0$ but accepts it at large $N$; we log these separately.

Finally we implement requirement-anchored verification with no extra training:

  1. The judge reads only the question and image indices, then names the required images.
  2. Each response step is verified against crops of those images only.
  3. A support check asks whether a step's visual claim matches an unrequired image better than any required one; if so, the step is flagged as misattributed.

Baselines

The naive reference point is the base judge given the full long context in a single prompt. We use Qwen2.5-VL-7B, which the PPJD paper reports at 69.5% accuracy in identifying perceptual errors in the short, single-diagram setting. That is our $N=0$ anchor, and every judge is rerun at $N=0$ on our items so the comparison uses identical data.

The strongest existing short-context method is Perception-Judge-Flex-7B, reported at 85.7% on the same PPJD measure. It is the most direct competitor because it was trained specifically against the plausibility trap. If its advantage survives long context, the problem is already solved by training. We use the released judge without retuning.

The strongest step-level critic is a GM-PRM-style judge that checks intent, image alignment and logic per step over the full context.

The most serious competitor is claim-similarity retrieval. Each step is embedded, the top-$k$ most similar images are retrieved, and the judge sees only those. We give it the same base judge, the same $k$ as the number of images our method selects, and an equal prompt-tuning budget of a fixed number of development-set iterations.

A frontier long-context judge such as GPT-4o serves as a ceiling, not as something to beat. No reported result on Lacuna was found for any judge's accuracy on long-context multimodal judging, so all long-context numbers will be new.

Two ablations isolate the key ingredient. In the first, requirement selection is replaced by oracle required images; this separates selection errors from verification errors. In the second, the support check is removed; this tests whether the out-of-set comparison is what catches misattribution.

Our method counts as beating the field only under two conditions. At $N \geq 32$ it must keep at least most of its $N=0$ accuracy, with the gap closed relative to the full-context judge. And on the misattribution category it must clearly beat claim-similarity retrieval, using paired bootstrap confidence intervals over items.

Why not just…?

Why not just retrieve the relevant images and judge in short context? This is the real threat. On perceptual slips and logic errors we expect claim-similarity retrieval to roughly match our method. If it also matches on misattribution, the honest conclusion is that long-context judging reduces to retrieval, and we will report it that way. Our prediction that it fails rests on a specific mechanism: similarity search finds the image that supports the false claim. The planted misattribution items test exactly this.

Why not just caption every image and use a text-only judge? GeoSQA shows that perfect human transcriptions of diagrams did not rescue reasoning. Plot Twist shows that accuracy collapses from about 79% to about 30% once text stops leaking the answer. Captions also discard precisely the fine values that perceptual perturbations alter. We still include a caption-then-text-judge arm, because if it holds up the result would be cheap and useful.

Why not just use the PPJD-trained judge or a bigger frontier model? Training against plausibility may transfer, but the MMNeedle results argue against scale alone being enough. There, GPT-4o's retrieval accuracy fell from 97.00% to 26.90% as image density grew. A judge that cannot locate evidence has nothing to be unbiased about. Both are run as baselines, so the curves themselves will settle the question.

Decisive Checks

The diagnosis fails if full-context judge accuracy is flat in $N$ and $d$. It also fails if perceptual and logic errors decay at the same rate, which would mean degradation is generic rather than perceptual.

The method fails if claim-similarity retrieval matches it on misattribution. It also fails if oracle requirement selection shows that most remaining errors come from verification rather than selection.

We will also check whether verdict flips concentrate at mid-sequence depths, which would tie the judge's failures directly to the lost-in-the-middle effect. And we will track judge output length, to confirm that our gains do not come with the rationale bloat that LongHalQA links to degradation.