Stubborn Judges: When Multimodal Evaluators Trust Memory Over the Image

Vision-language models are now routinely used as judges: a model looks at an image, reads two candidate answers, and decides which is better. These verdicts then train or select the next generation of models. We propose to study a failure that sits directly in this loop. Sometimes the image shows something the model's training says is unlikely, such as a blue bear, a medieval knight holding a smartphone, or a chart whose bars contradict its title. In those cases a judge may reward the answer that matches what it already "knows" over the answer that faithfully describes the picture. If that happens, then a reward model trained or prompted this way does more than fail to catch hallucinations. It actively pays for them, and it pays most when the visual evidence is surprising.

The idea in two sentences: we build paired test cases in which the faithful answer contradicts common-sense priors, and we measure how often multimodal judges side with the prior. We then use the small set of attention heads already known to carry visual evidence against memory to diagnose and repair this judge stubbornness at verdict time, intervening only when an internal probe signals a conflict.

Core Claim

Our claim is that multimodal judges suffer from parametric-contextual knowledge conflict. Parametric knowledge is what the model stored in its weights during training. Contextual knowledge is what the current input shows. In a judge, this conflict appears as a measurable bias: on counterfactual images, the judge prefers prior-consistent but visually wrong answers. On matched normal images, where the prior-consistent answer is also correct, it does not show this preference.

We also claim that the bias has a specific cause inside the model. When the judge reaches its verdict, too little attention from the upper-layer heads that point at anomalous image regions lands on the region that contradicts the prior. Boosting those heads, but only when a conflict is detected, should close most of the gap without hurting ordinary judging.

The prior work that motivates this has never been put together. Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models shows that multimodal models are more stubborn than text-only models, become more stubborn as they scale from 3B to 72B parameters, and detect conflicts (75–79% accuracy) without acting on them. That work studies models as answerers, never as graders. MLLM-as-a-Judge and VL-RewardBench show three things about multimodal judges: they hallucinate visual premises during step-by-step reasoning, they judge better from text descriptions than from pixels, and they fail mostly on basic existence and recognition. Neither benchmark builds cases where the correct verdict contradicts the judge's priors.

Our hypothesis is that these are one phenomenon. Recognition failures and the step-by-step reasoning paradox are concentrated on exactly the items where perception must overrule memory. That hypothesis is testable and has not been tested.

Research Program

The benchmark. The first piece is a counterfactual preference benchmark for judges. We draw images from three sources. WHOOPS! contributes 500 common-sense-violating images with human explanations. Visual CounterFact and WHOOPS-AHA! come from When Seeing Overrides Knowing. The entity-knowledge and visual-semantic conflict splits of MMKC-Bench complete the set. For each image and question, we write two answers:

  • a faithful answer that describes what the image actually shows;
  • a prior-consistent answer that describes the expected version of the scene.

We follow VL-RewardBench's practice of strict length matching so that the known preference for longer answers cannot decide the outcome. We also randomize answer order and score both orders.

Every conflict pair gets a mirrored control pair built from a normal image, where the prior-consistent answer is the faithful one. WHOOPS! supplies 100 normal control images, and we add more from the source datasets. The control lets us separate two cases. A stubborn judge fails conflict pairs but passes controls. A gullible judge, the counterpart failure described by Hong et al., prefers the unusual answer everywhere. The primary metric is the stubbornness gap: pairwise accuracy on control pairs minus pairwise accuracy on conflict pairs.

The mechanistic analysis. The second piece runs on open judges where attention is accessible: Qwen2.5-VL at 3B, 7B, and 72B, LLaVA-NeXT, Gemma3, and an MR. Judge-style reasoning judge if weights can be obtained. First, we locate the "counterfactual heads" and "factual heads" using the logit-lens procedure of When Seeing Overrides Knowing. That paper found these heads act as spatial pointers, giving anomalous objects a median of 4.41 times the background attention in Gemma3. It studied single-word completions by the answering model, never judging, and never multi-token reasoning.

On a 300-image subset we hand-draw boxes around the anomalous region. We then ask whether the attention mass these heads place inside the box, measured at the verdict token, predicts whether the judge's verdict is correct. When the judge reasons step by step before deciding, we track this mass across the reasoning tokens. Our prediction is that it decays as the judge's own text accumulates. That decay would be a concrete mechanism for the step-by-step reasoning paradox in MLLM-as-a-Judge on this class of items. It would also fit the "reasoning drift" toward text that Functional Attention Control reports in deep layers.

The intervention. The third piece is a conflict-gated intervention. Following Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning, we train a linear probe on mid-to-late-layer hidden states to detect vision-versus-prior conflict. That paper reports 93.2–98.8% AUC for detecting conflicts during reasoning. Here we reuse its method; it does not compete with ours. When the probe fires, we scale up the image-directed attention of the identified counterfactual heads during the verdict and, in the reasoning variant, during the judge's intermediate steps. When the probe is silent, the judge runs unchanged.

Tang et al. found that steering a model toward counter-attitudinal text is much harder than steering toward fact-consistent evidence. That result is a real constraint on our design. Our bet is that steering toward visual evidence through attention runs in the direction these heads already push. When Seeing Overrides Knowing shows these heads causally shift answers. If the asymmetry also holds for visual evidence in judges, the intervention will underperform, and we will report that.

Downstream consequence. The fourth piece checks whether judge stubbornness matters for training and selection. We use each judge as a Best-of-N reranker over sampled answers from a policy model on WHOOPS! explanation questions and MMKC-Bench questions. We then measure how often the selected answer is prior-consistent but visually wrong. VL-RewardBench reports that judge accuracy correlates at Pearson r > 0.9 with Best-of-N gains on MMMU-Pro. We ask whether the stubbornness gap predicts Best-of-N degradation on conflict items in the same way.

Baselines

Naive reference. The do-nothing reference is each judge used as published, with a standard pairwise rubric prompt. We run it with and without step-by-step reasoning, using the same prompts, both answer orders, and greedy decoding. We need both modes because the step-by-step paradox predicts reasoning may hurt here.

Strongest trained judge. This is MR. Judge, from MR. Judge: Multimodal Reasoner as a Judge. It trains Qwen2.5-VL with reasoning traces distilled from a text-only teacher and then reinforcement learning. It reports 75.5% overall accuracy on VL-RewardBench, against 65.8% for GPT-4o in the same paper; VL-RewardBench itself reports frontier models peaking near 62.5% macro accuracy. It is the strongest competitor because it is explicitly tuned against hallucination, scoring 83.2% on that subset. If any existing judge already handles counterfactual pairs, it should be this one.

No reported result was found on Lacuna for any judge on counterfactual preference pairs. Beating it therefore means one of two things. Either our gated intervention applied to an untrained Qwen2.5-VL-7B judge matches or exceeds MR. Judge-7B on conflict-pair accuracy, or the intervention applied on top of MR. Judge shrinks its stubbornness gap by at least half. In both cases, VL-RewardBench accuracy and control-pair accuracy must stay within one point of where they started. Frontier GPT-4o is evaluated behaviorally as a ceiling reference.

Strongest training-free attention methods. These are Functional Attention Control and attention-guided Ensemble Decoding. Functional Attention Control rescales shallow perception heads and deep reasoning heads by fixed gains of 1.1–1.4 and reports a 4.2-point average gain across MathVista, MathVision, HallusionBench, MMStar and SEED-Bench. Ensemble Decoding weights logits from high-resolution crops by where the model attends and reports +6.47% POPE accuracy. Both target general hallucination, not conflict, and neither is gated. We run each on the same judges, tuning its published hyperparameters on the same 20% development split we use for our own gain and probe threshold. Ensemble Decoding runs only on the patch-projector models it supports.

Ablations. Three ablations isolate the key ingredients. Removing the gate, so that counterfactual heads are always boosted, tests whether gating is what protects control pairs. Boosting an equal number of random heads, or all image-attending heads, tests whether the specific counterfactual heads matter. Boosting factual heads, which the source paper used to reach 74% factual accuracy in LLaVA-NeXT, should widen the gap and serves as a sign check.

Why not just…?

Why not just give the judge a text description of the image? MLLM-as-a-Judge finds judges do better from descriptions than pixels, so this is the obvious fix. But the description must come from somewhere. If a vision-language captioner writes it, that captioner is subject to the same stubbornness MMKC-Bench documents and may describe the blue bear as brown. We test this directly with a caption-then-judge baseline using the same backbone as captioner. We also run an oracle variant with human-written descriptions, from the WHOOPS! annotations, which gives an upper bound. If model-written captions close the gap, the problem lives in perception rather than in the verdict, and our attention intervention is unnecessary. That would itself be a clear finding.

Why not just tell the judge to trust the image over its knowledge? Instruction-based fixes are cheap and must be tried. We include a prompt that explicitly warns that images may depict unusual scenes. MMKC-Bench, however, shows models detect conflicts at 75–79% and still answer from memory, so knowing about the conflict is not the bottleneck. This also suggests a warning that raises awareness will not change the verdict. The control pairs will show whether such a prompt simply makes the judge gullible, trading stubbornness for a preference toward whatever answer is unusual.

Why not just fine-tune the judge on counterfactual preference pairs? This is the real threat. MR. Judge's pipeline of synthesizing corrupted responses can be pointed at counterfactual images, and supervised training on a few thousand such pairs may well erase the gap. We will run exactly this: Qwen2.5-VL-7B fine-tuned on the training split of our benchmark with matched compute. We will then test it on held-out source datasets, for example training on WHOOPS! and testing on MMKC-Bench and Visual CounterFact. The question is whether it generalizes or merely memorizes which anomalies are allowed. If fine-tuning wins across sources, we will say so. The contribution then narrows to the benchmark, the diagnosis of judge stubbornness, and the downstream evidence that it corrupts Best-of-N selection. The training-free intervention would remain useful only for closed or frozen judges.

Proposed Experiments and Decisive Checks

The first decisive check is whether the phenomenon exists. Across at least four open judges, conflict-pair accuracy should fall below control-pair accuracy by a margin larger than the order-swap disagreement rate. By analogy with MMKC-Bench, the gap should widen from Qwen2.5-VL-3B to 72B. A flat or shrinking gap with scale would contradict the transfer of answerer stubbornness to judges.

The second check is mechanistic. Counterfactual-head attention mass inside the anomaly box at the verdict token should predict verdict correctness on conflict pairs, measured by AUC on held-out items. That mass should also decline across the judge's reasoning tokens more steeply on items where reasoning flips a correct verdict to a wrong one.

The third check is causal and comparative. The gated intervention must beat Functional Attention Control, Ensemble Decoding and the instruction baseline on conflict-pair accuracy at equal tuning budget. It must also leave control-pair accuracy and VL-RewardBench accuracy within one point, and its random-head and ungated ablations must underperform it.

The fourth check is downstream. In Best-of-N reranking on WHOOPS! and MMKC-Bench questions, the rate of selected prior-consistent errors should track each judge's stubbornness gap and fall under the intervention.

We also run a smaller transfer study on Plot Twist's Chart101, where misleading titles drop answer accuracy from about 79% to the 30% range. Pairing title-consistent with data-consistent answers tests whether judge stubbornness extends from common-sense priors to textual shortcuts inside the image. That extension is the part of the multimodal reasoning gap where perception must again overrule an expectation.