Vision-language models, which pair an image encoder with a large language model, are increasingly aligned by preference tuning. In this setup the model is shown pairs of its own answers and trained to prefer the better one. A cheap and growing way to decide which answer is "better" is to ask another vision-language model to judge. These judges are known to hallucinate. They credit objects that are not in the image, they reward fluent but visually wrong explanations, and they favor long answers. Preference tuning is known to harden whatever bias sits in its training signal. Nobody has measured what happens when the two are chained together. This proposal asks whether a judge's visual blind spots become the tuned model's hallucinations, and whether they arrive at a rate worse than the judge's error rate alone would predict.
Our hypothesis is specific. A judge and the model it trains usually share language-model backbones and the same statistical priors, such as "a dining table implies a knife." As a result, judge errors are not random label noise. They land exactly where the model being tuned is already weakest. Direct Preference Optimization (DPO), a method that fits the model directly to preferred-versus-rejected pairs relative to a frozen reference copy, then pushes the model further into those shared weak spots. If this holds, the fix is not simply a more accurate judge. The fix is to identify, before labeling, which kinds of visual claims the judge cannot verify, and to verify those claims against the pixels by other means.
Core Claim
Preference tuning on labels from a multimodal judge does not merely inherit the judge's errors. It amplifies them selectively, in the categories of visual claim where the judge's blind spots overlap with the tuned model's own language priors. A random-noise model of judge errors will underpredict this damage.
We define the effect precisely. Let $e_c$ be the fraction of preference pairs in claim category $c$ that the judge labels wrongly. Examples of categories are existence of a co-occurring object, count, color, and spatial relation. Let $\Delta h_c^{\text{judge}}$ be the change in the tuned model's hallucination rate in category $c$ after training on judge labels. Let $\Delta h_c^{\text{rand}}$ be the same change after training on correct labels in which a random $e_c$ fraction of that category's pairs are flipped. The amplification ratio is
$$A_c = \frac{\Delta h_c^{\text{judge}}}{\Delta h_c^{\text{rand}}}.$$
$A_c \approx 1$ means the judge behaves like unstructured noise. $A_c > 1$ means its errors are systematic and preference tuning concentrates them. We predict $A_c > 1$ where the judge's blind-spot rate and the reference model's own hallucination rate are both high, and $A_c \approx 1$ elsewhere.
Where This Sits
Three lines of prior work motivate the question without answering it.
The first documents what judges get wrong. MLLM-as-a-Judge benchmarks vision-language models as evaluators. It finds high-score clustering, verbosity bias, self-preference, and a "CoT paradox" in which judges hallucinate visual details in their own reasoning and then rely on them. It only mentions preference tuning as future work. Mitigating Perceptual Judgment Bias names the "plausibility trap." It separates judges that cannot see a detail from judges that can see it but anchor to the candidate's false description, and it trains a better judge. MR. Judge likewise trains a reasoning judge on synthesized hallucinated candidates. All three treat the judge as the end product. None traces what a judge's residual errors do to a model trained on its labels, which is our object of study.
The second line shows that preference tuning hardens biases already present in its inputs. The Root Shapes the Fruit shows in text-only models that latent biases in the DPO reference model are solidified. It reports, for example, a 91.53% stigmatizing-selection rate for Llama 13B after supervised tuning plus DPO. The Silent Judge shows that text judges follow superficial cues while acknowledging them 0% of the time, so their errors cannot be caught by reading their rationales. We carry the "biased root" logic into vision. Our prediction that amplification concentrates where judge and reference model share a prior is the multimodal analogue of their reference-dependence finding, and it is directly testable.
The third line supplies the methods and the evidence that the shared prior exists. POPE shows that object hallucination follows co-occurrence statistics and a strong "yes" bias. MERLIM shows that inpainting an object out of an image exposes "hidden hallucinations," with roughly an 8% average precision drop on edited images. We reuse inpainting as our instrument for measuring judge blind spots. The hallucination-reducing preference methods SymMPO, P²-DPO, and the Bias-Driven Hallucination Sampling in Understanding Alignment in MLLMs all build preference pairs by construction rather than by judging. They are our strongest judge-free competitors. The last of these also constrains our design: it reports that preference-tuned models reduce hallucination partly by becoming short and conservative. We must therefore track length and helpfulness, not hallucination alone.
Research Program
The first component is a blind-spot map for each judge. On COCO images, we build response pairs that are identical except for one atomic visual claim in a known category. Each pair is shown to the judge twice: once with the original image, and once with a MERLIM-style inpainted image in which the claimed evidence has been removed or altered. A judge that sees correctly should flip its verdict when the evidence changes. The per-category blind-spot rate is the fraction of pairs where it does not flip. We compute the same map for the reference model acting as a yes/no answerer. The overlap between the two maps is our predictor.
The second component is the propagation study. The target model is LLaVA-1.5-7B, which SymMPO and P²-DPO also use, and it is tuned with standard DPO. We sample response pairs from this model on the RLHF-V prompts and TPO images those papers use. Four judges label the pairs:
- base Qwen2.5-VL-7B;
- Perception-Judge-Flex-7B;
- MR. Judge-7B;
- GPT-4o.
A human-verified subset provides ground-truth labels. Using an atomic-claim extractor like SymMPO's caption-anchored pipeline, we assign each pair to the claim categories on which its two responses differ. This yields $e_c$ per judge. For every judge we train a matched random-flip control at the same per-category error rates. We then measure $A_c$ on category-resolved benchmarks: POPE random, popular, and adversarial splits for existence; AMBER attribute and relation scores; HallusionBench; and MERLIM hidden-hallucination precision.
The third component is a blind-spot-aware labeling procedure that follows from the diagnosis. Some pairs have a deciding claim in a category where the judge's blind-spot rate exceeds a threshold. For those pairs we do not trust the judge's verdict. Instead we verify the claim directly by inpainting the claimed object or region and checking whether the target model's own grounding changes. If the claim cannot be verified, we drop the pair. Everything else keeps the judge's label. This procedure is cheap because it touches only the flagged fraction of pairs.
Baselines
The naive reference point is the untuned LLaVA-1.5-7B, together with the same model tuned by DPO on pairs labeled by base Qwen2.5-VL-7B with no auditing. This is the pipeline a practitioner would actually build. On HallusionBench, SymMPO reports about 41.7% accuracy for base LLaVA-1.5. We will re-run it under our evaluation code rather than rely on that figure.
The strongest judge-based competitor is DPO on labels from the best available judge. On VL-RewardBench, MR. Judge reports 75.5% overall accuracy and 83.2% on the hallucination sub-category for MR. Judge-7B, against 65.8% overall for GPT-4o. Mitigating Perceptual Judgment Bias reports 85.7% perceptual-error detection on its PPJD dataset, against 69.5% for base Qwen2.5-VL-7B. These judges are the strongest competitors because the obvious response to our question is "use a better judge." Every judge labels the identical pool of sampled pairs. The DPO hyperparameters are fixed across conditions, with the same β, learning rate, epochs, and a tuning budget of three seeds per condition. Blind-spot-aware labeling beats this baseline if, applied on top of the best judge, it lowers hallucination in high-overlap categories without lowering MM-Vet or shortening responses beyond the untuned model's length distribution.
The strongest judge-free competitors are SymMPO, which SymMPO reports at 87.7% AMBER F1 against 82.6% for standard DPO and 86.3% for mDPO, and Bias-Driven Hallucination Sampling, which Understanding Alignment in MLLMs reports at 88.75% on POPE. We will reproduce both on our backbone with the same number of preference pairs and the same training compute. These numbers come from different pipelines, so we treat them as targets for our reproductions, not as directly comparable scores.
Two ablations isolate the key ingredient. The random-flip control defines $A_c$ and separates structured from unstructured judge error. A matched-drop control removes the same number of pairs as blind-spot-aware labeling, but chooses them at random. It tests whether gains come from targeting blind spots or simply from discarding data.
Why not just…?
Why not just use the best judge and stop worrying? This is the most serious alternative. If MR. Judge or Perception-Judge labels yield $A_c \approx 1$ and hallucination rates comparable to judge-free methods, the amplification story is uninteresting for strong judges, and we will say so. Two points in the file argue against it. Mitigating Perceptual Judgment Bias reports that its judge still struggles with spatial and 3D reasoning. Both improved judges share the Qwen2.5-VL backbone family, so their residual errors may overlap with priors common to similar models. The propagation study measures $A_c$ separately for each judge, so it settles this question directly.
Why not just ensemble several judges or filter by judge confidence? Ensembling helps only if judge errors are independent. Our hypothesis is that they are correlated through shared co-occurrence priors of the kind POPE documents. The blind-spot maps measure cross-judge error correlation per category. We will also run a majority-vote-of-four condition and a confidence-filtered condition at the same pair count. If either matches blind-spot-aware labeling in high-overlap categories, the simpler method wins.
Why not skip the judge and construct pairs, as in SymMPO or Bias-Driven Hallucination Sampling? For existence errors this is a genuine threat. Constructed pairs avoid judge error entirely, and their reported POPE and AMBER numbers are strong. Their limitation, noted in the file, is coverage. SymMPO focuses on existence and struggles with counting, attributes, and relations. Masking-based sampling also produces the conservative length collapse that Understanding Alignment in MLLMs reports. Judges remain the only practical labelers for open-ended, multi-claim responses. Our comparison therefore reports results per category and per response length. We expect constructed pairs to win or tie on existence and to lose on relations and attributes. If they win everywhere, the practical conclusion is to avoid judges for hallucination-targeted tuning, and that finding is worth reporting in its own right.
Why not caption the image and use a text-only judge? MLLM-as-a-Judge found that judges given text descriptions sometimes do better. However, a captioner inherits the same language priors, and P²-DPO argues that supervision that ignores the image fails to correct the perceptual bottleneck. We include a caption-plus-text-judge labeler as a fifth judge in the propagation study. Its blind-spot map should overlap strongly with the reference model's, and we would expect it to show the largest $A_c$.
Decisive Checks
The central test is whether $A_c$ exceeds 1 and tracks overlap. Across judges and categories, we regress $A_c$ on the product of the judge's blind-spot rate and the reference model's blind-spot rate. The hypothesis fails if the correlation is near zero, or if $A_c \approx 1$ throughout. In that case judge errors act like ordinary label noise, and standard noise-robust training would suffice.
A second check swaps the reference model. We repeat the study with a target from a different backbone family than the judge. The overlap should shrink, and amplification should shrink with it. That result would isolate the shared prior as the mechanism rather than judge accuracy alone.
A third check guards against the helpfulness trap described in Understanding Alignment in MLLMs. Any hallucination reduction must hold at matched response length and must not come with an MM-Vet drop. Because MLLM-as-a-Judge reports verbosity bias, we also record whether judge-labeled tuning lengthens responses. Longer responses would give a second, length-mediated route by which judge bias reaches the model.
Finally, we release the per-judge blind-spot maps and the inpainted probe set. Anyone who uses a multimodal judge to build preference data can then audit it before training.
References
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
- Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge for Mathematical and Scientific Reasoning Evaluation
- MR. Judge: Multimodal Reasoner as a Judge
- The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Evaluating Object Hallucination in Large Vision-Language Models
- Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models
- Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
- P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
- Understanding Alignment in Multimodal LLMs: A Comprehensive Study