Vision-language models describe objects, attributes and relations that are not in the image. A strong recent remedy is on-policy preference training: the model writes several answers for each image, a scorer decides which is more faithful, and the model is trained with Direct Preference Optimization (DPO, a method that raises the probability of the preferred answer relative to the rejected one) on the resulting "better/worse" pairs. That scorer is the weak link. Cheap self-contained scores miss hallucinations that the image itself encourages. Asking a strong judge to compare every pair of answers is more reliable, but the cost grows quadratically with the number of answers, so pipelines fall back to few answers or to coarse one-at-a-time grades.
We propose to treat each image's candidate answers as a small ranking problem and solve it with the adaptive pairwise methods from the active-ranking literature. A cheap internal score gives an initial order. An expensive pairwise judge is then called only where that order is uncertain, and a noise-tolerant sorting routine arranges the calls so that roughly $K\log K$ comparisons rank $K$ answers instead of $K^2/2$. The savings buy more candidates per image at a fixed judging budget, and more candidates means rarer, genuinely faithful answers get found. The proposal also asks a question the file shows is unresolved: once the answers are ranked, should the training pair be the two extremes, two neighbours, or something in between?
Core Claim
At a fixed budget of judge calls, choosing which answer pairs to judge actively beats both pointwise scoring and exhaustive pairwise judging over fewer candidates as a way to build on-policy anti-hallucination preference data. The method sorts a larger candidate pool with a pairwise visual-grounding judge, seeded and gated by a cheap internal score. The gain should be concentrated where the cheap score is blind: hallucinations driven by the scene itself rather than by the language model's text-only prior.
Motivation and Positioning
The work this builds on most directly is IRIS. IRIS samples $K$ answers on-policy and scores each with Rectified Visual Guidance (RVG), a log-probability contrast that penalizes answers which are likely without the image but unlikely with it. It then pairs the highest- and lowest-scoring answers for DPO, and reports Object-HalBench CHAIRs falling from 54.70 to 8.66 on LLaVA-1.5-7B. This proposal keeps IRIS's on-policy loop and uses RVG as the cheap signal. It departs from IRIS in two ways: it does not trust RVG alone, and it replaces fixed max-min pairing with actively chosen pairwise judgments.
The reason for not trusting RVG comes from two papers that motivate the gap. MERLIM shows "hidden hallucinations": when an object is inpainted away, models keep asserting it because the rest of the scene implies it. POPE shows that co-occurrence-based negatives are hardest. In these cases the image, not the absent image, raises the hallucination's probability. RVG's contrast against the no-image condition is therefore structurally weak exactly there.
Two other papers define the competing pair-construction strategies. HD-DPO curates unstable images offline and weights pairs by GPT-4V severity grades. SymMPO fixes the objective for cross-image pairs but trains on a static pair set. Neither adaptively decides which responses to compare.
The ranking side supplies the method. Just Sort It! shows that Quicksort-style sorting is an effective active learner under noisy Bradley-Terry comparisons. Its errors stay local rather than cascading, and it cut rank displacement by 14–23% versus random pairs on GIFGIF. Anytime Sorting Algorithms gives Multizip-style routines that produce a usable complete order when the budget is cut early, which matters because judge budgets per image are small. Active Evaluation supplies the gating idea: it queries the expensive judge only when a cheap metric is uncertain, and eliminates clearly inferior candidates early, cutting human effort by up to 89% across 13 generation tasks. Here RVG plays the role of that cheap metric and a vision-language judge plays the role of the human.
K-Sort Eval is the closest existing combination of adaptive matchmaking and vision-language judges. It ranks image and video generators with uncertainty- and diversity-driven matchups and Bayesian noise correction. It evaluates generators, however. It does not rank one model's own answers for its own training, and it never feeds the ranking back into hallucination mitigation.
Two constraining results shape the design. First, MLLM-as-a-Judge reports that vision-language judges agree with humans about 79% of the time in the pairwise setting but reach only 0.490 Pearson similarity when scoring one answer at a time. That gap is the reason to prefer pairwise judgments. The same paper warns that judges show verbosity bias and hallucinate errors, especially with chain-of-thought, so the judge must be run with position swapping and without free-form reasoning. Second, there is a real tension about which pair to train on. GapPO finds that small-margin pairs inject label noise. ActiveDPO finds that choosing training pairs by smallest predicted margin can plateau below random. Adaptivity is therefore spent on deciding what to judge. What to train on is treated as a separate, empirically swept choice.
Research Program
For each training image, the policy (LLaVA-1.5-7B, to match IRIS, HD-DPO and SymMPO) samples $K$ answers to a description or question prompt. Each answer gets an RVG score, which costs one extra forward pass without the image. The answers are initially ordered by RVG.
A fixed pairwise judge then refines the order. The judge is an open-weight vision-language model larger than the policy, prompted to say which of two answers makes fewer claims unsupported by the image. Each query is asked in both orders, and inconsistent verdicts count as ties.
Judge calls are placed by a noise-tolerant Quicksort seeded with RVG-chosen pivots. A comparison is skipped whenever the RVG gap between two answers exceeds a threshold calibrated on held-out data where RVG and the judge agree. Candidates confirmed as worse than the current best by several margins are eliminated early, as in UCB elimination. An anytime ordering is maintained so that a hard cap of $B$ judge calls per image always yields a full ranking.
The budget arithmetic is what makes larger pools affordable. Exhaustive comparison needs $K(K-1)/2$ calls, while randomized Quicksort needs about $2K\ln K$ in expectation, the standard average-case count. At $B=20$, exhaustive judging caps $K$ at 6, whereas sorting reaches roughly $K=8$ before gating, and more once RVG removes the easy comparisons.
From the final ranking, training pairs are selected by rank gap $g$, the number of positions separating the chosen and rejected answers. The sweep runs from adjacent ($g=1$) to extremes ($g=K-1$), and also includes a mixture that takes the top answer against one mid-ranked and one bottom-ranked answer. Training follows IRIS's iterative rounds with standard DPO, so any gain can be attributed to pair construction rather than to the loss.
Two intrinsic measurements run alongside training. First, on held-out COCO images, each description's hallucinated-object count is computed from ground-truth annotations with the CHAIR procedure. Kendall rank agreement between the sorted order and this reference measures ranking quality per judge call. Second, every pair where the judge overrides RVG is logged. These are compared against MERLIM-style inpainted images to test whether overrides concentrate on scene-driven hallucinations.
Baselines
The naive reference point is the untrained LLaVA-1.5-7B, together with one round of DPO on randomly paired on-policy answers judged once each. The second reference shows what on-policy data buys before any selection is applied.
The strongest existing method is IRIS: RVG scoring of $K$ answers with max-min pairing and no external judge. Its reported result is Object-HalBench CHAIRs of 8.66 on LLaVA-1.5-7B, down from 54.70 (IRIS). IRIS will be re-run with the same prompts, images, number of rounds and DPO hyperparameters as the proposal. It will also be given extra sampling compute equal to the proposal's judge compute, spent on a larger $K$, so that the comparison is matched on total FLOPs. Beating it means lower CHAIRs and CHAIRi on Object-HalBench, higher AMBER F1, and a higher MMHal-Bench score across three seeds, with the largest margin on POPE-adversarial and inpainted-image probes.
Two published pair-construction methods serve as external reference points. SymMPO reports AMBER F1 of 87.7 against 82.6 for standard DPO (SymMPO). HD-DPO reports CHAIR hallucination falling from 47.3% to 4.1% (HD-DPO). These are quoted rather than re-run, because they depend on GPT-4V annotation and fixed curated data.
Three budget-matched ablations isolate the key ingredient. The pointwise-judge ablation spends the same $B$ calls grading answers one at a time and pairs the max with the min. The exhaustive-pairwise ablation compares all pairs of the largest $K$ that fits in $B$. The ungated-sort ablation removes RVG gating and seeding. The same judge, images and training recipe are used throughout.
ActiveDPO's gradient-based selection is included only on a subset of the data, since it requires backward passes over every candidate pair. For a test-time reference, RITUAL reports 88.87% POPE accuracy on LLaVA-1.5 (RITUAL), and it can be stacked on top of the trained model.
Why not just…?
Why not just use IRIS's RVG score and skip the judge entirely? This is the most serious threat, because RVG is nearly free and IRIS's reported gains are large. The argument against it is specific: RVG measures how much the image changes an answer's probability, and MERLIM's hidden hallucinations are cases where the image itself raises the probability of a wrong object. The experiment settles it in two ways. The compute-matched IRIS baseline is one. The override analysis is the other: if judge-overridden pairs do not concentrate on inpainted or co-occurrence cases, and removing them from training costs nothing, RVG alone suffices and the proposal should be reduced to "IRIS with larger $K$."
Why not just have the judge grade each answer once? Pointwise grading costs only $K$ calls. However, MLLM-as-a-Judge reports much weaker human agreement for scoring (0.490 Pearson) than for pairwise comparison (about 79%). IRIS also notes that discrete grades produce ties that hide fine distinctions. These are different metrics, so the claim is tested directly by the pointwise-judge ablation and by Kendall agreement with CHAIR-derived reference counts.
Why not just compare every pair over a small candidate set? Exhaustive comparison is simple and has no sorting noise. Its cost is that at $B=20$ it caps $K$ at 6, and faithful answers are rare for hard images. The exhaustive-pairwise ablation shows directly whether the larger pool that sorting affords actually yields better top answers, measured by the reference hallucination count of the chosen answer.
Why not just pick the pairs the model is least certain about? ActiveDPO reports that margin-based selection plateaus and can fall below random. That is why uncertainty is used here only to allocate judge calls, while training pairs are chosen by rank gap. The rank-gap sweep tests whether GapPO's warning about small-margin pairs carries over to multimodal DPO.
Proposed Experiments and Decisive Checks
Training uses the image pool of LLaVA-Instruct detail-23k, as in HD-DPO. Evaluation covers Object-HalBench and CHAIR on COCO, AMBER (both generative and discriminative), MMHal-Bench, HallusionBench, and POPE under random, popular and adversarial negatives. MERLIM's inpainted images serve as a hidden-hallucination probe. MME and MMBench are included to catch the reasoning loss HD-DPO observed.
The judge's own reliability is checked once on the pairwise split of the MLLM-as-a-Judge benchmark before any training. A judge that falls well below the reported roughly 79% agreement would be replaced before the main runs.
The first decisive check is a curve of downstream CHAIRs and AMBER F1 against judge calls per image. The claim holds only if sorted, gated pools dominate the pointwise and exhaustive curves at every budget.
The second check is the rank-gap sweep. The prediction is that mid-gap or mixed pairs beat both extremes, reconciling GapPO's noise argument with active ranking's information argument. If extremes win, the ranking still helps, but only by finding a better top answer.
The third check is the override analysis described above. Its result decides whether the pairwise judge is catching a failure mode RVG cannot, which is the proposal's reason for existing.
References
- IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination
- Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models
- Evaluating Object Hallucination in Large Vision-Language Models
- Adaptive Hallucination Alleviation in Multimodal Large Language Models: From Strategic Data Selection to Severity-Guided Training
- Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
- Just Sort It! A Simple and Effective Approach to Active Preference Learning
- Anytime Sorting Algorithms
- Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons
- K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
- GapPO: Gradient-Adaptive Pairwise Preference Optimization
- ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
- RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs