Image-Swap Judging: Selecting Hallucination-Alignment Data When the Judge Also Hallucinates

Training vision-language models to stop describing things that are not in the image now relies on automated pipelines. A strong multimodal model acts as a judge: it scores candidate responses, and the best-scoring ones become preference data for fine-tuning. These judges share the same weaknesses as the models they grade. They reward length, they give most answers a 4 or 5, and they fill gaps in what they see with what a scene usually contains. When a selection method maximizes judge score, possibly with a diversity term added, it learns the judge's blind spots. This proposal adds one inexpensive test. Each response is scored twice, once against its true image and once against a visually similar but different image, for example another kitchen. The score is trusted only to the extent that it depends on the true image. That difference becomes the "quality" input to a quality-diversity selection procedure for hallucination-focused preference optimization.

The central bet is simple. A judge's score that stays the same when the image is swapped for a similar scene reflects the judge's language prior, not visual evidence. Hallucination-reduction data selected on that basis teaches the model to write plausible, scene-typical text, which is the failure we want to remove. We also predict that adding diversity to selection makes things worse under an unreliable judge, because the unusual examples a diversity term pulls in are exactly where judges fail most. If the prediction holds, the quality term and the diversity term cannot be designed independently. The quality score has to be made trustworthy before diversity is added.

Core Claim

Let $J(y \mid v, p)$ be a judge's score for response $y$ to prompt $p$ about image $v$. Let $v'$ be the nearest different image to $v$ in an image-embedding space (CLIP), chosen so that $v'$ shares scene type and typical co-occurring objects. We define the grounded quality of a response as

$$g(y; v, p) = J(y \mid v, p) - J(y \mid v', p),$$

the part of the judge's approval that disappears when the actual visual evidence changes but the scene prior stays roughly the same. A preference pair $(y^+, y^-)$ is kept only if its grounded margin $g(y^+) - g(y^-)$ is positive. Pairs are then selected with a facility-location diversity objective over joint image-prompt embeddings, with $g$ in place of the raw judge score as the quality term.

Three pieces of prior work supply the ingredients. V-DPO (V-DPO) compares a model's image-conditioned outputs against its text-only outputs in order to train the model; we apply the same contrast to the judge, as an audit. POPE (Evaluating Object Hallucination in Large Vision-Language Models) showed that hallucinations follow co-occurrence statistics, such as a knife assumed in a kitchen. That result is why the swap uses a similar scene rather than a blank image: a blank image removes the scene prior as well as the evidence, so it would not isolate the prior. QDIT (Data Diversity Matters for Robust Instruction Tuning) supplies the selection objective. It combines judge quality with facility-location diversity, but it treats the judge's scalar score as correct and is text-only.

Why the gap is real

The motivating evidence comes from the judge-bias literature. MLLM-as-a-Judge benchmarks multimodal models as graders and reports three findings: non-GPT-4V judges cluster at scores of 4–5, length inflates scores, and judges do better from text descriptions of images than from the images themselves. That last finding means visual grounding is the judge's weakest channel. Justice or Prejudice? maps twelve text-judge biases and finds verbosity bias in nearly every model. The Silent Judge shows that judges change verdicts on irrelevant cues yet mention those cues in 0% of their justifications. We therefore measure judge reliance on the image through the judge's behavior under controlled input changes, never by reading its rationale.

The work this proposal targets is the recent group of hallucination-alignment pipelines that let a frontier judge define the data. HD-DPO (Adaptive Hallucination Alleviation) filters the 23k-sample detailed-description set to 5k "unstable" images and weights the DPO loss by GPT-4V severity scores. It applies no diversity regularization, lists reliance on GPT-4V as a limitation, and reports a slight MMBench dip. MISP-DPO (Importance Sampling for Multi-Negative Multimodal DPO) diversifies the negative responses for a fixed prompt pool but leaves the image-prompt distribution unselected, and it notes that it depends on GPT-4 for evaluation. These papers motivate the proposal: they optimize hardness or negative diversity while taking the judge as correct.

Two results constrain the design. The "CoT paradox" in MLLM-as-a-Judge, where analyze-then-judge prompting makes the judge hallucinate premises and grade on them, rules out fixing the judge by asking it to describe the image first; we score directly. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that attention drifts from image tokens to already-generated text as responses grow. A length-biased judge therefore selects the responses most likely to contain late-stage fabrications, which is the mechanism our length diagnostics test.

Research Program

The work has three parts, each building on the previous one.

The first part audits the judge. On the MLLM-as-a-Judge benchmark, which has human scores and pairwise labels, we compute raw $J$, $J$ with length regressed out, and $g$ for several open judges (LLaVA-1.6-class and larger open models) and one frontier judge. We then ask which quantity agrees better with humans. We also bin examples by how far their image-prompt embedding lies from dense regions of the pool, and measure judge-human agreement per bin. This tests the tail-unreliability hypothesis directly. If agreement does not fall in the tail, the second half of our argument fails, and we will report that.

The second part builds the selection pipeline. The pool is on-policy LLaVA-1.5-7B responses sampled for RLHF-V prompts and for the 23k detail-description set used by HD-DPO. Pairs are formed by judge score, filtered by grounded margin, and selected to a 5k budget, the same budget as HD-DPO, using lazy-greedy facility location with the diversity weight $\alpha$ tuned on a held-out split. Swapped images are retrieved once with CLIP. The only added cost is a second judge call per response.

The third part trains and evaluates. Every selected set trains the same DPO recipe on LLaVA-1.5-7B. Evaluation deliberately favors metrics that do not use an MLLM judge, for the reasons this proposal gives.

Baselines

The naive reference points are the full 23k pool trained with the same DPO recipe, and a uniform random 5k subset. HD-DPO already argues that random subsets of standard instruction data are too easy to trigger hallucinations, so random selection should be beatable. If our method fails to beat it, the whole selection story is in doubt. The heuristic baseline "keep pairs whose chosen response is longest" is also included, because it is the pure form of verbosity bias.

The strongest existing selection method is HD-DPO, run on the same 23k pool with its instability criterion and severity weighting. Because reimplementations vary, we also re-score its severity labels with the same open judge used for our method, so the comparison isolates the selection rule rather than judge access. HD-DPO reports a CHAIR hallucinated-object rate reduced from 47.3% to 4.1%, a 20.9% relative improvement on MMHal-Bench, and an AMBER discrimination aggregate of 90.6 (Adaptive Hallucination Alleviation). No HallusionBench number for HD-DPO was found on Lacuna. MISP-DPO reports a 30.09% average improvement over LLaVA-v1.5-7B across five benchmarks (Importance Sampling for Multi-Negative Multimodal DPO). Since it varies negatives rather than prompts, we treat it as a training-objective competitor and as a possible complement: we test our selected prompts under its multi-negative loss.

The key ablations share one selection rule and vary only the quality score. The rule is QDIT-style facility location, which our method uses unchanged. Selection is run with raw judge scores (QDIT applied naively to images), with length-residualized scores, with blank-image differencing, and with nearest-neighbor-swap differencing ($g$). Separately, $g$-top-K without diversity isolates the diversity term. All arms share the pool, the budget, the DPO hyperparameters, three seeds, and an equal tuning budget over $\alpha$ where relevant.

Beating the strongest baseline means three things together, all within HD-DPO's selection budget. The method must match or lower CHAIR and raise POPE-adversarial F1 relative to HD-DPO. It must not show HD-DPO's MMBench dip. And its gains must hold when response length is controlled, meaning they cannot be explained by shorter outputs alone.

Why not just…?

Why not just regress response length out of the judge scores? This is the most serious threat, since verbosity is the best-documented bias (Justice or Prejudice?; MLLM-as-a-Judge). Length correction, however, cannot detect a concise response that names a scene-typical object that is absent, and that is exactly POPE's co-occurrence failure. The length-residualized arm is a full baseline. If it matches $g$ on POPE-adversarial and HallusionBench, we will conclude that image-swap differencing adds nothing beyond length control and say so.

Why not just use a stronger judge, or an ensemble of judges? Frontier judges are the ones the target pipelines already use. Even GPT-4V reaches only 0.490 Pearson agreement with human scores on the MLLM-as-a-Judge benchmark, and it shows self-preference. On HallusionBench it falls from roughly 75–80% on original images to roughly 38% on edited ones (HallusionBench), which is the same prior-over-evidence failure our swap test targets. Averaging judges averages shared priors. Our audit includes a frontier judge and an ensemble of open judges, both with and without differencing, which shows whether scale alone closes the gap.

Why not just use reference-based metrics like CHAIR to pick data? CHAIR-style scoring needs object annotations and covers only object existence. It says nothing about attributes, relations, or the open prompts in RLHF-V. A COCO-subset arm that selects by CHAIR tests this, and we expect it to help on object metrics while doing worse on AMBER relations and MMHal-Bench.

Why not just use RLHF-V's human corrections? RLHF-V cut hallucination by 34.8% with 1,400 correctional samples. Human data is the gold standard, but it is small and costly, and the point here is to scale automated selection. We include RLHF-V human pairs as an upper-reference arm and report how much of the gap between the raw judge and humans the grounded score recovers.

Proposed Experiments

The decisive checks follow the program. On the MLLM-as-a-Judge benchmark, $g$ must raise pairwise human agreement over raw scores for open judges; GPT-4V's reported reference point is about 79% (MLLM-as-a-Judge). Agreement should also fall with distance into the tail more steeply for raw scores than for $g$. In training, we evaluate on POPE across its random, popular, and adversarial splits, on HallusionBench question-pair accuracy, AMBER, and CHAIR, all of which avoid an MLLM judge. MMHal-Bench is reported under two different judges alongside mean response length. MMBench and SEED-Bench track general capability.

We plot DPO outcomes against the fraction of selected chosen responses that are longer than their rejected counterparts, and against their $g$ margin. This tests whether raw-judge selection pushes the model toward long outputs of the kind the contextual-trap paper links to fabrication. The central interaction test is a two-by-two design crossing judge score type (raw or grounded) with diversity (on or off). Our hypothesis predicts that diversity helps under $g$ but helps less or hurts under raw scores. That sign flip, if it appears, is the result that most clearly ties the three directions together. If no interaction appears, the proposal reduces to a better quality score, which is still useful but narrower.