Grading LVLM Answers by Whether They Beat the Model's Own Blind Guess

Vision-language models (systems that pair an image encoder with a large language model so the model can talk about pictures) often describe objects that are not in the image. They tend to say "fork" because the table is a dining table, not because a fork is visible. Methods that train these models on preferences, where a judge says which of two answers is better, inherit a weakness from preference learning in general: a pairwise judgment only becomes a usable number if you choose the right thing to compare against. This proposal picks that comparison point on purpose. Each candidate answer is scored by how often an image-aware judge prefers it over answers the same model writes when it cannot see the image. That win rate is a single number per answer. It sits at 0.5 when the answer is no better than not looking, and it rises only when the model says something the image supports and its language habits would not have produced. Any standard reward-based learner can then optimize it.

This idea comes from bandit theory. The literature on "dueling-to-cardinal reductions" turns pairwise feedback into scalar rewards. The best-known version is the Borda score: an item's probability of beating an opponent drawn from a fixed pool. The pool is almost always uniform, and in text generation a uniform pool is degenerate, because any fluent answer beats a random string. The contribution here is a reference distribution built for multimodal hallucination: the model's image-blind completions. With that choice, the Borda score measures how much an answer depends on the image. We will test whether this reward reduces prior-driven hallucination more than pointwise judge scores, peer-sample win rates, and existing multimodal preference methods, at an equal judging budget.

Core Claim

We define the prior-referenced Borda score of a response $y$ to image $v$ and question $q$ as

$$B_0(v,q,y) = \mathbb{E}_{y' \sim \pi(\cdot \mid \varnothing, q)}\big[\mathbb{P}_J(y \succ y' \mid v, q)\big],$$

Here $\pi(\cdot \mid \varnothing, q)$ is the current model generating with its visual tokens zeroed out, and $\mathbb{P}_J$ is the probability that a judge who sees the image prefers $y$ to $y'$. Each duel is judged in both presentation orders and the results are averaged.

The claim is that $B_0$ isolates the part of answer quality that comes from looking. Suppose a hallucinated "fork" is also what the blind model says. Then on that detail the response ties with its opponent, and it can only win where the image overrides the prior. For this reason $B_0$ should target the failure that POPE's adversarial split (Li et al., which probes non-present objects that co-occur with present ones) and MERLIM's inpainting test (Villa et al., which erases objects and checks whether models still claim them) were built to expose. Win rates against peer samples, which share the image, and pointwise scores should not target it as directly. We do not claim that $B_0$ fixes counting or attribute binding. Those errors are not mainly prior-driven, and the program treats them as a control.

Relation to Prior Work

The reduction machinery comes from the bandit literature. Ailon et al. showed that reward-based bandit algorithms can be wrapped as black boxes to solve dueling problems. Borda-style analyses (Falahatgar et al., Jain et al., Singla et al.) treat the win rate against a random opponent as an unbiased cardinal reward. All of these assume a uniform opponent over a discrete set. Kernelized Offline Contextual Dueling Bandits lifted the Borda score to contexts and flagged that uniform opponents fail for generated text. Sample Efficient RLHF via Active Exploration scaled this to language models with dropout-based uncertainty and reported a gain of over 13% relative to uniform-sampling DPO, on text-only data. Both papers motivate this work. Neither proposes a principled in-distribution opponent, and neither touches images. We supply the opponent and the domain. Ailon et al. also identify left/right slot asymmetry as a source of bias, which is why every duel here is judged in both orders.

The closest multimodal work falls into three groups. V-DPO also contrasts image-conditioned and text-only behavior, but it does so through the model's own likelihood ratio, in a guidance-style term. A likelihood ratio rewards whatever the image makes more probable, including confident visual misreadings, and its data are static offline pairs. Our contrast passes through an image-aware judge, uses sampled blind answers rather than likelihoods, and is recomputed online. HDPO and SymMPO curate offline pairs. HDPO uses heuristically synthesized negatives; SymMPO uses a symmetric objective so that the Bradley-Terry normalizer cancels across two images. SymMPO's result constrains our design: every duel in $B_0$ shares one image, so no cross-image normalizer issue arises. OViP is online and uses a reference-based judge. It filters the model's own rollouts by score and has no notion of a reference opponent distribution. Its warning that hard penalization makes descriptions terse is the main risk we must measure.

Research Program

The base model is LLaVA-1.5-7B, chosen because most numbers in the file use it, with LLaVA-1.5-13B as a replication. Training uses about 20k Visual Genome images with detailed-description and short-question prompts, close in scale to HDPO's roughly 22k pairs. At each online round, for every image, the policy samples $N=4$ image-conditioned candidates and $M=4$ blind completions with visual tokens zeroed out. Each candidate is judged against every blind opponent in both orders, and the averaged wins give $\hat B_0$.

We use two judges so that judge noise can be separated from the reduction. The first is an oracle judge built from Visual Genome object, attribute and relation annotations. It prefers the response that makes fewer unsupported claims while covering more annotated content. Because the benchmarks below are not built from Visual Genome annotations, this does not train directly on the evaluation signal. The second is an open model-based judge (Qwen2.5-VL, following SymMPO's open pipeline), included because LVLM judges are known to hallucinate (Yan et al., Chen et al.). Both judges are instructed to weigh coverage alongside faithfulness. Otherwise a terse answer beats any long blind guess, and the score would reward saying nothing.

Because $\hat B_0$ is a scalar, the reduction allows any reward-based optimizer. We test two. The first is pair construction, which forms a DPO pair from the highest- and lowest-$B_0$ candidates. It keeps the comparison with SymMPO and HDPO honest. The second is group-normalized policy gradient with $B_0$ as the reward. A cheap active variant reallocates judging budget toward images whose bootstrap interval on $\hat B_0$ is widest. This replaces the expensive dropout passes of AE-Borda-DPO with resampling of judge outcomes, a stated cost limitation of that work. The variant is secondary and is reported only if the main effect holds.

On the analysis side, we check that $B_0$ is unsaturated. If its mean over candidates stays well inside $(0,1)$ during training, the uniform-opponent pathology is avoided. We also track how the blind opponent pool shifts as the policy improves, since the blind model co-evolves with the policy, and we test a frozen-blind-pool variant to see whether that drift matters.

Baselines

The do-nothing reference is the base LLaVA-1.5-7B. RITUAL reports its POPE accuracy at 84.13% (Woo et al.), and SymMPO reports its HallusionBench overall accuracy at about 41.7%. We rerun it under our own prompts rather than relying only on reported figures.

The strongest existing training-time method is SymMPO. On the 7B model it reports AMBER F1 of 87.7% (versus 86.3% for mDPO and 82.6% for standard DPO) and HallusionBench overall accuracy of 44.28%. It is the strongest competitor because it is a principled same-family preference method evaluated on the same base model. We will retrain it from its released pipeline on our image pool, with the same number of preference pairs and the same learning-rate sweep, so that differences come from the reward rather than the data. HDPO is a second strong competitor on long-form hallucination, reporting HalRate falling from 35.1% to 15.8% on LLaVA-v1.5-7B. OViP is the online competitor, reporting an HRI of 5.25 on LLaVA-1.5-13B after one epoch; we compare against it at 13B with equal judge calls. Beating these means exceeding SymMPO's retrained AMBER F1 and HallusionBench accuracy, and beating HDPO's HalRate, while keeping description coverage and MME perception within a pre-registered margin of the base model.

Three ablations isolate the key ingredient, the opponent distribution, while holding the judge, judging budget, images and optimizer fixed. The first scores candidates against peer image-conditioned samples, which is ordinary self-play win rate. The second uses uniform opponents drawn from other images' answers. It is expected to saturate near 1.0 and reproduce the pathology flagged by Mehta et al. The third mixes blind and peer opponents in equal parts. A fourth control drops the judge entirely and runs DPO with the image-conditioned answer as chosen and the blind answer as rejected. This tests whether the blind contrast alone does the work.

Why not just…?

Why not just ask the judge for a 1-to-10 score per answer? This is the most direct competitor. Pairwise judgments are generally more reliable than absolute ratings, which is the premise of the dueling literature, but that is a premise about human raters, not evidence about LVLM judges on this task. We therefore make it a head-to-head experiment. A pointwise scorer gets the same judge and the same number of judge calls, and we compare both the correlation of each reward with object-level hallucination on held-out Visual Genome images and downstream benchmark results. If pointwise scoring matches $B_0$, the reduction is not earning its place, and we would report that.

Why not just use the blind completion as the rejected answer in DPO, with no judge? This is cheap and close in spirit to V-DPO, so it is a real threat. The argument against it is MERLIM's finding that models often get objects right from scene context alone. A blind guess is frequently correct, and an image-conditioned answer is sometimes worse. Without a judge, the "blind is worse" label is wrong on exactly those cases. The judge-free control above settles this. If it matches $B_0$ on POPE adversarial and MERLIM, the contribution reduces to "use blind negatives," and the proposal should be rewritten around that simpler finding.

Why not just compare against the model's other image-conditioned samples? Peer win rates avoid the degenerate-opponent problem, and they are what most online methods implicitly use. Our hypothesis is that peers share the same prior-induced hallucinations, so their duels are decided by style and length rather than by grounding. The peer-opponent ablation tests this directly. The prediction is that peer and blind opponents perform similarly on random-split POPE, while blind opponents win on the adversarial split and the inpainting test. If no such split appears, the specific opponent choice is not what matters.

Why not just use test-time fixes like RITUAL or activation steering? RITUAL reports POPE accuracy of 88.87% on LLaVA-1.5 but runs extra image views at inference. SHARP cuts CHAIR sentence error from 52.8 to 34.8, but it requires choosing a layer and steering at inference time. Both methods are complementary rather than competing, since they leave the weights unchanged. We report them as reference points and test whether RITUAL stacked on our trained model adds further gains.

Proposed Experiments

The first check is cheap and is run before any training. On held-out Visual Genome images, we compute $\hat B_0$, peer win rate, uniform-opponent win rate and pointwise judge score for base-model answers, and correlate each with the oracle's per-answer unsupported-claim count. If $B_0$ is not the best-correlated reward for co-occurrence-type errors, we stop.

The main experiments train each method on LLaVA-1.5-7B and evaluate on POPE across all three splits, AMBER (generative F1 and discriminative), CHAIR on COCO captions together with object recall, HallusionBench, MERLIM's inpainted-object and prompt-variation tests, and MME perception and cognition. Response length and coverage are tracked throughout to catch the terseness failure OViP warns about.

The decisive result has three parts, all at equal judge calls. First, the blind-opponent reward must beat peer, uniform, pointwise and judge-free variants specifically on POPE adversarial and MERLIM's hidden-hallucination test. Second, it must match or exceed retrained SymMPO on AMBER F1 without a coverage drop. Third, it should show no special advantage on counting and attribute items. That last pattern would show that the reward does what it claims, removing reliance on language priors, rather than acting as a generic quality signal. The 13B replication and the active-sampling variant are then run against OViP at the same budget.