Consensus That Needs the Image: Prior-Contrastive Minimum Bayes Risk Decoding for Vision-Language Models

A common way to make a language model more reliable is to sample many answers and keep the one the samples agree on most. This is minimum Bayes risk (MBR) decoding: pick the candidate with the highest average similarity to the other samples. It works when the model's mistakes are scattered. Vision-language models (models that read an image and answer in text) fail in a different way. Their hallucinations are systematic. They name a refrigerator in a kitchen photo because kitchens usually contain refrigerators, and they answer "yes" to nearly any "is there an X?" question. When every sample makes the same prior-driven mistake, consensus decoding does not remove the hallucination. It certifies it.

We propose a simple fix. Draw a second, cheap pool of samples from the same model with the image removed, and score each claim by how much more often it appears when the model can see the image than when it cannot. The decoded answer is the candidate whose claims are both agreed upon and attributable to the image. Put in two sentences: ordinary consensus asks "what do my samples agree on?", while our rule asks "what do my samples agree on because of the image?" The blind pool measures directly the thing that fools ordinary consensus.

Core Claim

The research file names a known weakness of MBR. If misleading context skews the model's output distribution, the candidate pool fills with mutually consistent hallucinations and MBR selects a "hallucinated consensus". Learned similarity metrics add to this by rewarding generic, plausible sentences, a bias analyzed in Mitigating Metric Bias in Minimum Bayes Risk Decoding. The multimodal hallucination literature shows that this skew is exactly what vision-language models have. Evaluating Object Hallucination in Large Vision-Language Models shows that hallucinated objects track training-set co-occurrence and that "yes" rates reach nearly 99% in some settings. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that attention to image tokens decays as a response grows. Together these papers motivate the proposal: the error that consensus decoding cannot see is the error these models make most.

Our claim is that a sequence-level, claim-granular contrast between image-conditioned and image-ablated samples turns MBR from a method that amplifies priors into one that suppresses them. The contrast needs no training, no access to internal activations, and no auxiliary judge beyond a claim-matching function.

Concretely, let $H^{+} = \{y^{+}_1,\dots,y^{+}_N\}$ be samples given the image and question, and let $H^{-} = \{y^{-}_1,\dots,y^{-}_M\}$ be samples given the question alone. Decompose each candidate $y$ into atomic claims $c$, such as "there is a cup" or "the cup is red". Let $s^{+}(c)$ and $s^{-}(c)$ be the fraction of samples in each pool that entail $c$, judged by a claim-matching function. Each claim then gets an image-attribution score, and the decoded output maximizes claim-level agreement weighted by that score:

$$ \hat{y} = \arg\max_{y \in H^{+}} \frac{1}{|C(y)|}\sum_{c \in C(y)} s^{+}(c)\,\Big[\log \frac{s^{+}(c)+\epsilon}{s^{-}(c)+\epsilon}\Big]_{\ge -\tau} $$

Here $C(y)$ is the claim set of $y$, $\epsilon$ is a smoothing constant, and $\tau \ge 0$ caps how strongly a prior-explained claim is penalized. The log ratio estimates the pointwise mutual information between the claim and the image. It is large when seeing the image makes the claim more likely and near zero or negative when the claim is what the model would say anyway.

With the attribution term replaced by a constant, the rule reduces to ordinary claim-level MBR. That reduced version is our central ablation. A variant also deletes individual claims whose attribution falls below a threshold. This targets the "hallucination in the fluff" pattern documented by Evaluating the Quality of Hallucination Benchmarks for Large Vision-Language Models, where the main answer is right but the supporting explanation invents details.

Relation to Prior Work

The closest bridge in the file is CoCoA, which frames uncertainty as minimum achievable Bayes risk by combining token confidence with sample consistency. It is text-only and treats consistency as evidence of correctness. Our starting premise is that in the multimodal setting consistency is partly evidence of prior. CycleVLA applies MBR to robot action trajectories by choosing the trajectory nearest the centroid of 8 samples. It uses no contrast and never touches language.

Noisy-Channel Minimum Bayes Risk Decoding is conceptually closest in spirit, because it also rewards outputs that explain the input. We treat it as a direct competitor, discussed below. Our contrast is estimated from the model's own blind samples rather than from a separately trained reverse model, and it acts per claim rather than per sequence.

On the hallucination side, several methods contrast grounded and ungrounded behavior token by token during decoding. RITUAL combines predictions from the original and a randomly transformed image. The contrastive contextual decoding in HalTrapper works in a similar way. SHARP adds steering vectors, derived offline from contrastive data, at intermediate layers.

All of these intervene one token at a time inside a single decoding pass. That matters because of textual inertia, documented in Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts: once a wrong claim is emitted, later tokens condition on it, and models self-correct in under 10% of cases. Our method compares whole completed responses and never commits to a claim before seeing the full distribution over alternatives. This borrows the lesson from When Splitting Makes Stronger that isolating reasoning paths from one another limits error propagation. We apply that lesson without requiring the task to decompose.

In-context learning plays a secondary but testable role. In few-shot multimodal prompting, the demonstration images and answers become part of the conditioning context. They may inject their own priors, for example copying objects from a demonstration into the answer. If the blind pool keeps the demonstrations but drops only the query image, the contrast should absorb demonstration-induced priors along with language priors. This is a hypothesis the file does not test.

Research Program

The method is built and analyzed in three stages.

The first stage builds the decoder. It needs three components: a sampler for both pools, a claim extractor, and a claim-matching function. For short-answer tasks such as POPE, a claim is simply the answer. The method then reduces to calibrating the grounded answer distribution against the blind answer distribution. This is a clean test of whether the contrast removes yes-bias. For captions and explanations, the same vision-language model extracts claims with a fixed prompt, and an off-the-shelf entailment cross-encoder does the matching.

Because metric bias is a documented failure of MBR, we audit the matcher separately. On a few hundred hand-labeled claim pairs from HQH, we measure matcher accuracy on fine-grained contradictions such as color and count.

The second stage is a mechanism analysis. We measure how often plain MBR selects a response containing a CHAIR-flagged object that also appears in most blind samples. The rate at which plain MBR certifies prior-driven hallucinations is the quantity our method is designed to reduce. If that rate is already low, the premise is wrong and we will say so.

The third stage extends the idea to multi-step multimodal reasoning. We apply the attribution score to intermediate rationale steps on MMVP and MathVista. On MathVista the question often cannot be understood without the image, so the blind pool is degenerate. Thinking Before Looking reports only a modest ~3.15% gain there for a related reason. We expect our method to help least on this benchmark and report it as a boundary case.

Baselines

The naive reference point is greedy decoding and single-sample decoding of the same model: LLaVA-1.5-7B, LLaVA-NeXT, and Qwen2.5-VL-7B. RITUAL reports base LLaVA-1.5 POPE accuracy of 84.13%, and SHARP reports base LLaVA-v1.5 $CHAIR_S$ of 52.8. We will reproduce both figures under our prompts before comparing anything.

The most important competitor is plain claim-level MBR over image-conditioned samples, which is our rule with the attribution term removed. It is run with $N+M$ grounded samples, so it receives at least as many forward passes as our method. Blind samples are cheaper because they carry no image tokens, so this gives the baseline a slight compute advantage. Beating it at matched sample count is the core evidence that the contrast, not extra sampling, does the work. We also run standard sequence-level MBR with an embedding utility, to separate the effect of claim granularity from the effect of the contrast.

The strongest existing training-free hallucination methods are RITUAL and SHARP. RITUAL reports LLaVA-1.5 POPE accuracy rising from 84.13% to 88.87%. SHARP reports LLaVA-v1.5 $CHAIR_S$ falling from 52.8 to 34.8, and LLaVA-Next POPE Adversarial accuracy rising from 79.5% to 82.8%. Summary-Guided Decoding reports a 16.5% relative improvement in $CHAIR_S$ on LLaVA 1.5.

We rerun RITUAL and SHARP on the same models, images, and prompts, using their published hyperparameters and our own sweep capped at the tuning budget we give our method: one validation split and the same number of configurations. Beating them means lower $CHAIR_S$ than SHARP at equal or higher object recall, and higher POPE Adversarial accuracy than both.

For Noisy-Channel MBR, we implement its reranking using the model's own likelihood of the question given the candidate and the image. This is the closest principled alternative for "explain the input". We found no reported multimodal result for it on Lacuna.

Finally, we run an ablation on the source of the negative pool. The blind pool is compared with a pool drawn from a RITUAL-style transformed image. This tests whether removing the image, rather than merely perturbing it, is the active ingredient.

Why not just…?

Why not just use self-consistency or ordinary MBR with more samples? More samples from a skewed distribution estimate the skewed mode more precisely. The co-occurrence structure reported for POPE's Adversarial split, where MiniGPT-4 F1 falls from 78.86 to 71.37 per Evaluating Object Hallucination in Large Vision-Language Models, is exactly the regime where agreement and correctness diverge. The matched-compute plain-MBR baseline settles this question directly. If plain MBR matches us on the Adversarial split, the contrast adds nothing and the proposal fails.

Why not just use token-level contrastive decoding like RITUAL or HalTrapper? This is a real threat on POPE. There the answer is a single token, so token-level and sequence-level contrasts nearly coincide, and RITUAL may match us at lower cost. Our argument is confined to long outputs. A token-level method commits to each word as it goes, and textual inertia means a single early mistake is then defended rather than corrected, per Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts. The decisive comparison is therefore on CHAIR and on HQH extra-hallucination claims, broken down by response length. We predict our advantage grows with length. If it does not, we will report the method as a POPE-level calibration trick only.

Why not just steer activations with SHARP, which costs one pass? SHARP is cheaper and already strong. However, its vectors come from an offline contrastive dataset labeled by an external judge, and they are fixed across inputs. Our contrast is computed per input from the model's own behavior, so it should transfer to images and prompt styles unlike SHARP's training data. We test this on HQH categories such as Text and Environment, which HaloQuest-style data are unlikely to cover. The two methods are also compatible: we run SHARP-steered sampling inside our decoder. If the combination beats both, the methods address different errors.

Why not just rerank by image-text similarity with CLIP? CLIP-style scores reward mentions of salient objects. They are known to be weak on counts, attributes, and relations, which are exactly the HQH categories where GPT-4o's error rate is highest. We include CLIP reranking of the same pool as a cheap control.

Proposed Experiments

The main table covers POPE in all three splits, CHAIR on 500 COCO images, HQH across its eight categories with main-answer and extra-claim hallucination reported separately, AMBER, and MME perception. Each is run on three open models with $N = M = 8$ samples, the pool size CycleVLA uses. We report hallucination rate, object recall, and wall-clock cost per example.

The decisive checks are four. First, at matched forward passes, our method must beat plain claim-level MBR on POPE Adversarial and on $CHAIR_S$. Second, on long responses it must beat SHARP's reported $CHAIR_S$ of 34.8 on LLaVA-v1.5 without lowering recall. Third, the fraction of selected responses containing prior-shared hallucinations must drop relative to plain MBR, confirming the mechanism rather than just the headline metric. Fourth, recall on objects that are both present and highly predictable, such as a table in a dining room, must not fall by more than a few points. Too much recall loss would mean the contrast is punishing correct but expected claims, and the cap $\tau$ exists to prevent exactly that.

A secondary experiment tests the in-context hypothesis on LLaVA-NeXT-Interleave with two-shot image demonstrations. We measure how often objects from the demonstrations appear in answers about the query image, first when the blind pool keeps the demonstrations and then when it drops them.

A final experiment applies step-level attribution to MMVP rationales and to MathVista, where we expect gains on MMVP and little or nothing on MathVista. Both outcomes will be reported.