Vision-language models describe images by predicting one word at a time. They are fine-tuned with the standard likelihood objective, which rewards putting some probability on every word that appeared in any training caption. A known side effect is that the model learns safe, generic continuations: after "a dining table with plates," it will happily add "and a fork" whether or not a fork is in the picture. Current fixes for this kind of hallucination mostly buy precision with expensive synthetic preference data. One pipeline chains GPT-4 prompts, several Stable Diffusion renders per prompt, an image scorer, five separate vision-language models and a text reward model. Others use ChatGPT-guided image inpainting.
We propose that much of what these pipelines buy can be obtained from the model itself, at roughly the cost of one extra forward pass per caption. The model's own image-removed predictions reveal which words it would have said anyway. The method samples captions from the model, flags content words whose probability barely changes when the image is hidden, and adds a word-level penalty on only those words during fine-tuning. The image pool is the one the model was already trained on. The fix lives in the weights, so decoding costs nothing extra at deployment.
The contribution has two parts. The first is the method. The second is a cost-matched evaluation that reports hallucination reduction per GPU-hour spent building training data, a quantity the existing literature does not measure.
Core Claim
The likelihood objective minimizes forward KL divergence, a measure of mismatch between the data distribution and the model that punishes the model for assigning near-zero probability anywhere the data has support. It therefore favors recall over precision. On the Efficacy of Sampling Adapters makes this precise and shows that truncation decoding works by shifting the effective objective toward precision. A Diversity-Promoting Objective Function for Neural Conversation Models shows that subtracting the unconditional probability of a response removes the generic replies that likelihood training produces.
In vision-language models the analogous "unconditional" distribution is simply the model with its visual input removed. That distribution is exactly the language prior blamed for object hallucination by Evaluating Object Hallucination in Large Vision-Language Models and by V-DPO.
Our claim is that a precision-oriented, word-level correction gated by this self-computed visual signal will close most of the hallucination gap that external synthetic preference data closes, at one to two orders of magnitude less data-construction compute. The claim is falsifiable at two points. The visual signal must actually locate hallucinated words, which we check before any training. And the gains must survive the generative benchmarks on which The Mirage of Performance Gains showed that decoding-time contrastive methods collapse.
Method
For an image $v$, prompt $x$, and a caption $y$ sampled from a frozen snapshot of the model, define each word's visual grounding score as
$$s_t = \log p_\theta(y_t \mid v, x, y_{<t}) - \log p_\theta(y_t \mid \varnothing, x, y_{<t}),$$
where $\varnothing$ denotes the visual tokens zeroed out, as in V-DPO's text-only branch. A score near zero or below means the image did not make that word more likely.
Following Summary-Guided Decoding, which restricts its intervention to image-related parts of speech in order to protect fluency, we restrict attention to the set $C$ of content words: nouns, visual adjectives and spatial terms. Function words are left alone. Fine-tuning then minimizes
$$\mathcal{L} = \mathcal{L}_{\text{MLE}} \;-\; \lambda \sum_{t \in C} \mathbb{1}[s_t < \tau]\,\log\big(1 - p_\theta(y_t \mid v, x, y_{<t})\big).$$
The terms are as follows. $\mathcal{L}_{\text{MLE}}$ is the ordinary caption likelihood on the existing instruction data. The second term is an unlikelihood penalty: it pushes probability away from flagged words in the model's own samples. $\lambda$ is its weight and $\tau$ is the flagging threshold. The scores $s_t$ are computed once per refresh round from the snapshot, with no gradient through them.
The penalty is a mode-seeking, precision-oriented term in the spirit of MixCE, which mixes in reverse cross-entropy. The difference is that MixCE's reverse term rewards whatever the model already finds likely, and that is the language prior. Our term is gated by visual evidence, so it targets prior-driven mass specifically.
The total data-construction cost is the following. Sample $K$ captions per image from a 7B model. Run one extra image-removed forward pass per caption. Run a part-of-speech tagger. Nothing else is needed: no external generator, no diffusion model, no reward model, no new annotations.
Relation to Prior Work
Three lines of work motivate or constrain the design.
The synthetic-preference methods are the cost baseline we aim to undercut. Synth-Align builds preference pairs through the multi-model cascade described above. V-DPO combines a classifier-free-guidance term with ChatGPT-plus-inpainting image contrasts. HDPO constructs about 22K targeted negatives from Visual Genome and ShareGPT4V positives. HDPO is the cheapest of the three and the hardest to beat on cost. All three supervise at the sequence level and rely on external positives or generators. Our method uses neither.
TLDR shows that word-level hallucination signals are more useful than whole-response rewards. However, it trains a separate detector on Llama-3.1-70B-perturbed captions and explicitly leaves distilling that detector into the generator for future work. We skip the detector and take the signal from the generator itself.
Decoding-time methods supply our signal but not our deployment story. CMI-VLD samples tokens by conditional pointwise mutual information with the image, which is nearly our $s_t$, but applies it at every decoding step and leaves the weights unchanged. SHARP steers mid-layer activations at inference. BIMA does modify training, but it adds a normalizing-flow network and a reference distribution built from ground-truth captions. Our change is a loss term and nothing more.
The Mirage of Performance Gains constrains how we evaluate. Its results mean that POPE accuracy alone cannot count as evidence, so generative benchmarks carry the main claim.
Research Program
Stage one is a diagnostic. It decides whether to proceed.
On held-out COCO images, we sample LLaVA-1.5-7B captions and label each mentioned object as hallucinated or not using CHAIR's object annotations. We then measure how well a low $s_t$ predicts the hallucinated label, reported as the area under the precision-recall curve. We break this down by how strongly each object co-occurs with the objects that are present. The main worry is that a real fork on a real table also receives a low $s_t$, because the text context already implies it. If low-$s_t$ flags have poor precision on real co-occurring objects, the penalty would cut recall. In that case we stop, or we move to a variant that requires annotations.
Stage two trains the model with the loss above. We use LLaVA-1.5-7B, with 13B as a scale check, on images from the llava_v1_5_mix665k mixture. We alternate rounds of re-sampling and fine-tuning and log wall-clock GPU-hours for data construction separately from training.
Stage three is the cost-accounting study. We re-implement HDPO and V-DPO data construction on the same hardware and record GPU-hours. For Synth-Align we report the reported number of model calls per sample and API token counts, rather than inventing dollar figures. All methods are then placed on a hallucination-versus-construction-cost plot.
Baselines
The naive reference point is the unmodified LLaVA-1.5-7B. Alongside it we run a "continued likelihood fine-tuning" control: the same images and the same number of gradient steps with $\lambda = 0$. This rules out gains that come merely from extra training.
The strongest training-time competitor on the hallucination metric is HDPO. On LLaVA-v1.5-7B it reduces AMBER HalRate from 35.1% to 15.8%. It is the strongest on our question because it is both effective and relatively cheap: it needs no diffusion model and no API cascade. We run it from its released data recipe on the same base checkpoint, with the same training-step budget and the same hyperparameter sweep size (eight configurations per method, selected on a held-out validation split). Beating it means matching its AMBER HalRate within one point while keeping AMBER coverage and MME at least as high, at lower construction cost. Merely matching it would still be a meaningful result, because our method needs no external positives.
Synth-Align is the strongest heavy-pipeline competitor on MMHal-Bench. It reports a score improvement from 2.36 to 3.49 and a hallucination rate reduced from 51.0% to 25.0% on LLaVA-1.5-7B, plus 87.6% POPE accuracy. We use its released SynthAlign-9K data with the same DPO trainer and step budget. If we land within its MMHal score at a small fraction of its cost, that counts as a cost-efficiency win. If we do not reach it, we report that honestly.
V-DPO is the closest in spirit, because it already contrasts with the image-removed model. It reports a 2.2-point AMBER CHAIR improvement. We run it with the same 5K RLHF-V pairs plus 5K synthetic contrasts on the same base model.
On generative CHAIR we cite BIMA (CHAIR_S 34.4%) and SHARP (CHAIR_S 52.8 to 34.8 on LLaVA-v1.5) for reference only. CHAIR protocols differ in caption length and prompt across papers, so we re-run whichever has released code under our protocol rather than comparing headline numbers.
On the decoding side we run CMI-VLD. It reports a roughly 16% reduction in sentence-level hallucination ratio on LLaVA-1.5. We also run the trivial prompt-based adjustment from The Mirage of Performance Gains, which reaches 90.2% POPE accuracy on LLaVA-13B COCO Random and serves as a guard against POPE gaming.
The ablations isolate the key ingredient, which is word-level gating by the self-computed visual score. The first replaces the gate with random selection of content words at the same flag rate. The second penalizes all content words. The third applies an ungated MixCE-style reverse term. The fourth, and the most important, is a sequence-level variant: DPO in which the rejected response is the model's own image-removed sample and the chosen response is its image-conditioned sample.
Why not just…?
Why not just use CMI-VLD or contrastive decoding at inference? On quality alone this is a real threat. CMI-VLD computes nearly the same signal and reports CHAIR gains. Two differences remain. First, decoding-time contrast requires an extra image-removed forward pass at every generated token for every user query, and our method pays that cost once during training. Second, The Mirage of Performance Gains found that contrastive decoding methods produce zero improvement, or harm, on CHAIR and MME, and that most of their POPE gains come from a plausibility constraint alone. We run CMI-VLD under identical prompts and report both quality and per-token latency. If it beats us on quality at an acceptable latency, our contribution narrows to amortizing that signal into the weights, and we will say so.
Why not just do DPO with the model's own image-removed captions as negatives? This is the most dangerous alternative. It costs about the same as our method and resembles a cheap V-DPO. Our bet is that whole-sequence negatives penalize correct words together with wrong ones, which is the coarse-supervision problem TLDR documents, and that this harms coverage and fluency, the trade-off V-DPO's authors flag. The sequence-level ablation settles the question directly. If it matches the word-level version on AMBER HalRate and coverage, the gating is unnecessary and the simpler method is the result we report.
Why not just apply MixCE to vision-language fine-tuning? MixCE's reverse term reinforces what the model already finds probable, and without visual gating that includes the co-occurrence prior that causes hallucination. MixCE also requires delicate tuning to avoid repetitive output. The ungated reverse-term ablation tests this. We expect it to shorten and sharpen captions without lowering CHAIR_I.
Why not just generate shorter captions? Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that hallucination grows with length, so any method that shortens outputs looks better on CHAIR for free. We report mean caption length and AMBER coverage for every method. We also compare all methods at matched length by truncating their outputs to the base model's length distribution, and a gain that disappears at matched length does not count.
Decisive Checks
The idea stands or falls on four results.
The first is the stage-one diagnostic: low $s_t$ must flag hallucinated objects with useful precision, including among strongly co-occurring present objects. We will report the precision-recall area under the curve.
The second is generative evidence. On COCO CHAIR (sentence and instance level) and AMBER (HalRate and coverage), the method must beat the continued-likelihood control and the random-gate ablation at matched caption length. We report POPE across all three splits only together with the "yes" rate, following Evaluating Object Hallucination in Large Vision-Language Models.
The third is a capability check: MME and MMBench must not drop by more than one point relative to the base model. This guards against the fluency loss that V-DPO and SGD warn about.
The fourth is the cost plot. On the hallucination-versus-construction-GPU-hours plot, the method should sit on the frontier: below HDPO and V-DPO in cost, and close to them in AMBER HalRate and MMHal-Bench score.
If the first check fails, the project ends cheaply. If the second succeeds but the sequence-level self-DPO matches it, we report the simpler method.
References
- On the Efficacy of Sampling Adapters
- A Diversity-Promoting Objective Function for Neural Conversation Models
- Evaluating Object Hallucination in Large Vision-Language Models
- V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization
- The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
- Mitigating Hallucinations in LVLMs via Summary-Guided Decoding
- MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies
- Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment
- Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
- TLDR: Token-Level Detective Reward Model for Large Vision Language Models
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- BIMA: Bijective Maximum Likelihood Learning Approach to Hallucination Prediction and Mitigation in Large Vision-Language Models
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context