Vision-language models write image descriptions one word at a time, and every word they write becomes input for the next. Suppose such a model wrongly mentions a "fork" in a photo that contains no fork. It then tends to add a "knife" and a "plate of pasta," because it now treats its own mistake as a fact about the image. This proposal asks two linked questions. First, how much of the hallucination in long descriptions is inherited from earlier mistakes in the same response, as opposed to arising fresh at each step? Second, can a model be fine-tuned to recover from its own mistakes by training it on prefixes that contain its real hallucinations and supervising only what it writes afterwards? This is the vision-language version of the "exposure bias" fix from machine translation: the model sees its own imperfect outputs during training instead of only clean reference text.
The opportunity is that almost every existing remedy for multimodal hallucination tries to stop the first wrong word. Some change decoding, some change positional encodings, some curate training data. None of them measures or trains the separate skill of not compounding a wrong word once it exists. If inheritance accounts for a large share of long-response errors, recovery tuning addresses that share directly. It adds no inference cost, no extra forward passes and no new vocabulary.
Core Claim
Long-response hallucination in large vision-language models (LVLMs, an image encoder feeding an autoregressive language model) has two components. One is a fresh component, where the model drifts from the image regardless of what it has already written. The other is an inherited component, where a hallucinated word in the prefix raises the chance of further, semantically related hallucinations. The inherited component can be measured causally by inserting controlled words into a prefix. It can be reduced by fine-tuning on the model's own hallucinated prefixes with loss applied only to grounded continuations. The prediction is that this training cuts the inherited component far more than equal-budget fine-tuning on clean captions does. It should also leave single-token tasks such as yes/no object polling essentially unchanged, which serves as a built-in check that the gain comes from recovery rather than from generally better perception.
Relation to Prior Work
Two strands motivate the proposal. The first shows that errors propagate but stops short of fixing propagation inside a single response. Multi-Object Hallucination in Vision Language Models introduces the ROPE benchmark, which asks a model to name several objects marked by red boxes in one image. It shows that after naming four apples the model calls a fifth object, a pear, an apple as well, with accuracy "dropped to near zero" under teacher forcing and less than 20% attention on image tokens. That paper diagnoses the effect but offers no fix. Investigating and Mitigating the Multimodal Hallucination Snowballing shows the same snowballing across dialogue turns and fixes it at inference with Residual Visual Decoding. That method compares predictions with and without the dialogue history, so it cannot help when the poisoned context is the current response itself.
Why LVLMs Are More Prone to Hallucinations in Longer Responses partly cuts against this proposal. It argues that long-response hallucination is not passive accumulation of per-token errors but a "contextual trap," in which attention migrates from image tokens to the model's own text. We take this seriously. The injection experiment below is built to separate content-driven inheritance from content-independent attention drift. If the drift account explains nearly everything, recovery tuning should fail, and we will report that.
A third strand constrains the design. Generate, but Verify (REVERSE) also trains on hallucination-aware data, about 1.3M semi-synthetic examples. It teaches the model to tag phrases as confident or unconfident and then backtracks and resamples at inference. Recovery tuning differs in three ways. It uses the model's own natural hallucinations rather than synthetic ones. It needs no special tokens or inference-time rollback. It trains the continuation policy rather than a detector. CODE also uses the model's own error-prone description, but only as a negative reference at decoding time, at the cost of a full extra generation pass.
The method comes from Bridging the Gap between Training and Inference for Neural Machine Translation. That paper feeds the decoder its own predictions during training and chooses "sentence-level oracle" continuations by searching for good full sequences. It reports gains concentrated on long sentences, which is the pattern we expect for long image descriptions. The idea has not been applied to visual instruction tuning, and the vision setting supplies something translation lacks: object annotations that give free, automatic labels for which prefix words are hallucinated.
Research Program
Measuring inheritance. The first stage is a causal measurement. We take grounded captions generated by LLaVA-1.5-7B and Qwen-VL on COCO 2014 validation images and truncate each at position $k$. At that point we insert one of three phrases, each matched for fluency and length:
- an absent object that commonly co-occurs with objects in the image, chosen by the adversarial sampling rule of POPE;
- an absent object that rarely co-occurs;
- a present but not-yet-mentioned object, which serves as the control.
The model then continues greedily. We score the continuation alone, excluding the inserted word, with instance-level CHAIR (the fraction of mentioned objects absent from the image's annotations).
The inheritance gap is the hallucination rate after absent insertions minus the rate after present insertions. The present control absorbs any effect of added length or shifted attention that is independent of content. The difference between frequently and rarely co-occurring absent insertions tells us whether inheritance travels along co-occurrence statistics, the mechanism HalluTrace calls language-prior dominance. Sweeping $k$ tests whether early mistakes do more damage than late ones. That asymmetry has been shown for quantization noise by Progressive Mixed-Precision Decoding but has not been checked for hallucination. We also log the share of attention on image tokens after each insertion, so we can ask whether the absent-versus-present gap persists at matched attention levels. This is the direct test against the contextual-trap account.
Recovery tuning. The second stage builds the training set from the model's own sampled captions on COCO training images. Wherever COCO annotations flag a hallucinated object, we cut the caption just after that phrase. We sample several continuations from that prefix and keep the one with the best continuation-only score, meaning zero hallucinated objects and the highest recall of annotated objects. This mirrors the sentence-level oracle of the translation work. We then fine-tune with low-rank adapters. Cross-entropy is applied only to the selected continuation, and the prefix, including the hallucinated word, is masked from the loss. This masking isolates recovery from first-error avoidance.
As in DAgger-style training, the process is repeated for two rounds so that later prefixes reflect the tuned model's own errors. A variant adds an unlikelihood penalty on the hallucinated prefix word to test whether combining recovery with avoidance helps or whether the two interfere. Because COCO's 80-category vocabulary limits the labels, which is a known restriction also noted for HalluTrace, all transfer evaluation uses benchmarks whose annotations are independent of COCO's.
Baselines
The naive reference point is the untuned model with greedy decoding. On LLaVA-1.5-7B, HalluTrace reports CHAIR_i of 14.2 and POPE-Adversarial F1 of 85.9%. We rerun these under our own prompt and generation length rather than borrowing the figures.
The most important ablation is clean-caption tuning. It uses the same adapter, the same number of gradient steps, the same images and the same number of supervised tokens, but its targets are the model's own fully grounded sampled captions, with no hallucinated prefixes. This isolates the key ingredient, conditioning on real errors. Recovery tuning must beat it on the inheritance gap and on CHAIR_i, over at least three seeds with non-overlapping confidence intervals. Otherwise the gain is ordinary grounded fine-tuning.
The strongest training-based competitor is REVERSE. It reports a 12% reduction in CHAIR hallucination on MSCOCO. We will use its released checkpoint if one is available and report its training-data scale honestly, since recovery tuning uses far less data. Beating it means a larger relative CHAIR reduction from the same base model at comparable object recall and with a single decoding pass.
The strongest training-free competitors are:
- VCD (Visual Contrastive Decoding), which subtracts logits computed from a noised image and reaches CHAIR_i 10.8 on LLaVA-1.5-7B per HalluTrace;
- HAD (HalluTrace), which routes interventions by diagnosed error source and reaches CHAIR_i 8.1;
- ACD (Attention Contrastive Decoding), which moves the contrast into attention maps and reports an 11.7% CHAIR_i reduction on LLaVA-1.5 without breaking fluency.
All are run with their published hyperparameters and one small tuning sweep of equal size for every method. We report CHAIR_i, recall and BLEU-4 against references to expose fluency damage. Because these methods act on the first error while ours acts on the continuation, the decisive comparison is the inheritance gap under each. We also test recovery tuning combined with ACD. Being additive with ACD would count as a success even if recovery tuning alone did not reach HAD's 8.1.
Why not just…?
Why not just use contrastive decoding such as VCD or ACD everywhere? These methods lower the per-step pull of language priors, but nothing in them conditions on whether the prefix is already wrong. Once "fork" is in the context, the clean and corrupted-image branches both see it. Measuring the inheritance gap under ACD settles this question. If ACD alone closes the gap, recovery tuning is unnecessary for that model, and we will say so. VCD also doubles the forward passes per token, and ACD documents that logit-level contrast cuts BLEU-4 by about 42% and produces malformed outputs up to 69% of the time on Qwen3-VL-8B.
Why not just fine-tune on clean, grounded captions? This is the real threat, because better grounding also reduces first errors and therefore reduces inheritance. That is why clean-caption tuning is the matched ablation. The separating evidence is twofold. Recovery tuning should shrink the absent-minus-present inheritance gap more than clean tuning does. Both should move POPE by similar small amounts, since yes/no polling involves no prefix to recover from. If clean tuning matches recovery tuning on the inheritance gap, the hypothesis is refuted.
Why not just backtrack and resample, as REVERSE does? Backtracking works, but it needs a trained self-verification head, custom tokens, about 1.3M training examples and repeated decoding at inference. It also depends on the verifier recognizing its own errors, which Overthinking Causes Hallucination suggests is hard when a strong prior produces confident, low-entropy hallucinations. Recovery tuning needs no detection at inference. The comparison above shows whether that simplicity costs accuracy.
Why not just generate shorter descriptions? Truncation lowers CHAIR trivially and lowers recall along with it. We report all methods on CHAIR-versus-recall curves at matched output lengths, so any gain has to hold at equal coverage.
Proposed Experiments and Decisive Checks
The first check is a go/no-go on the measurement. If the inheritance gap on LLaVA-1.5-7B is not clearly positive, meaning absent insertions do not raise continuation CHAIR_i above present insertions, then inherited errors are a minor effect. The tuning study would lose its motivation, and the negative result would itself support the contextual-trap account.
If the gap is positive, the main results are:
- In-distribution effect: CHAIR_i and CHAIR_s on COCO with recall held fixed.
- Transfer: generative AMBER, which has independent annotations (no reported baseline number found on Lacuna for LLaVA-1.5 on AMBER).
- Within-response momentum: the ROPE adversarial AAAAB split. Its teacher-forced setting is exactly an injected-prefix test, and the reported baseline accuracy on the final object is near zero.
MMHalSnowball tests whether recovery learned within a response transfers to hallucinations injected across dialogue turns. There, Residual Visual Decoding raised LLaVA-1.5 accuracy from about 15% to 33% in the hallucinated-conversation setting, a concrete bar to compare against or combine with. POPE is the negative control. Large POPE gains would mean we trained general perception rather than recovery.
Finally, the position sweep, repeated after tuning, shows whether recovery tuning flattens the sensitivity to early mistakes. It also shows whether the remaining inheritance concentrates at early positions. If it does, costly decoding interventions could be applied only to the first few object mentions, which would reduce inference cost.
References
- Multi-Object Hallucination in Vision Language Models
- Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling
- CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models
- Bridging the Gap between Training and Inference for Neural Machine Translation
- Evaluating Object Hallucination in Large Vision-Language Models
- HalluTrace: Causal Attribution and Source-Targeted Decoding for Hallucination in Large Vision-Language Models
- Progressive Mixed-Precision Decoding for Efficient LLM Inference
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
- Beyond Logits: Coherent Hallucination Mitigation via Attention Contrastive Decoding
- Overthinking Causes Hallucination: Tracing Confounder Propagation in Vision Language Models