Vision-language models describe images in fluent text that drifts away from what the image shows, and the drift gets worse as the description gets longer. These models are large language models fed with image features. We think part of this drift is an old problem in a new form. During training, the model always continues a correct, human-written prefix ("teacher forcing"). At inference, it continues its own prefix, which it has never been trained on. Machine translation already showed that this mismatch, called exposure bias, pushes internal states into unfamiliar territory and causes fluent output detached from the input (On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation). Speech synthesis showed that golden and self-generated tokens form completely separate clusters in latent space (SpeechAlign). No one has measured or corrected this gap inside vision-language models.
Our proposal is to measure one quantity: how much the image changes the model's hidden state at each generated position. We call this the image's pull. We compare the pull on human prefixes with the pull on the model's own prefixes at matched positions. We then fine-tune a small adapter so the distribution of the pull on self-generated prefixes matches the distribution on human prefixes. Because the target is a match, not a maximization, the method should restore grounding late in long responses without the over-steering collapse seen with fixed steering vectors. Because the comparison against an image-free pass happens only during training, inference costs nothing extra.
Core Claim
The claim has two parts. The first is diagnostic and could fail. At equal position in a response, the image's pull on mid-to-deep hidden states is systematically weaker when the model continues its own text than when it continues human text. This gap accounts for a meaningful share of the rise in hallucination with response length that Why LVLMs Are More Prone to Hallucinations in Longer Responses documented through attention drifting onto generated text.
The second part is corrective. Aligning the distribution of this pull reduces late-response object hallucination more than three alternatives at matched data and compute:
- preference optimization on the same human versus self-generated pairs;
- inference-time steering;
- alignment of raw hidden states.
General ability, measured by MME, must not drop.
The quantity being aligned is the reason this is well-defined. Hidden states under two different prefixes are not directly comparable, because the text differs. The image's pull is comparable across prefixes, because it is a difference between two forward passes that share the same prefix. For a layer $l$, image $I$, instruction $x$ and prefix $y_{<t}$, define
$$v_l(t) = h_l(x, I, y_{<t}) - h_l(x, \varnothing, y_{<t}),$$
where:
- $h_l$ is the hidden state at the next-token position;
- $\varnothing$ is a null image, implemented as mean-pixel or a heavily noised image.
Let $P_b$ be the distribution of $v_l$ over positions in length bin $b$ under human prefixes, and $Q_b$ the same under the model's sampled prefixes. Bins are needed so that only positions at equal depth into a response are compared. The training objective is
$$\mathcal{L} = \mathcal{L}_{\text{LM}} + \lambda \sum_{l \in \mathcal{S}} \sum_{b} \text{MMD}^2\big(\text{sg}[P_b], Q_b\big),$$
where:
- $\mathcal{L}_{\text{LM}}$ is the ordinary next-token loss on human data;
- MMD is maximum mean discrepancy, a kernel distance between two sample sets;
- sg means the human-prefix branch is held fixed as a reference;
- $\mathcal{S}$ is a small set of layers;
- $\lambda$ is a weight.
Only low-rank adapter weights in those layers are trained. This follows HACL's evidence that full fine-tuning causes forgetting.
Relation to Prior Work
Several papers motivate the hypothesis without testing it. HalTrapper (Why LVLMs Are More Prone…) shows attention shifting from image tokens to generated text as responses lengthen. It does not separate "long" from "self-generated", and that separation is exactly what our matched-position diagnostic does. The machine-translation exposure-bias paper links teacher forcing to hallucination and reduces it with minimum risk training, but only for text-to-text. Pfram measures how well internal representations reflect ground-truth objects and correlates this with POPE accuracy at up to 77.62%. Its measurement uses fixed inputs, not the model's own rollouts, and it proposes no training fix.
The method borrows its form from two sources. From SpeechAlign we take the idea of pairing golden with self-generated sequences to close a training-inference gap. SpeechAlign closes the gap with iterative preference optimization on outputs; we instead align an internal statistic. From the counterfactual graph line we take the discrepancy-regularizer style of distribution alignment between factual and perturbed representations:
- Counterfactual Graph Learning for Link Prediction - Learning from Counterfactual Links
- Social Recommendation via Graph-Level Counterfactual Augmentation
Our perturbation is the model's own prefix rather than a synthetic edit. The null-image subtraction is close in spirit to the distorted-input reference used in contrastive decoding. There it is applied to output logits at every inference step; we apply it once, to hidden states, during training.
Other papers constrain the design.
- Fixed steering vectors. SHARP and REVIS steer hidden states with fixed directions at inference. They disagree on the right depth: SHARP favors layers 10–15, while REVIS argues separation only emerges deep. REVIS also reports collapse to the literal string "image.jpg" when visual steering is too strong. That is why we match a distribution instead of pushing a direction harder, and why the layer set $\mathcal{S}$ is chosen by the diagnostic rather than fixed in advance.
- Sparse autoencoder steering. SSL finds separate faithful and hallucination features with a sparse autoencoder and steers along them, again at inference.
- Projector-only contrastive training. HACL aligns image and text at the projection layer using GPT-4-written hard negatives. It works at the interface and on static pairs, so it cannot see prefix drift. We need no external model.
- Inference-time reward model. TITA adjusts token probabilities at inference with a separate reward model held in memory alongside the target model; we keep a single model.
- Context summarization. Summary-Guided Decoding shortens the text history at inference to protect visual attention. It treats the symptom at decode time, whereas we try to make the model's internals behave on its own prefixes the way they do on human ones.
Research Program
Diagnostic stage. On LLaVA-1.5-7B, chosen because most reported numbers in this area use it, we take COCO images with long human-style detailed descriptions from the model's own instruction-tuning detailed-description subset. Using the same images, we sample the model's own descriptions. At every position and every layer we compute $v_l(t)$ under both prefix types. We then plot the norm of the pull, its cosine to the pull at early positions, and MMD between the two prefix types, all as functions of position bin. We also record whether each generated object mention is a CHAIR hallucination and test whether a weak pull at the preceding position predicts it.
The diagnostic decides two things. If the pull under self-generated prefixes is indistinguishable from the pull under human prefixes at matched length, exposure bias is not the driver. In that case we stop and report the negative result. If a gap exists, the layers where it is largest define $\mathcal{S}$.
Correction stage. Rollouts are refreshed every few hundred steps, so the self-generated distribution $Q_b$ tracks the current model; this mirrors SpeechAlign's iterative rounds. We replicate the full pipeline on Qwen2.5-VL to compare with the REVIS and TITA setting.
Human prefixes are short relative to long rollouts. Bins beyond the human length range therefore use the last human bin as their reference, and we report results separately inside and outside the human length range.
Baselines
Unmodified model and length control. The do-nothing reference is LLaVA-1.5-7B with greedy decoding. We also run a length-matched variant that truncates generation at the median human description length. This tells us how much of any gain is simply shorter output, since CHAIR rewards brevity. SHARP reports a CHAIR_S of 52.8 for LLaVA-1.5 at baseline.
Golden-versus-self preference optimization. This is the strongest training-time competitor and the most direct test of our key ingredient. It is SpeechAlign's recipe ported to vision-language models: iterative preference optimization with human descriptions as preferred and the model's own descriptions as dispreferred. It uses the same images, the same number of rollout rounds, the same adapter rank and layers, and the same sweep of eight hyperparameter settings. It is the strongest competitor because it uses exactly the same signal we do (human versus self-generated) and differs only in acting on output likelihoods rather than internal statistics. No reported result found on Lacuna for this recipe on vision-language models, so we run it ourselves. Beating it means lower CHAIR_I in the late-position bins, at no worse MME and caption length.
Inference-time steering. SHARP is the strongest reported steering method on CHAIR. It reduces CHAIR_S on LLaVA-1.5 from 52.8 to 34.8 and raises MME perception from 510.0 to 613.3. We re-run it with its released settings on our evaluation split so numbers are comparable, and add SSL, which reports CHAIR_I of 8.03 falling to 6.32 on LLaVA-Next-8b. We do not expect to beat SHARP's absolute CHAIR_S by a wide margin; our method could plausibly only match it. The bar we set is therefore matching SHARP's CHAIR_S within noise with zero inference-time intervention. Our method should also stack with SHARP: our method plus SHARP should beat SHARP alone.
Test-time augmentation. RITUAL raises POPE accuracy on LLaVA-1.5 from 84.13% to 88.87% at roughly double the forward passes. It serves as the reference for POPE, where we expect at most parity, since POPE answers are one token and prefix drift barely applies.
Ablations isolating the key ingredient. The first aligns raw hidden states $h_l$ rather than the pull $v_l$. This tests whether the null-image subtraction is what makes cross-prefix comparison meaningful. The second removes position binning. The third replaces matching with maximizing the norm of $v_l$. We expect the third to over-steer in the way REVIS observed, and that would justify the distribution-matching form.
Why not just…?
Why not just do preference optimization on human versus self-generated descriptions, as SpeechAlign did? This is the real threat. The two methods share the data and the signal, so if preference optimization matches us on late-position CHAIR, the internal alignment adds nothing and we will say so. Our reason to expect a difference has two parts. First, sequence-level preferences give coarse credit: they penalize a whole self-generated description, including its correct parts. The research file lists this granularity mismatch as an open limitation of sequence-level DPO. Second, preference optimization can lower the likelihood of the model's own style (phrasing, length) without restoring visual grounding. We will measure whether each method actually raises the pull late in responses, so the experiment shows which mechanism works rather than assuming it.
Why not just steer hidden states at inference with SHARP or REVIS? Steering applies one fixed direction regardless of how far the model has drifted. It needs per-model tuning of strength, and too much strength collapses output, as REVIS's "image.jpg" failure shows. Steering also leaves training untouched, so the drift recurs whenever steering is off. The SHARP comparison and the combined condition above settle whether a trained correction is complementary or redundant.
Why not just make responses shorter or truncate context? HalTrapper reports that cropping images or enriching prompts makes hallucinations appear earlier, not disappear. MERLIM (Behind the Magic, MERLIM) reports up to 26.5% variance from rephrasing alone. Shortening also loses true content. The length-matched baseline, scored with CHAIR together with object recall, shows directly whether brevity explains our gain.
Why not just add contrastive decoding? It roughly doubles per-token latency: REVIS reports 0.053s versus 0.025s per token, and SHARP reports 13.97s versus 7.04s per sample. SGD reports that it degrades fluency. Our method has no inference cost, so the relevant question is only whether it reaches comparable CHAIR, which the SHARP comparison already brackets.
Proposed Experiments and Decisive Checks
Evaluation. Generative hallucination is scored with CHAIR_S and CHAIR_I on 500 COCO validation images. We report scores for the whole response and per position bin, together with object recall and response length so that brevity cannot masquerade as faithfulness. We also report the AMBER generative split and MMHal-Bench. POPE adversarial, with yes-ratio, checks that short-answer behavior is not harmed. MME checks general ability, where HACL showed that careless fine-tuning causes forgetting. All results average three seeds, with bootstrap confidence intervals on CHAIR.
Decisive checks.
- Gap existence. The diagnostic must show a gap between human and self-generated prefixes in the pull at matched positions, and this gap must predict hallucinated object mentions above position alone, measured as AUROC over a position-only predictor. Without this, the project stops.
- Gap closure. After training, the MMD between human-prefix and self-prefix pull distributions must shrink on held-out images. Late-bin CHAIR_I must fall by more than it does under golden-versus-self preference optimization at matched budget.
- Ingredient isolation. The raw-hidden-state ablation must underperform pull alignment, confirming that the null-image subtraction is the working ingredient.
- Generalization. Gains should replicate on Qwen2.5-VL, where REVIS reports CHAIR hallucination falling from 31% to 25%. They should also show up as reduced inpainting false positives on MERLIM's object-removal split. That second check tests whether stronger visual pull on the model's own prefixes also reduces hidden hallucinations, rather than only reducing the number of objects mentioned.
References
- On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
- SpeechAlign: Aligning Speech Generation to Human Preferences
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- Understanding Multimodal Hallucination with Parameter-Free Representation Alignment
- Counterfactual Graph Learning for Link Prediction
- Learning from Counterfactual Links for Link Prediction
- Social Recommendation via Graph-Level Counterfactual Augmentation
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- REVIS: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Models
- Steering LVLMs via Sparse Autoencoder for Hallucination Mitigation
- Hallucination Augmented Contrastive Learning for Multimodal Large Language Model
- Token-level Inference-Time Alignment for Vision-Language Models
- Mitigating Hallucinations in LVLMs via Summary-Guided Decoding
- Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models