Vision-language models describe things that are not in the image, and they do it more as the answer gets longer. The image reaches the language model only through attention. As the response grows, the model attends more and more to its own earlier words and less to the picture, so it fills in what kitchens or streets "usually" contain. We propose to give the model a second channel to the image that does not weaken with length. Before decoding, we sample a single latent "visual plan" from a distribution conditioned on the image. A latent variable here is a compact random vector that summarizes what the response should be about. This plan is added directly into the model's internal activations at every generated token, rather than placed in the context for attention to find.
The plan is trained variationally: an encoder that sees both the image and a verified, hallucination-free description teaches an image-only predictor what to put in the plan. The predictor's spread around its own guess then serves as a per-image confidence signal. We use that confidence to decide how hard to push the model. Where the plan is confident we inject it strongly, and where it is uncertain we inject it weakly. This addresses the failure of fixed-strength steering, which collapses into degenerate output when pushed too far. The claim we will test is narrow. A persistent, image-conditioned plan should flatten the curve of hallucination rate against response length, and the variational spread should predict which images will be hallucinated on. If a deterministic version of the same plan does equally well on both counts, the variational part is unnecessary, and the experiments are built to show that outcome if it is true.
Core Claim
Two papers explain why length is the problem. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that long-response hallucination is not simply accumulated error. It comes from self-attention shifting toward previously generated text to keep the narrative coherent. More Thinking, Less Seeing? reports the same drift in long chain-of-thought traces. Both findings indicate that hallucination appears because image evidence depends on an attention pathway that dilutes over a long context.
Our claim is that a global latent plan added to the residual stream at every step is immune to this dilution by construction. The residual stream is the running hidden-state vector that every layer reads from and writes to. Because the plan is not routed through attention, a long context cannot crowd it out. We further claim that parameterizing the plan as a distribution gives two things a deterministic plan cannot. The first is a calibrated injection strength. The second is a cheap signal for which images the model is likely to hallucinate on.
Where This Sits
Methods we borrow from. Variational sequence generation supplies the mechanism. Generating Sentences from a Continuous Space established that a single Gaussian latent can carry a global sentence plan. It also exposed posterior collapse: a strong decoder learns to ignore the latent entirely. Variational Prefix Tuning trains a conditional VAE over prefix vectors for a frozen pretrained model in order to diversify code summaries. We reuse its frozen-backbone conditional-VAE recipe. We differ in two ways: our objective is grounding rather than diversity, and our latent bypasses attention instead of sitting in the prefix.
The role of per-step latents. Variational Recurrent Neural Machine Translation argues that one global latent is too coarse for long outputs and introduces per-step latents. We deliberately start global, because a fixed plan is exactly what cannot drift. A recurrent variant that reads generated text would reintroduce the dependence on context that we are trying to escape.
The closest prior work. Visual Instruction Bottleneck Tuning (Vittle) already places a variational bottleneck inside a multimodal model. It compresses each token's representation at layer 24 for robustness to input noise. It reports POPE gains of up to +1.6% under corruptions and notes an OCR cost. Vittle has no persistent plan, injects nothing per decoding step, and does not study length.
Constraining work: steering. SHARP and REVIS show that hallucinated and faithful states are linearly separable in middle and deep layers, and that adding fixed vectors there helps. REVIS also reports collapse into the literal string "image.jpg" when steering is too strong. That failure is what motivates our confidence-scaled injection.
Constraining work: decoding fixes. The decoding-side methods treat symptoms at the output distribution without changing what the model carries forward. Visual Contrastive Decoding and RITUAL contrast or combine forward passes on altered images. Summary-Guided Decoding re-conditions image-related words on a text summary of the history.
Research Program
We attach a small plan module to a frozen vision-language model. We use LLaVA-1.5 7B, which most baselines report on, and Qwen2.5-VL for the REVIS comparison. The module has two parts:
- A prior network $p_\psi(z \mid x)$ reads the pooled image tokens $x$ and outputs a diagonal Gaussian over a plan vector $z$.
- A posterior network $q_\phi(z \mid x, y)$ additionally reads a reference description $y$.
The references are COCO training-split captions filtered so that every mentioned object appears in the image annotations. This filtering makes the posterior encode grounded content only.
At each decoding step $t$ and each chosen layer $l$, the hidden state is updated as follows:
$$h^{l}_{t} \leftarrow h^{l}_{t} + g(\sigma_\psi(x))\, W_l\, z,$$
The terms are:
- $W_l$ is a learned projection.
- $\sigma_\psi(x)$ is the prior's standard deviation.
- $g$ is a monotone gate that shrinks the injection when the prior is uncertain.
The layers are chosen within the 10–15 band where SHARP finds the strongest separation. At inference we use the prior mean, so the plan costs a single extra small forward pass per response, not per token.
Training maximizes the following objective:
$$\mathcal{L} = \mathbb{E}_{q_\phi}\big[\log p(y \mid x, z)\big] - \beta\,\mathrm{KL}\big(q_\phi(z \mid x,y)\,\|\,p_\psi(z \mid x)\big) - \lambda\,\big[m + \log p(y \mid x, z') - \log p(y \mid x, z)\big]_+ .$$
The terms are:
- The first term rewards a plan $z$ that helps the frozen model reproduce the grounded description.
- The KL term forces the image-only prior to predict what the grounded posterior knows.
- The final hinge uses a plan $z'$ inferred for a different image. It requires the correct plan to explain the description better than a mismatched one by a margin $m$.
The hinge is our guard against posterior collapse. It makes ignoring $z$ costly, rather than relying on KL annealing alone. We will report the number of active latent dimensions and the KL value throughout training to show that the plan is actually used.
Baselines
Naive reference points. The do-nothing reference is the frozen model with greedy decoding. Every metric below is reported at matched response lengths, because shorter captions trivially hallucinate less.
Strongest steering competitors. On CHAIR the strongest existing methods are the steering approaches. SHARP reports reducing sentence-level CHAIR on LLaVA-v1.5 from 52.8 to 34.8. REVIS reports reducing Qwen2.5-VL's CHAIR rate from 31% to 25% at greedy-decoding speed (0.025s per token). Both are the closest competitors because they also intervene in hidden states at near-zero cost.
Strongest decoding competitors. On POPE the strongest training-free comparison is RITUAL, which reports 88.87% accuracy on LLaVA-1.5 against a base of 84.13%. Visual Contrastive Decoding reports an MME total rising from 565 to 604 on LLaVA-1.5 and is run as the canonical contrastive method. Summary-Guided Decoding reports a 16.5% CHAIRs improvement on LLaVA-1.5 across 64 to 512 tokens. It is the only baseline already designed around length, so it is the direct comparison for our length curves.
Variational comparison. Vittle is retrained on the same filtered captions as the variational-in-the-model comparison.
Fairness protocol. All baselines run on the same images, prompts, and maximum lengths. Each training-free method receives the same hyperparameter search budget on a held-out COCO split as our module receives for $\beta$, $\lambda$, and the gate. That budget covers steering magnitude, threshold, contrast weight, and number of transformations.
Ablations of the key ingredients. Three ablations isolate our ingredients:
- A deterministic plan: the same networks and injection, with the KL replaced by direct regression of the prior onto the posterior mean and a fixed gate. This isolates the variational part.
- A prefix plan: $z$ is mapped to prefix tokens, as in Variational Prefix Tuning, so it reaches the model only through attention. This isolates the non-attention pathway.
- A LoRA fine-tune on the identical filtered captions. This isolates the effect of the clean data itself.
What counts as beating them. Beating a baseline means two things. First, lower CHAIRs at matched recall and length, with the gap widening at 512 tokens relative to SHARP, REVIS, and SGD. Second, no loss on MME or MM-Vet beyond the noise we measure across seeds.
Why not just…?
Why not just use contrastive decoding or image augmentation? VCD and RITUAL need no training. However, VCD runs at 0.053s per token against 0.025s for greedy decoding, as measured by REVIS. SHARP measures VCD at 13.97s against 7.04s per sample. RITUAL doubles the forward passes. More importantly, both act on per-token output distributions computed from the same diluted context, so nothing in their design targets length-driven drift. We will measure their hallucination-versus-length curves. If they stay flat at 512 tokens, our length argument fails and we will say so.
Why not just tune REVIS or SHARP more carefully? This is the most serious threat. Our deterministic-plan ablation is essentially a learned, image-specific version of their steering vector. If it matches the variational version, the contribution reduces to "learned per-image steering injected every step." In that case we will report the variational part as unnecessary. The variational part must earn its place in two concrete ways. It must avoid the over-steering collapse at injection strengths where fixed vectors degenerate, which we test with a strength sweep. Its prior spread must also predict per-image CHAIR better than REVIS's risk score does.
Why not just fine-tune on clean captions? Fine-tuning may remove much of the hallucination simply by changing the training data. The LoRA ablation is trained on the same captions with a matched parameter count. Our length hypothesis predicts specifically that LoRA helps at short lengths but still degrades with length, because it changes no pathway. If LoRA's curve is as flat as ours, the plan module is not needed.
Why not just summarize the history, as in SGD? SGD re-anchors image-related words on a compressed history. That compressed history is still text produced by the model, it requires a part-of-speech tagger, and it may require a second model. We compare directly on SGD's 64-to-512-token protocol. Our module needs neither a tagger nor an auxiliary model.
Proposed Experiments and Decisive Checks
Length curve. The first decisive experiment plots CHAIRs and CHAIRi, together with object recall, on COCO captioning at 64, 128, 256, and 512 tokens for every method. The prediction is a flatter slope for the residual-injected plan than for the prefix-plan ablation and for LoRA. That comparison separates the effect of bypassing attention from the effect of better data. We repeat the measurement on AMBER, and on RH-Bench's RH-AUC across reasoning budgets to test transfer to chain-of-thought drift. For RH-Bench the file reports 0.63 for Ocean-R1-7B, though not for our base models. We report the three POPE splits, with emphasis on the adversarial split that probes co-occurrence priors, and the full MME suite.
Uncertainty check. The second check asks whether the prior's spread $\sigma_\psi(x)$ ranks images by how much the frozen model hallucinates on them. We measure this as AUROC against per-image hallucination labels from CHAIR annotations. We compare it to REVIS's risk score and to the deterministic ablation's residual error. If spread carries no signal beyond these, the variational claim fails.
Strength sweep. The third check sweeps injection strength upward for REVIS, SHARP, the deterministic plan, and the gated variational plan. We record the strength at which fluency collapses, measured by repetition rate and degenerate strings, and test whether gating moves that point.
Capability preservation. We track MM-Vet and MME perception to confirm that general ability is preserved. MME perception matters because Vittle reports that compression hurts fine-grained reading.
Latency. Latency per token is reported against the 0.025s greedy figure.
Overall success criterion. The proposal succeeds if the length curve flattens relative to all baselines and the variational ablations show a measurable advantage on the uncertainty check or the strength sweep. If only the first holds, the result is still useful, but it should be reported as a finding about the injection pathway rather than about variational inference.
References
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
- Generating Sentences from a Continuous Space
- Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models
- Variational Recurrent Neural Machine Translation
- Visual Instruction Bottleneck Tuning
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- REVIS: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Models
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
- RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs
- Mitigating Hallucinations in LVLMs via Summary-Guided Decoding