Vision-language models, which pair an image encoder with a large text generator, often describe objects that are not in the picture. This is multimodal hallucination. The most reliable fixes today run the model two or more times per generated word. One pass sees the real image, another sees a noised or transformed copy, and the two output distributions are contrasted. That doubles or worse the cost of every answer. Visual contrastive decoding takes 13.97s per sample on LLaVA-Next, against 7.04s for a single-pass steering method (SHARP). We propose to train a small hypernetwork (a network that outputs another network's weights) to reproduce the effect of contrastive decoding in a single pass. At each point in the answer, it produces a low-rank weight correction for the vision-language model. The correction is conditioned on a signal the model already computes: how much of its middle-layer attention still lands on the image.
The bet has a specific shape. Contrastive decoding does not apply one fixed correction. Its correction is near zero for most tokens, grows as the answer lengthens and attention drifts away from the image, and differs in direction between missing-object and wrong-relation errors. A single static fine-tuned adapter trained to imitate it would have to average over this variation. A correction whose size and direction follow the model's own drift can track it. The proposal is built so that this bet can fail cheaply and visibly. If a static distilled adapter matches the conditioned one, we will report that, and the paper becomes a latency result rather than a dynamic-weights result.
Core Claim
The per-token gap between a contrastive-decoding teacher and the base model is heterogeneous across instances and across generation steps. That gap is largely predictable from the fraction of attention on visual tokens in layers 10–15. An adapter whose weights are synthesized from this signal can close most of the gap in one forward pass, and it will damage general visual question answering less than a static adapter trained on the same data.
Three findings in the file motivate the claim. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that attention shifts from the image toward previously generated text as responses grow, so the need for correction depends on the step. Beyond Global Editing shows that object and relational hallucinations occupy different activation subspaces, and that one global edit over-corrects some samples and under-corrects others. SHARP shows that faithful and hallucinated states are linearly separable, peaking around layers 10–15, which tells us where to condition and where to apply the correction. From the text-only side, CAMP finds that truthfulness and helpfulness conflict in only about 11% of preference pairs. That is indirect but consistent evidence that corrections should be sparse rather than uniform.
What Is New Relative to Existing Work
The hypernetwork adapter generators all condition on text. Text-to-LoRA maps a task description to low-rank adapter matrices. Compliance2LoRA maps a subset of safety policies to an adapter. MoEGen blends learned "expert codes" chosen by a prompt embedding and synthesizes only the down-projection matrix. Each produces one adapter per prompt, before generation starts, and none touches images or hallucination. We reuse MoEGen's asymmetric design, which it reports adds only 4.1% latency over plain LoRA, but change two things. The conditioning signal is a cross-modal quantity read from the model's own attention, not a sentence embedding. And the expert-code blend is re-evaluated every few tokens, so the weights change during generation.
On the hallucination side, SHARP adds fixed steering vectors to activations. Beyond Global Editing builds an instance-specific projection, but it needs an extra forward pass on a 70%-masked image and uses a fixed offline basis. AttnReal and ASCD edit attention scores directly. None of these changes weights, and none is trained to imitate a contrastive teacher. Our method changes weights, needs no probe pass, and learns its correction end to end from the teacher's behavior. Vision as LoRA shows that vision-specific LoRA layers inside the language model can carry perception. Its adapters are static and merged after training, so it only supports the plausibility of low-rank vision corrections and is not a competitor.
Research Program
The first stage is a cheap diagnostic that decides whether the rest is worth building. We run LLaVA-1.5 with two teachers, visual contrastive decoding and RITUAL (which combines predictions from the original and a randomly transformed image), on COCO captioning prompts. At every generated token we record the KL divergence between teacher and base distributions, the step index, and the visual attention fraction $s_t$ in layers 10–15.
If most of the variance in the teacher–base gap is explained by a single constant offset, conditioning is pointless and we stop. If much of it is explained by $s_t$ and step, as the contextual-trap analysis predicts, we proceed. We would also treat low within-instance variance combined with high between-instance variance as a reason to proceed with per-prompt conditioning only.
The second stage builds the adapter. For a target projection $W$ in layers 10–15, the adapted weight at step $t$ is
$$W_t = W + B\,A_\phi(z_t), \qquad z_t = \sum_{k=1}^{K} \pi_k(s_t, \bar h)\, e_k,$$
where $B$ is a static shared up-projection, $A_\phi$ is the hypernetwork that outputs the down-projection, the $e_k$ are $K$ learned expert codes, and $\pi_k$ are mixing weights. The mixing weights come from a small router that reads $s_t$ and a pooled mid-layer hidden state $\bar h$ of the image-and-question prefix.
The signal $s_t$ is computed from attention over the already-cached prefix during the ordinary forward pass, so no extra pass is needed. It is one step stale, which we accept. The hypernetwork is re-run only every $k$ tokens (we sweep $k \in \{1, 4, 16\}$), so the added cost is a small MLP call and a rank-$r$ matrix product.
Training uses self-generated captions and answers on COCO images. The only supervision is the teacher's token distributions. The loss is a KL term toward the teacher on image-related tokens (nouns and visual adjectives, following the part-of-speech restriction in Summary-Guided Decoding) and a KL term toward the unmodified base model on all other tokens. The second term is what should protect fluency and general question answering.
The third stage tests transfer and limits. We apply the same recipe to Qwen2-VL, which uses a direct projection connector. We also apply it to MiniGPT-4, whose compressing connector (a Q-Former, which squeezes the image into a fixed small set of tokens) is where Beyond Global Editing reports activation projection failing. Weight changes might recover grounding that activation projection cannot, or the upstream compression might cap every downstream method. Either outcome is informative. A secondary question is whether the correction can transfer to MiniGPT-4 when the hypernetwork was trained on LLaVA-1.5 alone.
Baselines
The do-nothing reference is the base model with greedy decoding under identical prompts. For context, Evaluating Object Hallucination in Large Vision-Language Models reports that MiniGPT-4's POPE F1 drops from 78.86 on the Random split to 71.37 on Adversarial, and that some models answer "yes" to nearly every POPE question.
The teachers themselves are the strongest quality baseline and an upper bound. RITUAL reports POPE accuracy of 88.87% on LLaVA-1.5 against 84.13% for the base model. We rerun both teachers in our harness. Our target is to recover at least 80% of the teacher's gain over the base model on POPE and CHAIR, at roughly single-pass latency, measured wall-clock on the same GPU and batch size.
The strongest single-pass competitors are SHARP, AttnReal and Beyond Global Editing. SHARP reports CHAIR_s falling from 52.8 to 34.8 on LLaVA-1.5 and MME perception rising from 510.0 to 613.3. AttnReal reports CHAIR_s/CHAIR_i on Qwen2-VL falling from 30.0/8.9 to 21.0/5.9. Beyond Global Editing reports CHAIR_I of 4.92 on LLaVA-1.5; it is single-pass only in the sense of needing one extra probe. These numbers come from different protocols, so we re-run all three with their published hyperparameters. We then give each the same tuning budget we give our method: a fixed number of validation evaluations on a held-out COCO split. Beating them means lower CHAIR_s and CHAIR_i at matched caption recall on the same generations, with no loss on MME.
The ablation ladder isolates the key ingredient, conditioning. It starts with a static LoRA of the same rank and layers, trained on the identical distillation data and loss. We found no reported result on Lacuna for static distillation of contrastive decoding into an adapter. Next is the same static LoRA multiplied by a learned scalar gate on $s_t$. Next is the full model conditioned only on the prompt, with no per-step update. The last is the full model. All four arms get equal trainable-parameter budgets, equal steps and equal hyperparameter-search budgets. The proposal succeeds only if the full model beats the static LoRA on hallucination metrics at no worse general question-answering accuracy, with the gap concentrated in long captions.
Why not just…?
Why not just distill the teacher into one static LoRA? This is the most serious threat. A fixed adapter trained on the same data might capture most of the teacher's effect. Nothing in the file rules it out, because no paper in the file reports this experiment. Our only evidence against it is indirect. Global edits over-correct heterogeneous hallucinations, according to Beyond Global Editing. The need for correction grows with length, according to Why LVLMs Are More Prone to Hallucinations in Longer Responses. The first-stage diagnostic and the static arm of the ablation ladder settle this directly. If the static adapter ties, we report a cheaper, simpler result and drop the dynamic claim.
Why not just gate a static LoRA by the attention signal? This could also work nearly as well. It would mean that the magnitude of the correction varies but its direction does not. The scalar-gated arm tests exactly this. A tie here would still be a clean finding, and the file's evidence that error types occupy distinct subspaces is the reason we expect direction to matter too. To test that expectation, we will break results down by object versus relation errors on AMBER.
Why not just use AttnReal, which needs no training and costs nothing at inference? AttnReal reallocates attention uniformly to all image tokens using a fixed threshold, and its authors note a trade-off with recall as the reallocation factor grows. Our method learns where and how much to correct from a teacher. The comparison at matched recall on CHAIR will show whether learning buys anything over a training-free rule. If it does not, AttnReal is the better tool and we will say so.
Why not just keep contrastive decoding and accept the latency? For offline captioning, that may be acceptable. For interactive use, it is not: a roughly doubled cost per sample (13.97s against 7.04s, per SHARP) is a real cost. ASCD also reports that visual and instruction contrastive decoding can degrade general question-answering benchmarks. Because our loss keeps non-visual tokens anchored to the base model, the distilled adapter may do better than its own teacher on general question answering. We test this on MMMU, ScienceQA, TextVQA and GQA.
Proposed Experiments
Models are LLaVA-1.5 (primary), Qwen2-VL and MiniGPT-4. Hallucination is measured with POPE on all three splits (accuracy, F1, "yes" rate), CHAIR_s and CHAIR_i on COCO captions with recall and BLEU reported alongside, AMBER, and MMHal-Bench. Capability preservation is measured on MME, MMMU, ScienceQA, TextVQA and GQA. Efficiency is measured as wall-clock seconds per sample and peak memory on fixed hardware.
The first decisive check is the stage-one variance decomposition, which is a precondition for everything else. The second is the static-versus-conditioned comparison at equal parameter count, plotted as CHAIR_i against caption length in words. The prediction is that the curves coincide for short captions and separate for long ones, where attention drift is largest. If the curves never separate, the dynamic-weights hypothesis is wrong.
The third is a counterfactual check on the conditioning signal. At test time we feed the router a shuffled $s_t$ taken from another sample. If performance does not drop, the hypernetwork is ignoring the signal and acting as a static adapter.
The fourth is the MiniGPT-4 result, which shows whether weight synthesis succeeds where activation projection failed. The fifth is latency: the full method must stay within about 10% of base-model wall-clock time at $k=4$. Otherwise the efficiency argument against the teachers collapses.
References
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs
- Conflict-Aware Adaptive Alignment for LLM Hallucination Mitigation
- Text-to-LoRA: Instant Transformer Adaption
- MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation
- Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMs
- ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM
- Vision as LoRA
- RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs
- Mitigating Hallucinations in LVLMs via Summary-Guided Decoding
- Evaluating Object Hallucination in Large Vision-Language Models