Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Models

Zhengtao Zou, Ya Gao, Jiarui Guan, Bin Li, Pekka Marttinen

TL;DR

Large Vision-Language Models (LVLMs) frequently hallucinate objects because visual information fades, or "dilutes," as text generation progresses. To solve this without sacrificing speed, the authors introduce RUDDER, a framework that extracts a zero-cost "visual anchor" from the model's initial prefill stage and adaptively injects it during decoding. RUDDER reduces object hallucinations by up to 24% while maintaining a remarkable 96% of the vanilla model's throughput.

What Problem Does This Paper Address?

Autoregressive Large Vision-Language Models typically process an image by appending its visual tokens as a prefix to the text prompt. However, as the model generates long text sequences token by token, this initial visual context undergoes visual dilution. The model essentially "forgets" the image details and begins over-relying on its internal language priors. For example, if the model generates the word "surface," its language training might confidently predict the next word to be "cup," even if no cup exists in the input image.

Existing methods try to fix this by contrasting logits (e.g., comparing text-conditioned vs. image-conditioned outputs) or iteratively refining text generation. Unfortunately, these approaches require multiple forward passes or external classifiers, introducing severe latency bottlenecks that make real-world deployment impractical.

Figure 1. (Left) An example where the vanilla LLaVA-1.5–7B (Liu et al., 2024a) hallucinates objects. Erroneous text is marked in red, while RUDDER’s corrected, factual output is in blue. (Right) A comparison showing that unlike existing non-steering and steering-based methods, RUDDER provides adaptive, low-overhead control without requiring extra forward passes.

Methods

RUDDER (Residual-Update Directed DEcoding Regulation) structurally prevents visual dilution within a single forward pass using two key innovations:

  1. CARD Vector (Contextual Activation Residual Direction): During the mandatory prefill stage (when the model initially processes the image and prompt), RUDDER places a lightweight, read-only hook at a target decoder layer. It extracts the self-attention residual updates across the prefill tokens and pools them into a single, normalized vector ($v_{\text{CARD}}$). This vector serves as a persistent, sample-specific "visual anchor" at zero extra computational cost.
  2. Beta Gate: Injecting a static visual vector uniformly can disrupt text fluency or over-steer the model on non-visual tokens (like punctuation or grammar). To prevent this, RUDDER uses a training-free Beta Gate inspired by a Beta-Bernoulli conjugate posterior. At each decoding step, it computes the cosine similarity between the current hidden state and $v_{\text{CARD}}$.
  3. High alignment indicates the model is on a visually grounded trajectory, so the gate opens wider to reinforce this path.
  4. Low/Negative alignment suggests the model is processing syntactic or non-visual tokens, so the gate suppresses the intervention to preserve natural language fluency.

The modulated steering vector is then injected directly back into the residual stream during the decoding loop.

The overall workflow of RUDDER split into the Prefill Stage (extracting CARD) and the Decoding Stage (Beta Gate modulation).

Datasets

The authors evaluated RUDDER across several standard benchmarks using a variety of base architectures (LLaVA-1.5, Idefics2, InstructBLIP, and Qwen2.5-VL):

  • CHAIR (Caption Hallucination Assessment with Image Relevance): Evaluated on 500 MSCOCO validation images to measure sentence-level ($CHAIR_S$) and object-level ($CHAIR_I$) hallucinations in open-ended descriptions.
  • POPE (Object Probing Evaluation): A dataset featuring targeted yes/no questions across random, popular, and adversarial splits to test object verification accuracy.
  • MME: A comprehensive benchmark used to ensure that mitigating hallucinations doesn't degrade general multimodal capabilities like reasoning or counting.
  • Calibration: A tiny, disjoint set of 100 MSCOCO images was used strictly for hyperparameter tuning ($\alpha_{\text{max}}$ and sensitivity $k$).

Key Insights and Ideas

  • Prefill is the Sweet Spot: The visual signal is at its maximum strength in the residual stream during the prefill stage. Capturing it here acts as a snapshot of visual evidence before the autoregressive loop dilutes it.
  • The Late-Layer Dilemma: Through layer ablation studies, the authors discovered that an intervention's leverage peaks in the late decoder layers (layers 26–32 in LLaVA-1.5-7B). However, the directional coherence of the model's natural updates collapses in these exact layers. This explains why a fixed steering vector fails and why an adaptive, token-wise gate (like the Beta Gate) is mandatory.
Figure 3. Layer ablation study on Idefics2. We fix (αmax = 8.0, k = 5.0) and vary the injection layer. Mid–late layers (L ≈28–30) prove most effective, with L=28 yielding a strong reduction.
  • Semantic Clustering: Visualizations (via t-SNE) revealed that CARD vectors naturally cluster by visual categories (e.g., airplanes, elephants), proving they capture genuine image semantics rather than random activation noise.

Key Results

  • Hallucination Reduction: Under greedy decoding, RUDDER-Beta reduced sentence-level hallucinations ($CHAIR_S$) by an average of 24.4% and object-level hallucinations ($CHAIR_I$) by 23.6% relative to vanilla baselines.
  • Maintained Efficiency: Unlike competing steering methods like VISTA (which drops throughput to 58.1% of the baseline), RUDDER-Beta maintains 96.0% throughput, operating comfortably within a single inference pass.
  • Visual Probing Performance: On the POPE benchmark, RUDDER consistently outperformed vanilla models and strong baselines, improving absolute accuracy by up to 1.0 percentage point on LLaVA-1.5 while maintaining high F1 scores.
  • Preserved Capabilities: MME evaluations confirmed that RUDDER successfully reduces hallucinations without sacrificing the model's core reasoning or perception skills.

Limitations and Future Work

The primary limitation of RUDDER is its sensitivity to hyperparameters. The optimal intervention layer ($L$), maximum steering strength ($\alpha_{\text{max}}$), and gate sensitivity ($k$) vary depending on the underlying model architecture. For instance, LLaVA-1.5 performs best with late-layer interventions ($L=30$), whereas Q-former architectures like InstructBLIP require early-layer interventions ($L=1$). Currently, these must be tuned manually using a small validation set.

The authors note that future work will focus on developing automated hyperparameter optimization techniques to make RUDDER completely plug-and-play across any new vision-language architecture.

Summary updated

Abstract

Large Vision-Language Models (LVLMs) typically process visual inputs as a prefix to the language decoder. As the model autoregressively generates text, this initial visual information inevitably undergoes "dilution" leading the model to over-rely on language priors and hallucinate objects. Existing interventions attempt to correct this by contrasting logits or iteratively refining outputs, but they incur prohibitive latency costs. We propose Residual-Update Directed DEcoding Regulation (RUDDER), a framework that counters visual dilution by creating a persistent visual anchor. We extract a robust evidence direction (CARD) directly from the model's prefill residual updates, and inject it into the decoding process. This injection is modulated by an adaptive gate, the Beta Gate, which acts as a trust mechanism and ensures the visual reminder is applied only when necessary. Experiments on LLaVA-1.5 (7B/13B), Idefics2, InstructBLIP, and Qwen2.5-VL demonstrate that RUDDER consistently mitigates hallucination (with greedy decoding, RUDDER reduces CHAIR_S by an average of 24.4% and CHAIR_i by 23.6% relative) and scales effectively across architectures, all while maintaining >96.0% throughput.
BibTeX
@inproceedings{zou2026adaptive,
  author = {Zhengtao Zou and Ya Gao and Jiarui Guan and Bin Li and Pekka Marttinen},
  title = {{Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=MEqmAlsKwS},
}
Figure