TL;DR
Large Vision-Language Models (LVLMs) frequently hallucinate objects because visual information fades, or "dilutes," as text generation progresses. To solve this without sacrificing speed, the authors introduce RUDDER, a framework that extracts a zero-cost "visual anchor" from the model's initial prefill stage and adaptively injects it during decoding. RUDDER reduces object hallucinations by up to 24% while maintaining a remarkable 96% of the vanilla model's throughput.
What Problem Does This Paper Address?
Autoregressive Large Vision-Language Models typically process an image by appending its visual tokens as a prefix to the text prompt. However, as the model generates long text sequences token by token, this initial visual context undergoes visual dilution. The model essentially "forgets" the image details and begins over-relying on its internal language priors. For example, if the model generates the word "surface," its language training might confidently predict the next word to be "cup," even if no cup exists in the input image.
Existing methods try to fix this by contrasting logits (e.g., comparing text-conditioned vs. image-conditioned outputs) or iteratively refining text generation. Unfortunately, these approaches require multiple forward passes or external classifiers, introducing severe latency bottlenecks that make real-world deployment impractical.
Methods
RUDDER (Residual-Update Directed DEcoding Regulation) structurally prevents visual dilution within a single forward pass using two key innovations:
- CARD Vector (Contextual Activation Residual Direction): During the mandatory prefill stage (when the model initially processes the image and prompt), RUDDER places a lightweight, read-only hook at a target decoder layer. It extracts the self-attention residual updates across the prefill tokens and pools them into a single, normalized vector ($v_{\text{CARD}}$). This vector serves as a persistent, sample-specific "visual anchor" at zero extra computational cost.
- Beta Gate: Injecting a static visual vector uniformly can disrupt text fluency or over-steer the model on non-visual tokens (like punctuation or grammar). To prevent this, RUDDER uses a training-free Beta Gate inspired by a Beta-Bernoulli conjugate posterior. At each decoding step, it computes the cosine similarity between the current hidden state and $v_{\text{CARD}}$.
- High alignment indicates the model is on a visually grounded trajectory, so the gate opens wider to reinforce this path.
- Low/Negative alignment suggests the model is processing syntactic or non-visual tokens, so the gate suppresses the intervention to preserve natural language fluency.
The modulated steering vector is then injected directly back into the residual stream during the decoding loop.
Datasets
The authors evaluated RUDDER across several standard benchmarks using a variety of base architectures (LLaVA-1.5, Idefics2, InstructBLIP, and Qwen2.5-VL):
- CHAIR (Caption Hallucination Assessment with Image Relevance): Evaluated on 500 MSCOCO validation images to measure sentence-level ($CHAIR_S$) and object-level ($CHAIR_I$) hallucinations in open-ended descriptions.
- POPE (Object Probing Evaluation): A dataset featuring targeted yes/no questions across random, popular, and adversarial splits to test object verification accuracy.
- MME: A comprehensive benchmark used to ensure that mitigating hallucinations doesn't degrade general multimodal capabilities like reasoning or counting.
- Calibration: A tiny, disjoint set of 100 MSCOCO images was used strictly for hyperparameter tuning ($\alpha_{\text{max}}$ and sensitivity $k$).
Key Insights and Ideas
- Prefill is the Sweet Spot: The visual signal is at its maximum strength in the residual stream during the prefill stage. Capturing it here acts as a snapshot of visual evidence before the autoregressive loop dilutes it.
- The Late-Layer Dilemma: Through layer ablation studies, the authors discovered that an intervention's leverage peaks in the late decoder layers (layers 26–32 in LLaVA-1.5-7B). However, the directional coherence of the model's natural updates collapses in these exact layers. This explains why a fixed steering vector fails and why an adaptive, token-wise gate (like the Beta Gate) is mandatory.
- Semantic Clustering: Visualizations (via t-SNE) revealed that CARD vectors naturally cluster by visual categories (e.g., airplanes, elephants), proving they capture genuine image semantics rather than random activation noise.
Key Results
- Hallucination Reduction: Under greedy decoding, RUDDER-Beta reduced sentence-level hallucinations ($CHAIR_S$) by an average of 24.4% and object-level hallucinations ($CHAIR_I$) by 23.6% relative to vanilla baselines.
- Maintained Efficiency: Unlike competing steering methods like VISTA (which drops throughput to 58.1% of the baseline), RUDDER-Beta maintains 96.0% throughput, operating comfortably within a single inference pass.
- Visual Probing Performance: On the POPE benchmark, RUDDER consistently outperformed vanilla models and strong baselines, improving absolute accuracy by up to 1.0 percentage point on LLaVA-1.5 while maintaining high F1 scores.
- Preserved Capabilities: MME evaluations confirmed that RUDDER successfully reduces hallucinations without sacrificing the model's core reasoning or perception skills.
Limitations and Future Work
The primary limitation of RUDDER is its sensitivity to hyperparameters. The optimal intervention layer ($L$), maximum steering strength ($\alpha_{\text{max}}$), and gate sensitivity ($k$) vary depending on the underlying model architecture. For instance, LLaVA-1.5 performs best with late-layer interventions ($L=30$), whereas Q-former architectures like InstructBLIP require early-layer interventions ($L=1$). Currently, these must be tuned manually using a small validation set.
The authors note that future work will focus on developing automated hyperparameter optimization techniques to make RUDDER completely plug-and-play across any new vision-language architecture.
Summary updated