Contrast Where It Matters: Visual Evidence-Guided Decoding for Mitigating Hallucinations in LVLMs

Yiheng Li, Wenhao Wang, Xingyi Li, Mingda Jia, Zichang Tan, Yinhang Tang, Zhen Lei, Yang YangarXiv 2026 · arXiv

TL;DR

Visual Evidence-Guided Decoding (VEGD) is a training-free framework that significantly reduces hallucinations in Large Vision-Language Models (LVLMs) by ensuring the model actually looks at the right parts of an image. By identifying query-relevant visual evidence and "anchoring" the model’s attention to that evidence throughout the decoding process, VEGD prevents models from being led astray by linguistic biases or irrelevant background details.

What Problem Does This Paper Address?

Large Vision-Language Models (LVLMs) are impressively capable, but they suffer from a persistent flaw: hallucination. This happens primarily for two reasons:

  1. Linguistic Priors: The model relies too much on its internal language training (e.g., "tennis players usually hold rackets") rather than what is actually in the image (e.g., the player is holding a water bottle).
  2. Off-target Attention: The model grounds its prediction in irrelevant or misleading image regions.

Existing "contrastive decoding" methods try to fix this by comparing a standard prediction with an "amateur" prediction (often generated from a blurred or corrupted image). The idea is that the difference between the two reveals what the model "knows" from the image versus what it's just guessing from text. However, these methods don't guarantee that the model is looking at the correct evidence in either branch. If the main branch is already looking at the wrong spot, contrasting it won't help.

Methods

VEGD introduces a two-stage, training-free approach to ensure "visual faithfulness" during generation:

1. Visual Evidence-Guided Auxiliary Branch Construction

Instead of blurring the whole image, VEGD first identifies exactly which pixels matter for the user's query.

  • Localization Head Selection: It performs a single "probe" forward pass and identifies specific attention heads that show strong visual responses and coherent spatial structure.
  • Evidence Mapping: These heads are aggregated into a "Visual Evidence" map.
  • Selective Weakening: VEGD creates an auxiliary image by masking/weakening only the identified evidence region while keeping the rest of the scene intact. This creates a much sharper contrast for the decoding stage.

2. Dual-Branch Evidence Anchoring

During the actual generation of text, VEGD runs two branches in parallel: a Main Branch (with the original image) and an Auxiliary Branch (with the weakened image).

  • Spatial Attention Intervention: It identifies "visual-carrier" heads in the final layers and forces them to attend to the previously identified evidence location. This reinforces intact evidence in the main branch and "anchors" the auxiliary branch to its weakened counterpart, preventing it from drifting toward other irrelevant regions.
  • Adaptive Fusion: Finally, it combines the predictions. If the two branches agree, it uses a collaborative fusion; if they diverge significantly (indicating a strong linguistic prior), it uses contrastive subtraction to favor the visually supported token.
Figure 2. Overview of Visual Evidence-Guided Decoding. (a) Visual Evidence-Guided Auxiliary Branch Construction identifies query- relevant visual evidence with selected localization heads and uses it to construct the auxiliary image. (b) Dual-Branch Evidence Anchoring guides visual-carrier heads in both branches toward the same evidence during decoding, followed by adaptive fusion of their predictions.

Datasets

The authors evaluated VEGD across eight different LVLM architectures (including LLaVA-1.5, InstructBLIP, and newer models like DeepSeek-VL2 and Qwen2.5-VL) using three major benchmarks:

  • POPE: Evaluates object existence across MS-COCO, A-OKVQA, and GQA.
  • CHAIR: Measures object hallucinations in open-ended image captioning.
  • MME-Hallucination: A comprehensive benchmark covering existence, count, position, and color.

Key Insights and Ideas

  • Hallucinations aren't always random guesses. The authors found that models often hallucinate because they are looking at the wrong evidence (e.g., mistaking a motorcycle for a bicycle). Masking these "misleading" regions can actually correct the hallucination (see Figure 3c).
  • Evidence access is a "middle-layer" phenomenon. Analysis showed that LVLMs focus most strongly on query-relevant evidence in their middle layers, but this focus often fades or shifts toward the output layers. This justifies why VEGD specifically "anchors" attention in those final layers to maintain focus.
  • Contrast where it matters. By only weakening the relevant evidence in the auxiliary branch, the contrastive signal becomes much cleaner. Global image corruption (like blurring the whole thing) often introduces noise that can actually degrade performance.
Figure 3. Visual-evidence analysis. (a) Correct predictions depend more strongly on query-relevant evidence. (b) Direct access to this evidence strengthens in middle layers but weakens later. (c) Hallucinations can instead follow a causally misleading visual region. Bands in (b) denote 95% bootstrap confidence intervals.

Key Results

VEGD consistently outperformed existing state-of-the-art training-free methods:

  • POPE: Achieved the highest average accuracy across all nine dataset/backbone combinations, with an average accuracy margin of 2.11% over the strongest competitor.
  • CHAIR: On LLaVA-1.5, it reduced sentence-level hallucinations ($CHAIR_S$) from 53.0 to 48.4 while actually increasing the recall of correct objects.
  • MME: Improved the total score by up to 100 points on ShareGPT4V-7B compared to regular decoding.
  • Modern Backbones: The method proved robust on the latest models, reducing hallucinations in DeepSeek-VL2 and Qwen2.5-VL without any architecture-specific tuning.

Limitations and Future Work

  • Computational Overhead: While training-free, VEGD requires a probe forward pass and running two branches during decoding, which increases inference latency.
  • Dependence on Localization Heads: The method relies on the model having at least some heads that can successfully localize the query. If a model is fundamentally incapable of finding the object in a probe pass, VEGD's guidance will be less effective.
  • Future Directions: The authors suggest exploring more efficient ways to identify evidence and extending the "anchoring" concept to video-language models where temporal evidence is also critical.

Summary updated

Abstract

Despite strong multimodal capabilities, large vision-language models (LVLMs) often hallucinate when linguistic priors outweigh visual evidence or when predictions are grounded in irrelevant or misleading image regions. Training-free contrastive decoding mitigates hallucinations by contrasting the original prediction with an auxiliary one that exposes linguistic priors. Yet existing methods focus on altering the auxiliary input or computation and do not ensure that either branch uses query-relevant visual evidence during decoding. We propose Visual Evidence-Guided Decoding (VEGD), a training-free framework that identifies query-relevant visual evidence and guides its use throughout contrastive decoding. VEGD locates this evidence by aggregating localization heads with strong visual responses and coherent spatial structure, then selectively weakens it in the auxiliary image. During generation, VEGD directs visual-carrier heads in both branches toward the same evidence location, strengthening access to intact evidence in the main branch while anchoring the auxiliary branch to its weakened counterpart. An adaptive distribution fusion strategy then combines or contrasts their predictions according to distribution divergence. Experiments across diverse LVLMs and benchmarks demonstrate that VEGD consistently reduces hallucinations.
BibTeX
@misc{li2026contrast,
  author = {Yiheng Li and Wenhao Wang and Xingyi Li and Mingda Jia and Zichang Tan and Yinhang Tang and Zhen Lei and Yang Yang},
  title = {{Contrast Where It Matters: Visual Evidence-Guided Decoding for Mitigating Hallucinations in LVLMs}},
  year = {2026},
  eprint = {2611.25524},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25524},
}
Figure