Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction

Guangtao Lyu, Xinyi Cheng, Chenghao Xu, Qi Liu, Muli Yang, Fen Fang, Huilin Chen, Jiexi Yan, Xu Yang, Cheng DengarXiv 2025 · DOI · arXiv

Introduction

Large Vision-Language Models (LVLMs) are powerful but prone to hallucinations—generating text that contradicts visual evidence. A curious "perception-generation mismatch" exists: models often attend to the correct image regions in intermediate layers but still output incorrect tokens. This research investigates the layer-by-layer dynamics of visual perception and token generation to address the root cause of these hallucinations.

Understanding the Dynamics: GATE and SAD Patterns

The authors identify two key patterns in how LVLMs process information:

  1. The GATE Pattern (Visual Perception): Visual attention evolves through three stages: Global scanning (early layers), Approaching and Tightening focus on task-relevant content (intermediate layers), and Exploring supplementary regions for context (late layers).
  2. The SAD Pattern (Token Generation): Hallucinations typically follow a Subdominant Accumulation to Dominant pattern. Incorrect tokens often start as low-probability candidates (ranks 2–5) and gradually accumulate probability across layers until they overtake the correct token in the final output.

Proposed Method: Validated Dominance Correction (VDC)

To mitigate these issues, the researchers propose Validated Dominance Correction (VDC), a training-free decoding strategy. VDC uses the "logit lens" to monitor the hidden states of Attention and Feed-Forward Network (FFN) modules at every layer.

The process involves:

  • Validation: Checking the top-predicted token from intermediate modules.
  • Detection: Flagging a final output token as a hallucination if it never appeared as a dominant token in intermediate layers.
  • Correction: Replacing the flagged token with the most frequently dominant token from across the layers, ensuring the output is grounded in visual evidence or internal knowledge.

Because VDC collects these multi-layer logits in a single forward pass, it operates at near-standard inference speeds.

Experimental Evaluation

VDC was evaluated on benchmarks like POPE (object existence), CHAIR (open-ended captioning), and MME (comprehensive attributes).

Key Results: * Broad Compatibility: Effective across various architectures including LLaVA-1.5, Qwen-VL, and MiniGPT-4. * Significant Reductions: On the CHAIR benchmark with LLaVA-1.5, VDC reduced sentence-level hallucinations from 55.1% to 33.6%. * Stackability: VDC can be combined with other methods (e.g., VCD) for even greater performance gains.

Conclusion

By uncovering the internal "Subdominant Accumulation" that leads to hallucinations, VDC provides an efficient, training-free way to align model outputs with their internal perception. Future work may explore adaptive inference, using these layer-wise dynamics to skip unnecessary computation for "easy" tokens.

Summary updated

Abstract

Large Vision-Language Models (LVLMs) have shown remarkable capabilities, yet hallucinations remain a persistent challenge. This work presents a systematic analysis of the internal evolution of visual perception and token generation in LVLMs, revealing two key patterns. First, perception follows a three-stage GATE process: early layers perform a Global scan, intermediate layers Approach and Tighten on core content, and later layers Explore supplementary regions. Second, generation exhibits an SAD (Subdominant Accumulation to Dominant) pattern, where hallucinated tokens arise from the repeated accumulation of subdominant tokens lacking support from attention (visual perception) or feed-forward network (internal knowledge). Guided by these findings, we devise the VDC (Validated Dominance Correction) strategy, which detects unsupported tokens and replaces them with validated dominant ones to improve output reliability. Extensive experiments across multiple models and benchmarks confirm that VDC substantially mitigates hallucinations.
BibTeX
@misc{lyu2025revealing,
  author = {Guangtao Lyu and Xinyi Cheng and Chenghao Xu and Qi Liu and Muli Yang and Fen Fang and Huilin Chen and Jiexi Yan and Xu Yang and Cheng Deng},
  title = {{Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction}},
  year = {2025},
  eprint = {2512.18813},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.18813},
}
Figure