Introduction
Large Vision-Language Models (LVLMs) are powerful but prone to hallucinations—generating text that contradicts visual evidence. A curious "perception-generation mismatch" exists: models often attend to the correct image regions in intermediate layers but still output incorrect tokens. This research investigates the layer-by-layer dynamics of visual perception and token generation to address the root cause of these hallucinations.
Understanding the Dynamics: GATE and SAD Patterns
The authors identify two key patterns in how LVLMs process information:
- The GATE Pattern (Visual Perception): Visual attention evolves through three stages: Global scanning (early layers), Approaching and Tightening focus on task-relevant content (intermediate layers), and Exploring supplementary regions for context (late layers).
- The SAD Pattern (Token Generation): Hallucinations typically follow a Subdominant Accumulation to Dominant pattern. Incorrect tokens often start as low-probability candidates (ranks 2–5) and gradually accumulate probability across layers until they overtake the correct token in the final output.
Proposed Method: Validated Dominance Correction (VDC)
To mitigate these issues, the researchers propose Validated Dominance Correction (VDC), a training-free decoding strategy. VDC uses the "logit lens" to monitor the hidden states of Attention and Feed-Forward Network (FFN) modules at every layer.
The process involves:
- Validation: Checking the top-predicted token from intermediate modules.
- Detection: Flagging a final output token as a hallucination if it never appeared as a dominant token in intermediate layers.
- Correction: Replacing the flagged token with the most frequently dominant token from across the layers, ensuring the output is grounded in visual evidence or internal knowledge.
Because VDC collects these multi-layer logits in a single forward pass, it operates at near-standard inference speeds.
Experimental Evaluation
VDC was evaluated on benchmarks like POPE (object existence), CHAIR (open-ended captioning), and MME (comprehensive attributes).
Key Results: * Broad Compatibility: Effective across various architectures including LLaVA-1.5, Qwen-VL, and MiniGPT-4. * Significant Reductions: On the CHAIR benchmark with LLaVA-1.5, VDC reduced sentence-level hallucinations from 55.1% to 33.6%. * Stackability: VDC can be combined with other methods (e.g., VCD) for even greater performance gains.
Conclusion
By uncovering the internal "Subdominant Accumulation" that leads to hallucinations, VDC provides an efficient, training-free way to align model outputs with their internal perception. Future work may explore adaptive inference, using these layer-wise dynamics to skip unnecessary computation for "easy" tokens.
Summary updated