Large Vision-Language Models (LVLMs) such as LLaVA and InternVL have demonstrated remarkable capabilities in multimodal tasks. However, they remain prone to "hallucinations"—generating descriptions of objects or attributes not present in the visual input. While existing solutions often involve computationally expensive retraining or complex decoding strategies, recent research suggests a simpler, training-free intervention at the model's early layers.
The Problem: Sparse Vision Sinks
Hallucinations often occur when an LVLM prioritizes linguistic priors over actual visual evidence. For instance, if a model identifies a "dining table," it might reflexively predict the presence of a "person" regardless of the image content.
The paper "Seeing Clearly by Layer Two" identifies a structural correlate for this behavior: "vision sinks." These are specific columns in the attention matrix where image tokens capture significant attention. The researchers found that when early layers—specifically Layer 2—exhibit "sparse" attention patterns (few vision sinks), the model is significantly more likely to hallucinate.
Methodology: Enhancing Attention Heads (EAH)
To address this, the authors propose Enhancing Attention Heads (EAH), a "plug-and-play" method that requires no additional training. The approach follows three steps:
- Identify Vision Sinks: Define columns in the attention matrix where the average attention score for image tokens exceeds a specific threshold ($\beta$).
- Locate the "Dense" Head: In Layer 2, identify the attention head with the highest proportion of these vision sinks.
- Broadcast the Map: Extract the attention matrix from this "densest" head and copy (broadcast) it to all other heads within the same layer.
By anchoring all attention heads in Layer 2 to the most image-focused pattern, the model maintains a stronger visual "grounding" before proceeding to deeper reasoning layers.
Key Insights and Results
The core finding is that the information flow for image tokens is heavily concentrated in the first two layers. If the model fails to establish dense vision sinks by Layer 2, the visual anchor is lost. Statistical analysis shows a direct correlation: non-hallucinated tokens typically activate a much larger number of dense heads than hallucinated ones.
EAH was evaluated across several benchmarks with impressive results:
- Hallucination Reduction: On the CHAIR dataset, EAH reduced the sentence-level hallucination score ($CHAIR_S$) for LLaVA-1.5 from 47.0 to 36.4.
- Object Grounding: On the POPE benchmark, it achieved an F1 score of 85.7, surpassing other training-free methods like VCD and OPERA.
- Generalization: The method proved effective across seven different LVLMs (including InternVL and Shikra) and even showed benefits for pure LLMs like Llama-3.
Conclusion and Future Work
EAH offers a highly efficient way to mitigate hallucinations by simply "aligning" the model's internal eyesight at a critical early stage. While it can lead to a slight decrease in recall (the model mentions fewer objects overall), its training-free nature and minimal overhead make it a powerful tool for improving LVLM reliability. Future work may explore how the optimal intervention layer shifts as model architectures become deeper and more complex.
Summary updated