Introduction
Large Vision-Language Models (LVLMs), such as LLaVA and GPT-4V, have demonstrated remarkable capabilities in multimodal understanding. However, they are frequently plagued by object hallucination, where the model describes objects or attributes absent from the input image. A primary driver of this issue is a linguistic "Yes-Man" bias: when faced with challenging visual queries, models often rely on language priors—guessing based on common object co-occurrences—rather than visual evidence. This leads to a significant imbalance in "Yes" responses, particularly in adversarial scenarios.
Methodology: Energy-Guided Decoding
To address this, researchers propose Energy-Guided Decoding, a training-free and hyperparameter-free strategy. The core idea leverages the "Logit Lens" technique to inspect the model's internal states across all transformer layers.
The process involves three key steps:
- Logit Projection: For each generated token, the hidden states of every layer are projected into the vocabulary space.
- Energy Calculation: An energy score (the negative LogSumExp of the logits) is calculated for each layer. In machine learning, lower energy scores often correlate with higher reliability and better out-of-distribution detection.
- Dynamic Layer Selection: The algorithm selects the hidden state from the layer with the minimum energy score to produce the final output. This approach avoids the "over-processing" often found in the final layer, where linguistic biases can override visual grounding.
Experimental Evaluation
The method was evaluated using several rigorous hallucination benchmarks, including POPE (Polling-based Object Probing Evaluation), MME, and MMVP (Multimodal Visual Patterns). These datasets specifically target the model's ability to distinguish between present and absent objects under various sampling strategies (random, popular, and adversarial).
Results and Discussion
Energy-Guided Decoding significantly outperforms standard greedy decoding, achieving an average accuracy boost of 4.82% across benchmarks. Crucially, it mitigates the "Yes" bias that plagues baseline models.
Key findings include:
- Penultimate Layer Dominance: Visualizations indicate that the energy score is frequently lowest at the penultimate layer, suggesting that the very last layer may sometimes introduce more bias than accuracy.
- Bias Reduction: The "yes-ratio gap" (deviation from a balanced 50/50 split) was reduced by an average of 8.81%, bringing the model closer to an ideal distribution of responses.
- Efficiency: Unlike contrastive decoding methods that require multiple forward passes or "expert/amateur" model pairs, this method requires only a single forward pass and minimal additional computation, making it highly efficient for real-world deployment.
Conclusion
Energy-Guided Decoding offers a simple yet effective internal solution to the hallucination problem in LVLMs. By dynamically selecting the most "reliable" internal representation based on energy scores, the method curbs linguistic over-reliance and improves visual grounding. While the method shows some limitations in object-instance level descriptions (CHAIR_I), its success in sentence-level accuracy and bias reduction marks a significant step toward more reliable multimodal AI.
Summary updated