Energy-Guided Decoding for Object Hallucination Mitigation

Xixi Liu, Ailin Deng, Christopher ZacharXiv 2025 · DOI · arXiv

Introduction

Large Vision-Language Models (LVLMs), such as LLaVA and GPT-4V, have demonstrated remarkable capabilities in multimodal understanding. However, they are frequently plagued by object hallucination, where the model describes objects or attributes absent from the input image. A primary driver of this issue is a linguistic "Yes-Man" bias: when faced with challenging visual queries, models often rely on language priors—guessing based on common object co-occurrences—rather than visual evidence. This leads to a significant imbalance in "Yes" responses, particularly in adversarial scenarios.

Methodology: Energy-Guided Decoding

To address this, researchers propose Energy-Guided Decoding, a training-free and hyperparameter-free strategy. The core idea leverages the "Logit Lens" technique to inspect the model's internal states across all transformer layers.

Overview of the Energy-Guided Decoding framework, illustrating layer-wise energy scores and the selection of the most reliable hidden state.

The process involves three key steps:

  1. Logit Projection: For each generated token, the hidden states of every layer are projected into the vocabulary space.
  2. Energy Calculation: An energy score (the negative LogSumExp of the logits) is calculated for each layer. In machine learning, lower energy scores often correlate with higher reliability and better out-of-distribution detection.
  3. Dynamic Layer Selection: The algorithm selects the hidden state from the layer with the minimum energy score to produce the final output. This approach avoids the "over-processing" often found in the final layer, where linguistic biases can override visual grounding.

Experimental Evaluation

The method was evaluated using several rigorous hallucination benchmarks, including POPE (Polling-based Object Probing Evaluation), MME, and MMVP (Multimodal Visual Patterns). These datasets specifically target the model's ability to distinguish between present and absent objects under various sampling strategies (random, popular, and adversarial).

Results and Discussion

Energy-Guided Decoding significantly outperforms standard greedy decoding, achieving an average accuracy boost of 4.82% across benchmarks. Crucially, it mitigates the "Yes" bias that plagues baseline models.

Comparison of "Yes" ratios between Greedy and Energy-guided decoding across different datasets and sampling strategies.

Key findings include:

  • Penultimate Layer Dominance: Visualizations indicate that the energy score is frequently lowest at the penultimate layer, suggesting that the very last layer may sometimes introduce more bias than accuracy.
  • Bias Reduction: The "yes-ratio gap" (deviation from a balanced 50/50 split) was reduced by an average of 8.81%, bringing the model closer to an ideal distribution of responses.
  • Efficiency: Unlike contrastive decoding methods that require multiple forward passes or "expert/amateur" model pairs, this method requires only a single forward pass and minimal additional computation, making it highly efficient for real-world deployment.

Conclusion

Energy-Guided Decoding offers a simple yet effective internal solution to the hallucination problem in LVLMs. By dynamically selecting the most "reliable" internal representation based on energy scores, the method curbs linguistic over-reliance and improves visual grounding. While the method shows some limitations in object-instance level descriptions (CHAIR_I), its success in sentence-level accuracy and bias reduction marks a significant step toward more reliable multimodal AI.

Summary updated

Abstract

Mitigating object hallucination in large vision-language models (LVLMs) is critical to their safe deployment. Existing methods either are restricted to specific decoding methods, or demand sophisticated modifications to visual inputs, or rely on knowledge from external models. In this work, we first reveal the phenomenon that VLMs exhibit significant imbalance in the ``Yes'' ratio ( \ie, the fraction of ``Yes'' answers among the total number of questions) across three different visual question answering (VQA) datasets. Furthermore, we propose an energy-based decoding method, which dynamically selects the hidden states from the layer with minimal energy score. It is simple yet effective in reducing the bias for the yes ratio while boosting performance across three benchmarks (POPE, MME, and MMVP). Our method consistently improves accuracy and F1 score on three VQA datasets across three commonly used VLMs over several baseline methods. The average accuracy improvement is 4.82% compared to greedy decoding. Moreover, the average yes-ratio gap reduction is 8.81%, meaning the proposed method is less biased as shown in Figure 1.
BibTeX
@misc{liu2025energy,
  author = {Xixi Liu and Ailin Deng and Christopher Zach},
  title = {{Energy-Guided Decoding for Object Hallucination Mitigation}},
  year = {2025},
  eprint = {2507.07731},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.07731},
}
Figure