Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)

Avshalom Manevich, Reut Tsarfaty

Abstract

Large Vision-Language Models (LVLMs) are an extension of Large Language Models (LLMs) that facilitate processing both image and text inputs, expanding AI capabilities. However, LVLMs struggle with object hallucinations due to their reliance on text cues and learned object co-occurrence biases. While most research quantifies these hallucinations, mitigation strategies are still lacking. Our study introduces a Language Contrastive Decoding (LCD) algorithm that adjusts LVLM outputs based on LLM distribution confidence levels, effectively reducing object hallucinations. We demonstrate the advantages of LCD in leading LVLMs, showing up to 4% improvement in POPE F1 scores and up to 36% reduction in CHAIR scores on the COCO validation set, while also improving captioning quality scores. Our method effectively improves LVLMs without needing complex post-processing or retraining, and is easily applicable to different models. Our findings highlight the potential of further exploration of LVLM-specific decoding algorithms.

The Challenge of Object Hallucination: The "Likelihood Trap"

While state-of-the-art LVLMs like LLaVA and InstructBLIP demonstrate remarkable multimodal capabilities, they frequently suffer from object hallucinations—the tendency to describe objects that are absent from the visual scene. This phenomenon is largely driven by the "likelihood trap," where the model's internal LLM backbone prioritizes strong linguistic priors over actual visual evidence.

For instance, given the prompt "A man is walking his...", a standard LLM will overwhelmingly predict "dog" based on common language patterns, even if the accompanying image depicts a bear. Because LVLMs are built upon these frozen or semi-frozen LLMs, the linguistic bias often wins the "tug-of-war" against visual inputs, leading to inaccurate generations.

The 'Likelihood Trap': An LLM's strong linguistic prior for 'dog' can override the visual evidence of a 'bear' in standard LVLM decoding.

Methodology: Language-Contrastive Decoding (LCD)

To address this, we propose Language-Contrastive Decoding (LCD). LCD is a training-free, plug-and-play decoding strategy that forces the model to prioritize visual information by penalizing linguistic biases.

At each decoding step, the algorithm calculates two probability distributions:

  1. The LVLM Distribution: The probability of the next token given both the image and the text prefix.
  2. The LLM Distribution: The probability of the next token given only the text prefix (effectively "blinding" the model to the image).

By subtracting the LLM's "blind" predictions from the LVLM's predictions, the model effectively filters out tokens that are suggested purely by language co-occurrence.

Dynamic Weighting via Entropy

A key innovation of our approach is the use of a dynamic weight ($\beta_t$) based on the entropy of the LLM's distribution. If the LLM is highly certain (low entropy), the penalty is increased to prevent bias from dominating. Conversely, if the LLM is uncertain (high entropy), the penalty is reduced to allow the model to generate naturally without unnecessary interference.

Experimental Results

We evaluated LCD across four prominent architectures: InstructBLIP (Flan-T5 and Vicuna variants), LLaVA 1.5, and mPLUG-Owl2. Our benchmarks included the POPE (Polling-based Object-Probing Evaluation) and detailed image captioning on the COCO dataset.

Hallucination Reduction and Quality Gains

LCD significantly reduced object hallucinations across all models. Notably, we observed up to a 36% reduction in CHAIR scores (a standard metric for object hallucination) for the InstructBLIP-Vicuna model.

Critically, unlike other mitigation strategies that often result in shorter or less descriptive outputs, LCD actually improved the quality of generated captions. Metrics such as METEOR and ROUGE showed consistent gains, indicating that the descriptions became more faithful to the visual content without losing detail.

Evaluation of LCD across multiple LVLMs on the COCO validation set, showing improvements in captioning quality (METEOR, ROUGE) and significant reductions in hallucination metrics (CHAIR).

Benchmark Performance

On the POPE benchmark, which tests the model's ability to correctly identify the presence or absence of objects, LCD provided up to a 4% improvement in F1 scores. The method proved robust across random, popular, and adversarial object categories, demonstrating its ability to handle both common biases and complex co-occurrence traps.

Conclusion and Future Work

Language-Contrastive Decoding offers a simple yet powerful way to mitigate hallucinations in LVLMs without the need for expensive retraining. By dynamically penalizing linguistic priors, we enable models to "see" more clearly. While the current implementation introduces some computational overhead due to the dual-pass decoding, future work involving smaller, distilled LLMs for the contrastive base could significantly improve inference speed. Our results suggest that refining decoding algorithms is a highly promising path for improving the reliability of multimodal AI.

Summary updated

Abstract

Large Vision-Language Models (LVLMs) are an extension of Large Language Models (LLMs) that facilitate processing both image and text inputs, expanding AI capabilities. However, LVLMs struggle with object hallucinations due to their reliance on text cues and learned object co-occurrence biases. While most research quantifies these hallucinations, mitigation strategies are still lacking. Our study introduces a Language Contrastive Decoding (LCD) algorithm that adjusts LVLM outputs based on LLM distribution confidence levels, effectively reducing object hallucinations. We demonstrate the advantages of LCD in leading LVLMs, showing up to %4 improvement in POPE F1 scores and up to %36 reduction in CHAIR scores on the COCO validation set, while also improving captioning quality scores. Our method effectively improves LVLMs without needing complex post-processing or retraining, and is easily applicable to different models. Our findings highlight the potential of further exploration of LVLM-specific decoding algorithms.
BibTeX
@inproceedings{manevich2024mitigating,
  author = {Avshalom Manevich and Reut Tsarfaty},
  title = {{Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=uMKCBre1O0},
}
Figure