Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)

Avshalom Manevich, Reut Tsarfaty

Abstract: Large Vision-Language Models (LVLMs) are an extension of Large Language Models (LLMs) that facilitate processing both image and text inputs, expanding AI capabilities. However, LVLMs struggle with object hallucinations due to their reliance on text cues and learned object co-occurrence biases. While most research quantifies these hallucinations, mitigation strategies are still lacking. Our study introduces a Language Contrastive Decoding (LCD) algorithm that adjusts LVLM outputs based on LLM distribution confidence levels, effectively reducing object hallucinations. We demonstrate the advantages of LCD in leading LVLMs, showing up to %4 improvement in POPE F1 scores and up to %36 reduction in CHAIR scores on the COCO validation set, while also improving captioning quality scores. Our method effectively improves LVLMs without needing complex post-processing or retraining, and is easily applicable to different models. Our findings highlight the potential of further exploration of LVLM-specific decoding algorithms.

The Challenge of Object Hallucination

Large Vision-Language Models (LVLMs) like LLaVA and InstructBLIP have revolutionized multimodal AI by combining the reasoning capabilities of Large Language Models (LLMs) with visual perception. However, these models frequently suffer from "object hallucinations"—the tendency to describe objects that are not present in the image.

This phenomenon is largely driven by language bias. Because LVLMs are built upon powerful LLMs, they often prioritize linguistic statistical patterns over actual visual evidence. For instance, if a model processes the text prefix "A man is walking his...", the underlying LLM's strong prior for the word "dog" may override the visual signal of a different animal, such as a bear. This creates a "language trap" where the model sees what it expects to read rather than what is actually there.

Figure 1: An illustration of language bias. While a standard LLM predicts 'dog' with 94% confidence based on text context, LCD allows the LVLM to correctly identify the 'bear' by penalizing the linguistic prior.

Language-Contrastive Decoding (LCD)

To address this, we propose Language-Contrastive Decoding (LCD), a training-free, plug-and-play algorithm designed to mitigate linguistic bias during the decoding process. The core intuition is to treat the LVLM's output as a combination of visual information and linguistic priors, and then explicitly subtract the latter.

The Dual-Stream Approach

At each generation step, LCD performs two parallel calculations:

  1. The Multimodal View ($P_{LVLM}$): The model predicts the next token based on both the image and the text prefix.
  2. The Language-Only View ($P_{LLM}$): The model predicts the next token based only on the text prefix, effectively ignoring the image.

By contrasting these two distributions, the algorithm identifies tokens that are being suggested primarily due to linguistic habit rather than visual grounding.

Dynamic Weighting via Entropy

A key innovation in LCD is the use of entropy-based dynamic weighting. Language priors are not always harmful; they provide necessary grammatical structure. LCD uses the entropy of the LLM's distribution to determine the penalty strength. If the LLM is highly confident (low entropy), the penalty is high, forcing the model to look at the image. If the LLM is uncertain (high entropy), the penalty is reduced, allowing the model to rely on its standard multimodal logic.

Empirical Results

We evaluated LCD across several state-of-the-art models, including InstructBLIP, LLaVA 1.5, and mPLUG-Owl2. Our benchmarks included POPE (Polling-based Object Probing Evaluation) and CHAIR (Caption Hallucination Assessment in Image Description).

Table 1: Performance metrics across various LVLMs. LCD consistently improves accuracy and reduces hallucination rates (CHAIRs/CHAIRi) while maintaining or improving captioning quality (METEOR, ROUGE_L).

As shown in the results, LCD achieved a significant reduction in hallucinations, with CHAIR scores dropping by up to 36% for InstructBLIP models. Crucially, unlike other anti-hallucination methods that can make model outputs repetitive or overly brief, LCD actually improved standard captioning metrics like METEOR and ROUGE-L. This suggests that by removing linguistic "noise," the models become more descriptive and accurately grounded in the visual data.

Conclusion

Language-Contrastive Decoding offers a simple yet powerful way to improve the reliability of multimodal AI. By dynamically subtracting linguistic biases, we can steer LVLMs toward better visual grounding without the need for expensive retraining or complex post-processing. Future work will explore combining LCD with visual-contrastive methods to further refine the balance between linguistic fluency and visual accuracy.

Summary updated

Abstract

Large Vision-Language Models (LVLMs) are an extension of Large Language Models (LLMs) that facilitate processing both image and text inputs, expanding AI capabilities. However, LVLMs struggle with object hallucinations due to their reliance on text cues and learned object co-occurrence biases. While most research quantifies these hallucinations, mitigation strategies are still lacking. Our study introduces a Language Contrastive Decoding (LCD) algorithm that adjusts LVLM outputs based on LLM distribution confidence levels, effectively reducing object hallucinations. We demonstrate the advantages of LCD in leading LVLMs, showing up to %4 improvement in POPE F1 scores and up to %36 reduction in CHAIR scores on the COCO validation set, while also improving captioning quality scores. Our method effectively improves LVLMs without needing complex post-processing or retraining, and is easily applicable to different models. Our findings highlight the potential of further exploration of LVLM-specific decoding algorithms.
BibTeX
@inproceedings{manevich2024mitigating,
  author = {Avshalom Manevich and Reut Tsarfaty},
  title = {{Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=uMKCBre1O0},
}
Figure