MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding

Jingyuan Deng, Yujiu Yang

Abstract

Large vision-language models (LVLMs) have shown remarkable performance in visual-language understanding for downstream multimodal tasks. While their capabilities are improving, problems emerge simultaneously. Among those problems, hallucinations have attracted much attention, where LVLMs generate content contradictory to their input visual and text contents. Many approaches have been proposed to deal with this issue, such as contrastive decoding and attention manipulation. However, contrastive decoding methods struggle in constructing appropriate contrastive samples, and attention manipulation methods are highly sensitive, lacking stability. In this work, we propose image head Masked Contrastive Decoding (MaskCD). Our approach utilizes the "image heads" in LVLMs, masking them to construct contrastive samples for contrastive decoding. We evaluated MaskCD on LLaVA-1.5-7b and Qwen-VL-7b, using various benchmarks such as CHAIR, POPE, AMBER and MME. The results demonstrate that MaskCD effectively alleviates hallucinations and retains the general capabilities of LVLMs.

Introduction

Large Vision-Language Models (LVLMs) like LLaVA are impressive at describing images, but they suffer from a persistent flaw: hallucination. They often describe objects, attributes, or relationships that simply aren't there, undermining their reliability in critical fields like medical imaging or autonomous driving.

Existing solutions generally fall into two camps:

  1. Contrastive Decoding (CD): These methods subtract the output of a "bad" sample (e.g., a version of the image with added noise) from the original output to amplify correct visual features. However, creating a "good" bad sample is difficult; if it's too noisy, the model loses useful information.
  2. Attention Manipulation: These methods try to fix "abnormal" attention patterns but are often highly sensitive to hyperparameters and can be unstable across different tasks.

MaskCD aims to bridge this gap by using the model's internal architecture to create a high-quality "bad" sample without needing external noise or retraining.

Methodology: Image Head Masked Contrastive Decoding

The core innovation of MaskCD is the discovery and utilization of "image heads." The researchers observed that within the transformer layers of an LVLM, certain attention heads consistently pay much higher attention to image tokens than others.

The MaskCD pipeline consists of two main steps:

  1. Image Head Identification: The authors query the model with a set of images and record which attention heads focus most on the visual tokens. They then create a binary "Image Head Mask" where these specific heads are set to zero.
  2. Contrastive Decoding: During inference, the model generates two sets of "logits" (raw prediction scores): one from the original model and one where the image heads are masked. The final prediction is calculated by subtracting the masked (visually impaired) logits from the original ones. This "pushes" the model away from guesses it would make if it weren't looking at the image properly.
The MaskCD pipeline: first identifying image heads to create a mask, then using that mask to perform contrastive decoding.

Key Insights: The Role of Image Heads

A striking finding of this research is the consistency of "image heads." Whether the model is looking at a real photo or an AI-generated one, the same specific heads are responsible for processing the visual data. Unlike previous methods that add Gaussian noise to the input image (which can be unpredictable), MaskCD targets the model's internal visual processing pathways. This makes the "bad" sample more precise and the decoding process more stable.

Visualization showing that specific attention heads (yellow/green spots) consistently focus on image tokens across both real-world and AI-generated images.

Furthermore, the researchers found that "image heads" are more active when the model generates attribute tokens (like colors) compared to simple nouns. This suggests that hallucinations regarding attributes might be more deeply tied to specific attention mechanisms.

Experimental Results

MaskCD demonstrated significant improvements over baseline models and other state-of-the-art mitigation techniques like VCD and OPERA across several benchmarks:

  • Hallucination Reduction: On the CHAIR benchmark, MaskCD reduced sentence-level hallucinations (CHAIR_s) for LLaVA-1.5-7b from 50.20% to 40.6%, a nearly 20% relative improvement.
  • Robustness: In the POPE adversarial setting, MaskCD achieved an F1 score of 81.12, outperforming the baseline and other CD methods.
  • Preserving Capabilities: Comprehensive evaluation on the MME benchmark shows that MaskCD maintains or even slightly improves the model's general performance across various tasks like OCR and landmark recognition, ensuring that hallucination mitigation does not degrade general reasoning abilities.

Conclusion and Future Work

While effective, MaskCD currently requires pre-computation to identify image heads for each specific model family and uses a static mask. Future work may explore dynamic masking—identifying which heads to mask on-the-fly for each specific query—to further refine the process. Overall, MaskCD provides a clever, "white-box" approach to a complex problem, proving that understanding a model's internal attention mechanisms is key to improving its factual grounding.

Summary updated

Abstract

Large vision-language models (LVLMs) have shown remarkable performance in visuallanguage understanding for downstream multimodal tasks.While their capabilities are improving, problems emerge simultaneously.Among those problems, the hallucinations have attracted much attention, which stands for the phenomenon where LVLMs generate contradictory content to their input visual and text contents.Many approaches have been proposed to deal with this issue, such as contrastive decoding and attention manipulation.However, contrastive decoding methods struggle in constructing appropriate contrastive samples, and attention manipulation methods are highly sensitive, lacking stability.In this work, we propose image head Masked Contrastive Decoding (MaskCD).Our approach utilizes the "image heads" in LVLMs, masking them to construct contrastive samples for contrastive decoding.We evaluated MaskCD on LLaVA-1.5-7b and Qwen-VL-7b, using various benchmarks such as CHAIR, POPE, AMBER and MME.The results demonstrate that MaskCD effectively alleviates the phenomenon of hallucinations and retains the general capabilities of LVLMs.
BibTeX
@misc{deng2025maskcd,
  author = {Jingyuan Deng and Yujiu Yang},
  title = {{MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding}},
  year = {2025},
  eprint = {2510.02790},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.02790},
}
Figure