Multi-Frequency Contrastive Decoding: Alleviating Hallucinations for Large Vision-Language Models

Bingqian Liu, Fu Zhang, Guoqing Chen, Jingwei ChengEMNLP 2025

TL;DR

This paper introduces Multi-Frequency Contrastive Decoding (MFCD), a novel, training-free method to reduce "object hallucinations" in large vision-language models (LVLMs). By analyzing how image frequency information impacts hallucinations, MFCD dynamically contrasts model outputs based on original and frequency-filtered images, significantly improving factual accuracy without altering model architecture.

What Problem Does This Paper Address?

Large vision-language models (LVLMs) are powerful tools that combine visual understanding with language generation. However, a persistent challenge is "object hallucination," where the model generates descriptions that include objects not present in the image. Previous work attributed this to linguistic biases and data imbalances. This paper delves deeper, proposing that a key factor is the LVLM's neglect of specific frequency components within the image. Low-frequency information outlines general shapes, while high-frequency information captures fine details and edges. When LVLMs fail to adequately process these frequencies, existing linguistic priors are amplified, leading to the generation of non-existent objects. This issue undermines the reliability and credibility of LVLMs.

Methods

The core of this paper is the Multi-Frequency Contrastive Decoding (MFCD) method. It's a post-processing technique applied during the decoding (generation) phase, meaning it doesn't require any retraining of the LVLM.

Here's how it works:

  1. Frequency Filtering: The input image is processed in three ways:
  2. Original Image: The standard input.
  3. High-Pass Filtered Image: This image retains high-frequency details (edges, textures) but removes low-frequency information (smooth areas, overall shapes).
  4. Low-Pass Filtered Image: This image retains low-frequency information (smooth areas, contours) but removes high-frequency details.
  5. Contrastive Decoding: MFCD then compares the output probability distribution of the LVLM when given the original image against the distributions generated from the high-pass and low-pass filtered images.
  6. The intuition is that if an object is hallucinated, its probability will be higher when certain frequency information is missing from the input image.
  7. By contrasting these distributions, MFCD dynamically adjusts the model's output probabilities to suppress the likelihood of generating hallucinated objects.
  8. Adaptive Plausibility Constraints (Optional but included in MFCD-Plus): To further refine the process and avoid penalizing reasonable tokens, MFCD can incorporate adaptive plausibility constraints. This mechanism helps ensure that the model doesn't wrongly penalize correct predictions or wrongly reward incorrect ones.
  9. MFCD-Plus: An extension that dynamically adjusts the contrast parameters based on the similarity between the original and hallucinated distributions, and the model's uncertainty at each time step.

Datasets

The paper evaluates its method on several standard benchmarks:

  • POPE (Perception of Objects in Pictures Evaluation): A dataset designed to evaluate object hallucination in LVLMs. It uses a binary-class visual question answering format ("Is there a <object> in the image?"). The paper uses POPE constructed from MSCOCO.
  • CHAIR (CHAIRi and CHAIRs): A metric specifically designed for evaluating object hallucination in image captioning tasks.
  • MME (Multimodal Multi-task Evaluation): A comprehensive benchmark that includes perception and cognitive tasks, with a subset specifically for evaluating object hallucinations.
  • LLaVA-Bench (in-the-wild): Used to evaluate capabilities beyond hallucination alleviation, focusing on generation quality across dimensions like helpfulness, relevance, accuracy, and level of detail.

Key Insights and Ideas

  • Frequency Matters: The paper's central insight is that the neglect of specific image frequency information (both high and low) is a significant contributor to object hallucinations in LVLMs, amplifying existing linguistic biases.
  • Training-Free Solution: MFCD is a "plug-and-play" method, meaning it can be applied to existing LVLMs without any fine-tuning or architectural changes, making it highly practical.
  • Dynamic Contrast: The core mechanism of contrasting outputs from original and filtered images is a novel way to identify and suppress hallucinated content.
  • Model Sensitivity Varies: The paper shows that different LVLMs have varying sensitivities to frequency information. For instance, Qwen2.5-VL relies heavily on low-frequency information, while LLaVA-NeXT seems to utilize it less, impacting how much MFCD helps each model.

Key Results

  • Significant Hallucination Reduction: MFCD consistently outperforms baseline decoding methods (Sample, Dola, VCD, SID) across various LVLMs (LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL) on POPE, CHAIR, and MME benchmarks.
  • POPE Performance: On the POPE dataset, MFCD achieved the best accuracy and F1 scores across Random, Popular, and Adversarial settings for LLaVA-1.5 and Qwen2.5-VL. For example, on LLaVA-1.5, MFCD achieved 87.07% accuracy and 87.73% F1 score in the Random setting, outperforming other methods.
  • CHAIR Improvement: MFCD significantly reduced CHAIRi and CHAIRs scores, indicating fewer hallucinated objects and sentences in image captions.
  • MME Performance: MFCD showed improvements across all levels of MME hallucination evaluation (Existence, Count, Position, Color), with notable gains in the challenging Count and Color tasks.
  • MFCD-Plus Benefits: The adaptive MFCD-Plus variant further improved accuracy and F1 scores on POPE compared to the base MFCD, demonstrating the effectiveness of adaptive parameters.
  • Beyond Hallucinations: Experiments on LLaVA-Bench suggest MFCD can also improve other generation quality aspects, like helpfulness and detail.

Limitations and Future Work

  • Scope of Hallucinations: MFCD primarily addresses hallucinations caused by the neglect of image frequency information. It may not fully mitigate hallucinations stemming from other factors, such as incorrect user instructions or inherent limitations in image information processing by LVLMs.
  • Modality Support: The current MFCD method is designed for LVLMs that process both image and text inputs. It does not yet support models handling video or other modalities.
  • Potential Bias Amplification: The authors acknowledge that contrastive decoding, in general, might inadvertently amplify other unknown biases within the model.
  • Future Work: The paper suggests exploring MFCD-Plus for more dynamic parameter adaptation and extending the method to other modalities. Further investigation into the specific reasons for varying model sensitivities to frequency information is also warranted.

Summary updated

BibTeX
@inproceedings{liu2025multi,
  author = {Bingqian Liu and Fu Zhang and Guoqing Chen and Jingwei Cheng},
  title = {{Multi-Frequency Contrastive Decoding: Alleviating Hallucinations for Large Vision-Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure