Multimodal Large Language Models (MLLMs) frequently suffer from a frustrating flaw: they "hallucinate" objects, attributes, or spatial relationships that do not exist in the input image. When asked to describe a kitchen scene, an MLLM might confidently mention a refrigerator or a microwave simply because those items typically appear in kitchens.
This happens because MLLMs are fundamentally text models augmented with vision. During text pretraining, they absorb powerful statistical correlations—language priors—that dictate which words commonly follow one another. At inference time, these internal linguistic biases often overpower the actual visual conditioning. The model generates text based on what it expects to see rather than what is actually there.
To fix this without expensive retraining, a powerful paradigm has emerged: Visual Contrastive Decoding. Instead of modifying the model's weights, this family of training-free methods alters the decoding process in logit space, subtracting a "biased" or "blind" prediction from the standard prediction to isolate and amplify the true visual signal.
The Core Intuition: Bias Subtraction
At each step of text generation, an MLLM outputs a vector of unnormalized log-probabilities (logits) over the vocabulary. In standard decoding, we apply a softmax function to these logits to sample the next token.
Visual contrastive decoding introduces a second, parallel inference path. This parallel path is intentionally crippled or biased so that it relies primarily on the model's internal flaws (such as language priors or uninformative background noise). By subtracting the logits of this negative path from the standard path, we cancel out the shared biases:
$$\text{Logits}_{\text{final}} = \text{Logits}_{\text{standard}} - \alpha \cdot \text{Logits}_{\text{negative}}$$
Where $\alpha$ is a scaling hyperparameter.
To see why this works, imagine the model is generating the next word for a kitchen image that does not contain a refrigerator.
- In the standard path (with the full image), the token
"refrigerator"gets a high score because of the strong linguistic association with kitchens, even though the visual evidence is zero. - In the negative path (where the image is removed or blurred), the token
"refrigerator"still gets a high score because the text context alone strongly implies it. - When we subtract the two, the high scores cancel out, preventing a hallucination.
Conversely, if there is an unexpected object in the kitchen—say, a "parrot":
- In the standard path,
"parrot"receives a moderate score because the visual evidence forces the model to consider it, despite a low language prior. - In the negative path,
"parrot"receives a near-zero score because it makes no sense from text context alone. - Subtracting the negative path leaves the score for
"parrot"intact or relatively amplified, ensuring the model grounds its output in reality.
Strategies for Constructing the Negative Baseline
The primary differentiator among recent papers is how they construct the negative or biased baseline ($\text{Logits}_{\text{negative}}$). The field has explored several distinct dimensions of contrast.
1. Language-Only Baselines (Blind Paths)
The most direct way to isolate language bias is to run a forward pass completely devoid of visual information.
- Avshalom Manevich et al. in their work on Language-Contrastive Decoding (LCD) subtract the output probabilities of a language-only model from the multimodal model, penalizing tokens that are highly predictable from text alone.
- Shitian Zhao et al. introduce a post-hoc framework called Likelihood Composition in Unleashing the Potentials of Likelihood Composition for Multi-modal Language Models, which subtracts text-only responses to force the model to rely on visual evidence.
- Similarly, Zezhong Lv et al. apply this concept to video in Counterfactual Cross-modality Reasoning for Weakly Supervised Video Moment Localization, rectifying predictions by subtracting language-only scores to ensure reliance on video cues.
- Le Yu et al. formalize this as a causal graph in Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs, generating a vision-favored and a text-favored prediction path to filter out text-biased hallucinations.
- Yuqing Lei et al. incorporate a similar penalty in Selective Seeing: Context-Aware Attention Interventions for Mitigating Hallucinations in Large Vision-Language Models to counteract the model's inherent tendency to default to language priors.
2. Visual Distortion and Noise Baselines
Completely removing the image can sometimes cause the model's text generation to lose coherence. An alternative is to provide a corrupted or noisy version of the image. This preserves the global context (the model still knows it is looking at a kitchen) but destroys the fine-grained visual details needed to confirm specific objects.
- Sicong Leng et al. pioneered Visual Contrastive Decoding (VCD) in Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding, showing that subtracting the output distribution of a distorted, noisy image cancels out shared statistical biases.
- Xiaoyu Liang et al. build on this with Re-Balancing Contrastive Decoding (RBD) in Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding, injecting noise into visual inputs during inference to isolate and penalize internal textual bias.
- Zhangchen Xu et al. apply this distortion strategy to safety contexts in Steering Multimodal Large Language Models Decoding for Context-Aware Safety, using Gaussian noise to obscure images and isolate tokens that depend strictly on fine-grained visual details.
3. Attention-Guided and Distractor Baselines
Instead of applying uniform noise, some methods look inside the model's attention layers to identify "distractor" tokens—image patches that represent background noise or uninformative regions that cause the model to lose focus.
- Xuan Gong et al. propose DAMRO in DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination, which calculates a negative distribution derived solely from identified background distractor tokens, forcing the model to prioritize object-level features.
- David Wan et al. introduce Contrastive Region Guidance (CRG) in Contrastive Region Guidance: Improving Grounding in Vision-Language Models Without Training, contrasting predictions from an original image with a version where specific regions of interest are masked to factor out internal spatial biases.
- Fushuo Huo et al. develop Self-Introspective Decoding (SID) in Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models, which refines the output by identifying and subtracting logits generated from low-importance visual tokens that create spurious associations.
- Bin Li et al. present Instruction-Aligned Visual Attention (IAVA) in Instruction-Aligned Visual Attention for Mitigating Hallucinations in Large Vision-Language Models, generating a negative logit distribution from a masked image containing only query-irrelevant distractor tokens.
- Sangmin Woo et al. introduce AVISC in Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models, which isolates "blind tokens"—image patches that attract massive attention but contain zero task-relevant information—and subtracts their influence.
4. Structural, Multi-scale, and External Baselines
Other approaches manipulate the structure of the visual input or the model architecture itself to create the contrastive gap.
- Resolution and Granularity: Xianwei Zhuang et al. organize visual information into a hierarchical tree in Game on Tree: Visual Hallucination Mitigation via Coarse-to-Fine View Tree and Game Theory, using Shapley values to contrast highly relevant visual granularities against less relevant ones. Woohyeon Park et al. implement SECOND in SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding, contrasting low-resolution processing stages against high-resolution stages to filter out linguistic biases.
- Internal Model Layers: Xinhao Xu et al. introduce iTaD in Mitigating Hallucinations in Multi-modal Large Language Models via Image Token Attention-Guided Decoding, performing inter-layer contrastive decoding by subtracting the log-probabilities of a less-visually-informed intermediate layer from the final layer's output.
- Generative Reconstruction: Junsuk Choe et al. propose ConVis in ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models. They use a text-to-image model to visualize the MLLM's initial caption and then contrast the token distributions of the original image against the reconstructed image to penalize ungrounded tokens.
- Retrieval: Dingchen Yang et al. introduce Retrieval Contrastive Decoding (RCD) in Retrieve-then-compare mitigates visual hallucination in multi-modal large language models, retrieving similar reference images and subtracting the common hallucinatory patterns they induce to isolate the unique details of the target image.
Advanced Extensions: Amplification and Training
While the core of contrastive decoding is subtraction, researchers have also explored complementary mechanisms to further widen the gap between the positive and negative paths.
Image Amplification
Instead of just penalizing the negative path, some methods actively boost the positive path. Lanyun Zhu et al. introduce Image-Biased Decoding (IBD) in IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding, which calculates the difference between a standard model's distribution and a version where visual attention is artificially amplified, prioritizing tokens strongly supported by visual features.
Dynamic Reflection
Rather than applying a static subtraction factor ($\alpha$), the decoding process can adapt dynamically based on how much the model is currently hallucinating. Maying Shen et al. propose GACD in Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection, using gradients at inference time to measure how much the model is listening to image vs. text tokens, dynamically rebalancing the contrastive weights to prevent co-occurrence bias.
Moving from Inference to Training
The intuition of contrastive decoding can also be backpropagated into the model's weights during data generation. Ji Ma et al. introduce the C³L framework in C3L: Content Correlated Vision-Language Instruction Tuning Data Generation via Contrastive Learning. They identify instances where a model relies too heavily on language priors and use them as negative examples in a contrastive learning framework, explicitly teaching the model to distinguish between visually grounded responses and generic linguistic knowledge.
Why It Works: A Summary
Visual contrastive decoding works because hallucinations are highly predictable from text context alone, whereas true visual facts require the image.
By setting up a mathematical tug-of-war between a fully informed model and a visually impaired baseline, logit subtraction acts as a real-time filter. It strips away the generic, statistically plausible fluff that language models love to guess, leaving behind a crisp, verified output that is strictly anchored in the visual reality of the scene.
Key Foundational Work
Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD) (2024) — contributed 2 concept elements to this direction.
Unleashing the Potentials of Likelihood Composition for Multi-modal Language Models (2024) — contributed 2 concept elements to this direction.
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2024) — contributed 2 concept elements to this direction.