Vision-language models, meaning language models that read an image and write about it, often name objects that are not in the picture. They do this because the text half of the model has strong habits: a kitchen "should" have a sink. The most reliable training-free fix is contrastive decoding. It runs the model a second time with a degraded or missing image and subtracts that "blind" opinion from the real one. The cost is that every generated token now takes two or more full passes, so latency roughly doubles. A separate line of work speeds these models up with speculative decoding. A small, fast "draft" model guesses several tokens ahead, and the large model checks all of them in a single parallel pass. Surprisingly, the draft model can ignore the image entirely and still yield a 2.37x memory-bound speedup, because most tokens in a caption are linguistic scaffolding that text habits predict well (On Speculative Decoding for Multimodal Large Language Models).
This note starts from one observation. An image-blind drafter is a small model of exactly the language habits that cause hallucination. Contrastive decoding pays for an extra full pass to estimate those habits. Speculative decoding has already computed them, for free, as the drafter's probabilities. We propose to use the drafter's distribution as the contrastive "negative" inside the large model's verification step. The result is a single verification pass that both accelerates generation and rejects tokens that only the language habit wanted. If this works, hallucination suppression becomes a by-product of a speedup method rather than a tax on it.
Core Claim
Let $v$ be the image, $x$ the text generated so far, $p_T(y \mid v, x)$ the target model's next-token distribution, and $q(y \mid x)$ an image-blind drafter's distribution. We verify drafted tokens against a contrastive target:
$$ p_c(y) \propto \frac{p_T(y \mid v,x)^{1+\alpha}}{q(y \mid x)^{\alpha}}, \quad y \in \mathcal{V}_\beta = \{ y : p_T(y \mid v,x) \ge \beta \max_{y'} p_T(y' \mid v,x) \}. $$
Here $\alpha \ge 0$ sets the contrast strength. The set $\mathcal{V}_\beta$ is the usual plausibility cutoff from visual contrastive decoding, which prevents rewarding tokens the target itself finds unlikely. A drafted token is accepted with probability $\min(1, p_c(y)/q(y))$. Rejected positions are resampled from the normalized residual $\max(0, p_c - q)$.
By the standard speculative-sampling argument, the output is distributed exactly as $p_c$. The procedure is therefore an exact contrastive decoder, but it needs only one target forward pass per drafted block and no distorted-image pass.
The mechanism works as follows. On scaffolding tokens, $p_T$ and $q$ agree, the contrast is near-neutral, and acceptance stays high. On a prior-driven object word, $q$ is high while $p_T$ is only moderately high, so $p_c/q$ is small and the token is likely rejected. On a truly visual token, $p_T \gg q$, so the target overrides the draft.
The novelty is making the drafter serve as a calibrated language-prior negative. We train it by distilling the target run with the image removed, so that $q(y \mid x) \approx p_T(y \mid \varnothing, x)$. Without this calibration, the contrast would measure "small model versus large model," which is the classic text contrastive-decoding signal, rather than "blind versus sighted."
Relation to Prior Work
Several papers motivate the gap. Hallucinated objects track statistical co-occurrence rather than failures of the image encoder alone (Evaluating Object Hallucination in Large Vision-Language Models). Long responses drift further toward text-driven fabrication as attention shifts from image tokens to generated text (Why LVLMs Are More Prone to Hallucinations in Longer Responses). Both findings say the harmful signal is the text prior, which an image-blind model captures directly.
The contrastive decoders we inherit from are VCD (Leng et al.), which subtracts a noisy-image pass, and RITUAL (Woo et al.), which contrasts against transformed images. Both pay at least one extra full pass per token. We keep their plausibility cutoff but replace the negative pass with the drafter.
The closest method-supplying work is text-only. Speculative Contrastive Decoding lets a small amateur model act as both drafter and contrastive negative, but for text language models with a generic small model as the amateur. Decoupled Contrastive Decoding via Expert-Aligned Drafting argues against that coupling. It shows the contrastive signal is small for 81.1% of tokens and that drafter error is amplified by $(1+\alpha)$ when drafts come from a contrastive distribution.
DCD constrains our design, and we address it directly. We draft from $q$ itself, not from a contrastive distribution, so the amplification DCD describes does not enter the draft. The contrast enters only in the acceptance ratio. Whether acceptance still collapses is an empirical question that our ablations test. The multimodal case also differs from DCD's setting in a way that favors coupling: the blind drafter's error is concentrated on visual tokens, which is exactly where we want the contrast to act.
Multimodal speculative decoders supply the acceleration machinery. ViSpec gives the drafter a compressed image. SpecVLM uses elastic visual compression. MSD decouples visual and text inputs to the drafter. FastVLM self-drafts from early layers. All of these are lossless and therefore leave hallucination exactly where the target puts it. We deliberately change the output distribution.
Two efficient anti-hallucination methods are the real competitors. VASparse obtains a cheap vision-agnostic distribution at the embeddings and language head and prunes visual tokens. RUDDER injects a single-pass steering vector while keeping 96.0% throughput. Neither makes generation faster than plain greedy decoding by a speculative margin, and neither uses a learned, calibrated prior model as the negative.
Research Program
The first component is the drafter. On LLaVA-1.5-7B and 13B, we train a roughly 115M-parameter text-only drafter, following the scale in Gagrani et al., by logit distillation from the target run with the image tokens removed on the target's instruction data. We compare it against a drafter distilled from the target with the image present but not given the image. That second model is the Gagrani-style drafter. It learns to guess the sighted target's output from text alone, so it partly absorbs image-conditional regularities and is a less clean negative. This comparison is central: it tests whether calibrating the negative to "blind target" is what makes the contrast useful.
The second component is the decoder: the acceptance rule above with a single $(\alpha, \beta)$ pair. We choose that pair on a held-out COCO split and freeze it across models. This directly addresses the hyperparameter sensitivity that SHARP and RUDDER report.
The third component is a diagnostic study that could falsify the premise. We run vanilla lossless speculative decoding with the blind drafter and log, for every object that CHAIR marks as hallucinated, whether the drafter proposed it and the target accepted it. If most hallucinated objects are not drafter-proposed, the method's lever is weak. In that case we would report that negative result rather than tune around it.
Baselines
The naive reference is greedy decoding on each target. Its reported LLaVA-1.5 figures are CHAIR$_S$ 52.8 (SHARP), instance-level CHAIR 7.22% (VASparse), and POPE accuracy 84.13% (RITUAL). All re-runs use the same prompts and a 512-token cap, so we recompute these baselines ourselves rather than mixing paper numbers.
The standard contrastive baseline is VCD with a distorted-image second pass. On LLaVA-Next it costs 13.97s per sample versus 7.04s for SHARP (SHARP). Our method should roughly match VCD's CHAIR reduction at a small fraction of its wall-clock time.
The lossless speed reference is the same blind drafter with exact verification, which is our method at $\alpha=0$. Gagrani et al. report up to a 2.37x memory-bound speedup for this setup. MSD reports 2.46x on LLaVA-1.5-13B and 2.29x on 7B (MSD). Beating this baseline means holding at least about 80% of its measured speedup while cutting CHAIR.
The strongest efficient anti-hallucination competitors are VASparse, RUDDER, and SHARP. VASparse reports 7.22% to 5.82% instance CHAIR. RUDDER reports a 24.4% relative CHAIR$_S$ reduction at 96.0% throughput. SHARP reports CHAIR$_S$ 52.8 to 34.8 with steering at layer 10. We run each with its published hyperparameters, on the same hardware and batch size 1, and give it the same held-out tuning budget we use for $(\alpha,\beta)$.
The most dangerous baseline is a stack: lossless speculative decoding with the same blind drafter, combined with RUDDER or SHARP steering applied to the target. Speculation stays lossless with respect to the steered target, so this stack also delivers speed and hallucination reduction together. We claim a win only if our method matches the stack's CHAIR at equal or better tokens per second, or dominates the stack on the speed-versus-hallucination curve.
The key ablations isolate our ingredients. The first swaps the calibrated blind drafter for the uncalibrated one. The second draws the negative from the drafter but drafts with an image-aware EAGLE-style drafter; this is the DCD-style decoupled variant and costs one drafter pass more. The third replaces $q$ with a generic small language model to isolate "blindness" from "smallness."
Why not just…?
Why not just stack a single-pass steering method like RUDDER on top of ordinary speculative decoding? This is a real threat, and it is the stacked baseline above. RUDDER already achieves a 24.4% CHAIR$_S$ reduction at 96.0% throughput (RUDDER), and lossless speculation adds speed on top. Our argument is not that the stack fails. It is that the stack requires per-architecture choice of the layer and strength, whereas our negative is learned once per model and our knobs are frozen across models. If the stack wins on the Pareto curve, the honest conclusion is that the drafter-as-negative idea is only a calibration convenience. The experiments are designed to report that outcome if it occurs.
Why not just use VASparse, which already gets a vision-agnostic distribution cheaply? VASparse's negative is computed at the embeddings and language head, so it is a crude proxy for what the full decoder would say without the image. It also does not reduce the number of sequential target passes the way speculation does. Its reported gain is 7.22% to 5.82% instance CHAIR (VASparse), and its speed claims are relative to HALC rather than to speculative decoders. Head-to-head wall-clock and CHAIR on identical hardware will settle whether a distilled blind model is a better negative than VASparse's head-level shortcut.
Why not just batch the VCD passes? Decoding is memory-bandwidth bound, so a second image-conditioned pass means a second KV cache streamed from memory at every token (Pranab Sahoo et al.). The measured VCD-versus-SHARP gap of 13.97s versus 7.04s shows that batching has not hidden this cost in practice. We nonetheless include batched VCD in the timing table.
Why not just fine-tune the target with preference data? Preference fine-tuning is orthogonal and costly. The file's evidence that training-free methods can beat reinforcement-learning-trained ones is limited to video, where SEASON beats ArrowRL by 5.6% on VidHalluc (SEASON). We treat preference tuning as complementary rather than as something we have shown to be worse.
Proposed Experiments and Decisive Checks
We evaluate on CHAIR$_S$ and CHAIR$_I$ with 500 MSCOCO captioning prompts, on POPE (random, popular, and adversarial splits), and on MME perception. The main quality guard is MM-Vet, where FastVLM reports 27.8 versus 28.0 as a reference for acceptable drift. The latency measures are tokens per second, per-sample wall-clock time, and mean accepted length, measured on LLaVA-1.5-7B and 13B at batch size 1. POPE answers are short, and FastVLM notes that speculation gives little speedup on short outputs. We therefore report POPE for hallucination only and base speed claims on captioning.
Four checks are decisive. The first is the diagnostic share of CHAIR hallucinations that are drafter-proposed; if it is low, the idea is weak. The second is whether CHAIR$_S$ falls by at least RUDDER's 24.4% relative reduction while speedup stays above 1.8x on 7B. The third is whether the calibrated blind drafter beats the uncalibrated drafter and the generic small language model as a negative; if it does not, the "blindness" story is wrong. The fourth is whether acceptance length degrades gracefully as $\alpha$ grows, which is the empirical test of DCD's warning. A further check examines long captions, split by output position, to see whether the gain concentrates late in generation, where the "contextual trap" predicts the text prior dominates.
References
- On Speculative Decoding for Multimodal Large Language Models
- Evaluating Object Hallucination in Large Vision-Language Models
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
- RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs
- Speculative Contrastive Decoding
- Decoupled Contrastive Decoding via Expert-Aligned Drafting
- ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding
- SpecVLM: Fast Speculative Decoding in Vision-Language Models
- Speculative Decoding Reimagined for Multimodal Large Language Models
- FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference
- VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification
- Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision-Language Models
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models
- SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decoding