Blind Drafters as Free Hallucination Probes in Audio-Visual Language Models

Audio-visual language models, which answer questions about videos with sound, often ignore their senses. They hear speech but report a sizzling pan because a pan is on screen, or they call a silent dog "barking." The best training-free fix contrasts the model's normal prediction against predictions made with audio or video hidden, and suppresses whatever the model would have said anyway. That fix costs up to four full forward passes per generated token, on models that are already slow because they emit one token at a time. The main speed-up method for such models, speculative decoding, runs a tiny text-only "draft" model to guess several tokens and has the big model check them all in one pass. That draft model is blind by construction. It is therefore already computing, for free, the prediction that contrastive decoding pays extra passes to obtain.

This note proposes to merge the two methods. We use the blind drafter's distribution as the "what would you say without senses" reference, and we fold the contrast into the speculative accept/reject step. Drafted tokens that the grounded model endorses mostly because a blind model would also say them are then rejected more often. The target is hallucination suppression at speculative-decoding speed rather than at a multiple of baseline cost. A cheap signal, agreement between a confident target and the blind drafter on a content word, would decide when to spend an expensive masked-modality check. That signal targets exactly the "confidently wrong" tokens that entropy-based gating misses.

Core Claim

Speculative decoding runs in two stages. A small model $q$ proposes a block of tokens. The large target model $p_T$ then accepts each proposed token $y$ with probability $\min(1, p(y)/q(y))$ and resamples on rejection. This procedure is exact for whatever distribution $p$ is placed in the numerator. On Speculative Decoding for Multimodal Large Language Models shows that an image-blind 115M drafter works well for LLaVA-7B. It accepts 2 to 3 tokens per cycle and reaches up to 2.37x memory-bound speedup, because most generated tokens are linguistic glue.

We replace the numerator with a contrastive distribution built from the target and the blind drafter itself:

$$p_\alpha(y) \propto p_T(y \mid a, v, x)^{1+\alpha}\, q(y \mid x)^{-\alpha}\quad \text{for } y \in V_t,$$

Here $a, v, x$ are the audio, video and text context, $\alpha \ge 0$ sets the contrast strength, and $V_t$ is the set of tokens whose target probability is at least a fraction $\beta$ of the top token's.

With this numerator, the verification pass is the contrastive step. It adds no extra target passes beyond ordinary speculative decoding, and it samples exactly from $p_\alpha$. The paired claim is diagnostic. The per-token log ratio $\log p_T(y)/q(y)$ measures how much the senses moved the prediction. When the target is confident and this ratio is near zero on a content word such as a sound event or an object, the target is agreeing with a blind prior. That is the signature of the late-layer "prior dominance" that Compositional Failure in Audio-Visual LLMs found through layer-wise probing.

This corrects one intuition in the research file. The file suggests that blind drafting might itself amplify hallucination. Under exact rejection sampling it cannot: the output distribution equals the target's regardless of the drafter. Amplification is only possible under the lenient or greedy acceptance rules often used in practice. Our first experiment tests this directly rather than assuming it.

Research Program

Build the drafter. We first build blind drafters for two audio-visual models used in prior hallucination work, VideoLLaMA2 and video-SALMONN. Following the ft-llava-text recipe, each drafter is a small language model that shares the target's tokenizer and is fine-tuned on the target's own text outputs.

Measure raw speculative decoding. Next we measure plain speculative decoding on AVHBench and DAVE. AVHBench contains audio-swapped videos and tests audio-driven and video-driven hallucination. DAVE separates synchronization, sound-absence detection and sound discrimination. We record acceptance rates split by conflict versus consistent items and by content versus function tokens, under both exact and greedy acceptance. This answers two questions. Does acceptance itself flag sensory-dependent positions? Does greedy acceptance lock in hallucinations?

Contrastive verification. Third, we implement the contrastive verification rule above and sweep $\alpha$ and $\beta$.

Budgeted escalation. Fourth, we add a budgeted escalation rule. When the cheap test flags a content token with high target confidence and small sensory log ratio, we run one AVCD-style masked-modality target pass for that position only.

The organizing lens for this last component comes from guessing-based decoding with abandonment. Ensemble-Tight Second-Order Asymptotics and Exponents for Guessing-Based Decoding with Abandonment proves that a search decoder with a capped query budget has error exponent $\min\{E_r(R), E_a(r)\}$. Reliability is limited by whichever is worse: the code itself, or the errors introduced by stopping early. We do not claim that theorem transfers to neural decoders. We borrow its structure as an evaluation principle. Total hallucination splits into errors the fully checked model would still make and errors created by skipping checks. We therefore report hallucination rate as a function of target forward passes per token, and locate the budget at which skipping stops being the dominant term.

Baselines

Naive references. The do-nothing reference is the base model with ordinary decoding. Compositional Failure reports 51.7% accuracy for VideoLLaMA 2-7B-AV on AVHBench's scored exact-string Yes/No subset. DAVE reports roughly 31.3% for Video-LLaMA-2. The latency reference is plain exact speculative decoding with the same blind drafter. For audio-visual models, no reported result was found on Lacuna for speculative decoding. The only reported figure is the 2.37x speedup on LLaVA-7B. We reproduce it on our targets, with block size 5 as in that paper.

Strongest hallucination competitor. This is AVCD. AVCD masks the most-attended audio or video tokens, subtracts the resulting logits from clean logits, and uses entropy-guided gating to skip contrast when the model is confident. The authors report accuracy gains on AVHBench of about 2% for VideoLLaMA2 and 7% for video-SALMONN, with gating cutting latency by over 30%. It is the strongest competitor because it targets the same failure, without training, on the same models. We will run the authors' method with gating on and off, on identical prompts, decoding settings and hardware. Each method gets the same hyperparameter search budget on a held-out AVHBench split. The bar for beating AVCD is to match or exceed its AVHBench accuracy and its captioning hallucination rate while using fewer target forward passes per token. If we only match its accuracy at lower cost, that alone would be a useful result.

Output-layer bias correction. This is the output-layer modification studied in The Mirage of Performance Gains, which reached 89.6% on POPE MSCOCO-Random versus 88.6% for visual contrastive decoding. It stands in for "just shift the Yes/No prior." We tune it to match our method's Yes-rate so that any remaining gap reflects grounding rather than rebalancing.

Key ablation. The ablation that isolates our ingredient replaces the drafter reference $q$ with the target model itself run with audio and video removed. This costs one extra full pass. It tells us whether a tiny blind model is an adequate stand-in for the target's own prior.

Why not just…?

Why not just use AVCD with entropy gating? Its cost scales with the number of masked passes, and its own authors concede that entropy gating cannot catch confidently wrong tokens. Our escalation rule is designed for those tokens. The pass-budget curve against AVCD with gating directly tests whether our gate finds more hallucinations per extra pass.

Why not just fine-tune the model for better alignment? Compositional Failure tested three alignment tuning strategies on VideoLLaMA 2-7B-AV: audio-conditioned concatenation, timestamp interleaving, and modality dropout. None exceeded the base 51.7%. Probing showed they shift response bias without moving the late layer where the model commits to its prior. LoRA fine-tuning on AVHBench's own training split did lift Video-LLaMA from about 50% to 83.9% in AVHBench. That gain is in-distribution. We will check whether it transfers to DAVE, and whether our decoding method adds on top of a fine-tuned model.

Why not just debias the Yes/No output? This is a real threat on binary benchmarks. The Mirage of Performance Gains shows that contrastive gains there are largely unidirectional answer shifts. It also shows that the plausibility cutoff $\beta$ quietly turns sampling into greedy search. Our method inherits both risks, so the decisive evidence cannot come from Yes/No accuracy. We report Yes-rate-matched accuracy, AVHBench captioning hallucination, and DAVE sound-absence detection, where Gemini 1.5 Pro fell to about 1%. We also ablate $\beta$ to zero. If gains vanish once bias is matched, the idea fails and we will say so.

Why not just use the target's own blind pass as the reference? This alternative may simply work better, since the 115M drafter's prior differs from the target's in fluency as well as in grounding. The ablation above settles it. If the target-blind reference wins by a wide margin, the contribution shrinks to the escalation gate and the diagnostic.

Decisive Checks

Acceptance as a detector. On conflict items, acceptance of drafted content tokens should fall relative to consistent items under exact acceptance. If it does not, the blind drafter carries no sensory signal and the approach is unsupported.

Fixed-compute comparison. At an equal number of target passes per token, contrastive verification should beat AVCD with gating on bias-matched AVHBench accuracy and on captioning hallucination. General ability should be preserved on MSVD-QA and MUSIC-AVQA, where AVCD reports roughly a 1% gain on MSVD-QA.

Budget curve. The hallucination-versus-budget curve should show a knee. Near that knee, escalating only flagged confident-agreement tokens should recover most of the benefit of checking every token. That would demonstrate an error floor from skipping that a well-chosen cheap test can drive below the model's own error.