TL;DR
Whisper-CD is a training-free, inference-time framework that significantly reduces hallucinations and repetition loops in long-form speech recognition. By contrasting the model's predictions on clean audio against three "negative" perturbed versions (noise, silence, and time-shifts), it steers the decoder toward text actually supported by the audio, achieving up to a 24.3 percentage point reduction in Word Error Rate (WER) while remaining faster than traditional beam search.
What Problem Does This Paper Address?
While OpenAI’s Whisper has revolutionized Automatic Speech Recognition (ASR), it struggles with "long-form" audio (recordings lasting minutes or hours). Because Whisper processes audio in 30-second chunks and often uses the transcript of the previous chunk as context for the next, errors tend to snowball.
The authors identify three primary failure modes:
- Silence-region hallucinations: The model generates "stock phrases" (e.g., "Thank you for watching") during periods of dead air.
- Repetition loops: The model gets stuck repeating the same phrase across segment boundaries.
- Content skips: The model ignores portions of the spoken content.
Existing solutions often require retraining the model or adding complex auxiliary modules. The challenge was to find a "drop-in" fix that works at inference time without sacrificing the speed or the inherent capabilities of the pre-trained model.
Methods
The researchers introduce Whisper-CD (Contrastive Decoding). The core idea is to adjust the probability of the next word (token) by comparing the "positive" logits (the model's raw output from clean audio) with "negative" logits derived from intentionally degraded audio.
The Multi-Negative Approach
Instead of using a single "bad" version of the audio, Whisper-CD uses three acoustically motivated perturbations to capture different error patterns:
- Gaussian Noise Injection: Adds white noise to the signal. This weakens fine-grained phonetic details, highlighting tokens the model might be "guessing" based on coarse structure.
- Silence Signal: Replaces the audio with an all-zero waveform. This forces the model to reveal its internal biases—the phrases it wants to say even when there is no sound at all.
- Audio Temporal Shift: Shifts the audio waveform forward in time (by 7 seconds). This creates a mismatch between the previous text context and the current audio, helping the model identify when it is relying too heavily on text history rather than the actual sound.
The Objective Function
These three negative signals are combined using a log-sum-exp operator. This allows the system to penalize a token if it appears likely in any of the negative scenarios. A contrastive strength parameter ($\alpha$) controls how aggressively the system "subtracts" these negative tendencies from the final prediction.
Datasets
The authors evaluated Whisper-CD on five diverse English long-form benchmarks:
- CORAAL: Regional African American language (noted for high baseline error rates).
- Earnings22: Transcripts of financial earnings calls.
- VoxPopuli: European Parliament plenary sessions.
- TED-LIUM: TED talk presentations.
- REV-16: A mix of podcasts and interviews.
Key Insights and Ideas
- Context is a Double-Edged Sword: Passing the previous segment's text as a prefix helps maintain consistency, but it is the primary driver of error accumulation. Whisper-CD allows the model to keep using context while providing a "sanity check" against the audio.
- Hallucinations are High-Confidence: One reason hallucinations are hard to fix is that the model predicts them with very high probability. Contrastive decoding works because the model also predicts these hallucinations with high probability when given silence or noise. Subtracting the two reveals what is uniquely supported by the clean audio.
- Efficiency via Batching: Even though the method requires four forward passes (one clean, three perturbed), these can be batched together on modern GPUs. This makes the process much faster than beam search, which requires exploring multiple branching paths.
Key Results
- Massive Accuracy Gains: On the CORAAL dataset using Whisper Large-v3, Whisper-CD reduced the WER from a staggering 208.76% (caused by infinite repetition loops) to 45.77%.
- Superior to Beam Search: Compared to a standard beam search (size 5), Whisper-CD achieved lower WER across the board while being 48% faster in terms of token throughput.
- Robustness: The multi-negative approach proved more effective than any single perturbation strategy, suggesting that different types of hallucinations require different "negative" signals to be caught.
Limitations and Future Work
- Hyperparameter Sensitivity: The optimal contrastive strength ($\alpha$) varies depending on the dataset and the specific Whisper model used. The authors suggest that dynamic $\alpha$ (adjusting the strength per segment or per token) is a promising area for future research.
- Perturbation Diversity: While the three chosen perturbations cover many bases, others like frequency masking or temporal warping could potentially address even more niche failure modes.
- Architecture Scope: The current work focuses on encoder-decoder models. Adapting this to newer "decoder-only" ASR architectures (where audio and text are processed in a single stream) remains an open challenge.
Summary updated