Introduction
OpenAI’s Whisper has set a high bar for Automatic Speech Recognition (ASR), demonstrating remarkable robustness across various languages and conditions. However, like most parametric models, it faces significant hurdles when encountering "out-of-distribution" (OOD) data. Whether it is a specialized medical domain, a rare regional accent, or a speaker with unique prosody, Whisper’s performance can degrade. Traditionally, the solution has been fine-tuning, but this approach introduces the risk of catastrophic forgetting and requires substantial computational resources to store and maintain separate model weights for every niche application.
In this work, we explore a non-parametric alternative: token-level $k$-nearest neighbor ($k$NN) retrieval. By allowing Whisper to "consult" a datastore of previously seen examples during inference, we enable the model to adapt to new domains and speakers on the fly without changing its underlying parameters.
Methodology: Token-Level $k$NN Retrieval
Our approach transforms Whisper into a retrieval-augmented model. Instead of relying solely on its internal weights, the model performs an "open-book exam" during the decoding process.
The Datastore
First, we construct a datastore by passing a set of reference audio files through the Whisper encoder. We collect the model's internal hidden states (keys) and the corresponding ground-truth tokens (values). These pairs are stored in an optimized index using the FAISS library for efficient similarity searching.
Inference-Time Retrieval and Interpolation
During transcription, at each decoding step, Whisper generates a hidden state for the current frame. This state serves as a query to find the $k$ most similar entries in the datastore. We then calculate a probability distribution over the retrieved tokens and blend it with Whisper’s original output distribution using an interpolation weight, $\lambda$. This allows the model to lean on the datastore when its internal confidence is low or when the retrieved context is highly relevant.
Experimental Evaluation
We evaluated our $k$NN-augmented Whisper across four diverse datasets: VoxPopuli (English parliamentary speeches), LibriSpeech (clean English audiobooks), CommonVoice 18.0 (Dutch), and RixVox (Swedish parliamentary data).
As shown in Table 1, the $k$NN mechanism provides substantial gains on more challenging datasets. For instance, on the VoxPopuli dataset, the Word Error Rate (WER) for the large-v3 model dropped from 8.13 to 7.31. Interestingly, we observed a "clean data ceiling" on LibriSpeech; because Whisper is already highly optimized for clean English audiobooks, the retrieval mechanism introduced more noise than signal, slightly increasing the WER. This confirms that $k$NN is most effective when the base model is operating outside its primary training distribution.
Key Insights: Small $k$ and Bias Mitigation
Our analysis revealed several critical nuances in applying $k$NN to ASR. Unlike in Natural Language Processing, where large neighbor counts (high $k$) are often beneficial, we found that a very small $k$ (e.g., $k=4$) performed best for speech. Larger values tended to introduce irrelevant tokens that confused the decoder. Furthermore, the hidden states captured not just the current token but also predictive information about the subsequent word, facilitating a smoother transcription "flow."
One of the most compelling findings was the model's ability to mitigate inherent performance biases. In our Dutch CommonVoice experiments, we analyzed performance across gender and regional accents.
As detailed in Table 2, $k$NN retrieval provided a larger relative improvement for female speakers compared to male speakers. It also significantly closed the performance gap for Belgian Dutch speakers, who typically experience higher error rates than those from the Netherlands. This suggests that $k$NN can effectively "specialize" the model to underrepresented speech patterns present in the datastore.
Speaker Adaptation and Efficiency
On the Swedish RixVox dataset, we tested the model's capacity for speaker adaptation. Using a general datastore reduced the mean WER from 16.09 to 13.87. We also experimented with "personalized" datastores containing only a specific speaker's data. While the full datastore offered the highest accuracy, personalized datastores were remarkably efficient, reducing the transcription time for a test file from 75 seconds to just 7 seconds due to the smaller search space.
Conclusion and Future Work
Integrating $k$NN retrieval into Whisper offers a powerful, flexible path for domain and speaker adaptation without the overhead of fine-tuning. While the approach adds some computational latency and requires careful hyperparameter tuning (specifically for $\lambda$), its ability to mitigate bias and improve OOD performance is significant. Future research will investigate the impact on morphologically rich languages and explore ways to further reduce the search latency for real-time applications.
Summary updated