Presented at NAACL 2025, this research explores a non-parametric method to enhance OpenAI’s Whisper model without the computational burden of traditional fine-tuning. By integrating a $k$-nearest neighbor ($k$NN) retrieval mechanism, the authors demonstrate a path toward more inclusive, adaptable, and accurate speech recognition that can be implemented at inference time.
The Challenge of Adaptation in ASR
Large-scale Automatic Speech Recognition (ASR) models like Whisper are powerful but often struggle with "out-of-domain" data, such as medical jargon, rare regional accents, or unique individual speech patterns. While fine-tuning is the standard solution, it suffers from two major drawbacks: catastrophic forgetting, where the model loses general knowledge while learning new data, and high computational costs associated with retraining massive architectures.
The authors propose a solution for inference-time adaptation: making a model better at recognizing specific speakers or dialects without altering the underlying model weights.
Methodology: Token-Level $k$NN Retrieval
The researchers employ a non-parametric approach using token-level $k$NN search. The process involves three main steps:
- The Datastore: A repository is created containing the hidden states (mathematical representations) of the model paired with their corresponding text tokens.
- Retrieval: During transcription, the model compares its current hidden state against the datastore to find the $k$ most similar states.
- Interpolation: The model’s original prediction is combined with the distribution of tokens from the $k$NN search. An interpolation parameter ($\lambda$) determines the weight given to the retrieved data versus the model's internal logic.
This allows the model to effectively "consult a reference book" during the transcription process, leveraging specific context that may not be well-represented in its static weights.
Performance Across Diverse Datasets
The study evaluated this approach across several benchmarks, including LibriSpeech (English), CommonVoice (Dutch), and RixVox (Swedish). The results consistently showed that $k$NN retrieval improves performance, particularly on more challenging datasets.
As shown in the table above, the Swedish RixVox dataset saw a significant reduction in Word Error Rate (WER) from 16.71 to 14.41 when using the large-v3 model. The hidden states were found to capture significant context; retrieved neighbors often shared identical preceding and following words, aiding the model in predicting sentence structure more accurately.
Mitigating Performance Bias
One of the most significant findings involves the mitigation of demographic biases. Whisper naturally performs better on some groups than others, but the $k$NN approach provides a larger relative boost to the groups the model typically struggles with.
Key insights from the bias analysis include:
- Gender Gap: In Dutch experiments, female speakers saw a WER reduction of 0.56, while male speakers saw a reduction of only 0.03, effectively narrowing the performance gap.
- Accent Adaptation: Belgian Dutch speakers, who typically face higher error rates than Netherlands Dutch speakers, benefited more from the $k$NN mechanism (a reduction from 5.29 to 4.98).
- Speaker Adaptation: While a massive general datastore provides the best absolute accuracy, the authors found that a personalized datastore for a specific speaker was 10x faster to process while still significantly outperforming the base model. This efficiency makes personalized $k$NN a viable candidate for real-time, on-device applications.
Limitations and Future Directions
While promising, the study notes that its focus was primarily on Indo-European languages (English, Dutch, Swedish). Future work is needed to determine if these benefits translate to different linguistic typologies. Additionally, the researchers suggest that more exhaustive hyperparameter tuning and richer demographic metadata could unlock even greater performance gains and deeper insights into bias mitigation. The ultimate goal remains refining these search mechanisms to be fast enough for seamless use on mobile and edge devices.
Summary updated