Speaker-Smoothed kNN Speaker Adaptation for End-to-End ASR

Shaojun Li, Daimeng Wei, Jiaxin Guo, Zongyao Li, Zhanglin Wu, Zhiqiang Rao, Yuanchang Luo, Xianghui He, Hao Yang

Abstract

End-to-End (E2E) Automatic Speech Recognition (ASR) systems often struggle with vocal characteristic mismatches when encountering speakers or dialects not well-represented in their training data. Traditional fine-tuning methods frequently suffer from data sparsity and catastrophic forgetting. This paper presents Speaker-Smoothed kNN, a novel adaptation framework that integrates k-Nearest Neighbors (kNN) retrieval with speaker embeddings (x-vectors). By dynamically adjusting interpolation weights and temperature parameters based on speaker similarity, the proposed method achieves significant error rate reductions in both single-speaker and multi-speaker scenarios without modifying the base model's weights.

Introduction

While modern E2E ASR models have achieved remarkable performance, their generalization to new speakers remains a significant hurdle. When a speaker's voice or accent deviates from the training distribution, performance typically degrades. This challenge, known as vocal characteristic mismatch, is traditionally addressed through fine-tuning. However, fine-tuning on limited adaptation data (often just a few minutes of audio) leads to two primary issues: overfitting due to data sparsity and catastrophic forgetting, where the model loses its ability to recognize general speech or previous speakers.

To overcome these limitations, we explore a non-parametric approach using kNN-augmented models. These models allow for "hot-swappable" adaptation by retrieving relevant acoustic contexts from a datastore during inference, rather than retraining the model itself.

Methodology

The kNN-Augmented Framework

In a standard kNN-augmented ASR system, the model does not rely solely on its internal neural weights. Instead, it queries a datastore containing pairs of acoustic contexts and their corresponding tokens. During decoding, the system retrieves the $k$ most similar examples and creates a probability distribution. This distribution is then blended with the base ASR model's output using a fixed interpolation weight ($\lambda$) and a temperature parameter ($T$).

Speaker-Smoothed Innovation

The core limitation of standard kNN is that fixed parameters ($\lambda, T$) cannot account for varying data density across different speakers and dialects. We introduce the Speaker-Smoothed Module to make these parameters dynamic:

  1. X-vector Integration: We utilize x-vectors—embeddings that capture unique speaker characteristics—to quantify the similarity between the current speaker and the examples stored in the datastore.
  2. Dynamic Estimators: Instead of global constants, we employ two small neural networks to calculate $\lambda_s$ and $T_s$ for every token. These estimators process retrieval distances, the count of distinct tokens in the neighborhood, and x-vector similarities.
  3. On-the-fly Adaptation: This mechanism allows the system to adjust its reliance on the datastore in real-time. If the retrieved neighbors are acoustically similar but come from a speaker with a vastly different x-vector, the system dynamically lowers $\lambda_s$, placing more trust in the base ASR model.

Experimental Setup

Datasets

The research utilized two primary Mandarin Chinese corpora:

  • KeSpeech: A large-scale dataset (895 hours) covering eight subdialects, used for training the base model and constructing the kNN datastore.
  • MagicData: Specifically the Sichuan and Zhengzhou subsets, providing out-of-domain data to test the system's robustness in single-speaker adaptation tasks.

Results and Discussion

Performance Gains

The Speaker-Smoothed kNN method demonstrated superior accuracy across multiple settings. In "All-Domain" evaluations, the method achieved state-of-the-art results, reducing the Character Error Rate (CER) by 12.35% for single speakers. The most significant improvement was observed in multi-speaker scenarios, where the CER dropped by 24.68% compared to the baseline.

Stability and Robustness

A critical advantage of this approach is its stability. While standard fine-tuning improved performance for a specific target speaker, it caused the CER to spike (from 36.71 to 52.31) when the speaker changed. In contrast, the Speaker-Smoothed kNN maintained or improved performance across all speakers, as the base model weights remained untouched.

Dialect Overlap and Filtering

Analysis of the acoustic space revealed that different dialects often produce overlapping acoustic representations for distinct tokens, creating "noise" in standard kNN retrieval. By using x-vectors as a filter, the Speaker-Smoothed module effectively ignores these misleading neighbors, ensuring that the retrieval process is grounded in speaker-relevant context.

Conclusion and Future Work

Speaker-Smoothed kNN provides a robust, efficient, and stable solution for ASR speaker adaptation. It effectively mitigates the risks of overfitting and catastrophic forgetting inherent in fine-tuning. While the method introduces some computational overhead, the inference speed remains at approximately 87.1% of the base model, making it viable for many real-world applications.

Future research will focus on optimizing datastore scaling to handle billions of tokens and exploring similar smoothing techniques for other sequence-to-sequence tasks, such as machine translation.

Summary updated

BibTeX
@inproceedings{li2024speaker,
  author = {Shaojun Li and Daimeng Wei and Jiaxin Guo and Zongyao Li and Zhanglin Wu and Zhiqiang Rao and Yuanchang Luo and Xianghui He and Hao Yang},
  title = {{Speaker-Smoothed kNN Speaker Adaptation for End-to-End ASR}},
  booktitle = {Interspeech},
  year = {2024},
  doi = {10.21437/interspeech.2024-1873},
}
Figure