Retrieve and Copy: Scaling ASR Personalization to Large Catalogs

Sai Muralidhar Jayanthi, Devang Kulshreshtha, Saket Dingliwal, Srikanth Ronanki, Sravan Bodapati

Abstract

Personalization in automatic speech recognition (ASR) is critical for accurately transcribing rare words and domain-specific entities. However, traditional attention-based contextual biasing methods struggle to scale beyond a few thousand entities due to computational latency and phonetic confusion. We present "Retrieve and Copy" (RAC), a framework that utilizes Approximate Nearest Neighbor (ANN) search to maintain low latency, alongside a Hard Negative Fine-tuning (HNFT) strategy to improve discriminative power. Our approach enables scaling to catalogs of up to 20,000 items, achieving up to a 6% Word Error Rate reduction (WERR) and a 3.6% absolute improvement in F1-score, while providing at least a 20% inference speedup per acoustic frame.

Introduction

Modern ASR systems often fail on the "long tail" of language—proper names, technical terminology, and niche products that appear infrequently in general training corpora. Contextual biasing addresses this by providing the model with a "catalog" of relevant entities. While effective for small lists, scaling these catalogs to tens of thousands of entries introduces significant challenges: linear increases in computational latency and a higher probability of phonetic confusion between similar-sounding words.

The Scaling Challenge

Current Contextual Adapter (CA) architectures typically compute attention scores across the entire catalog for every audio frame. As the catalog size ($N$) grows, the computational cost $O(N)$ becomes prohibitive for real-time applications. Furthermore, larger catalogs are denser in "phonetic neighbors"—words like "Ruben" and "Rueben" that are acoustically similar. Without specialized training, models often struggle to distinguish these, leading to increased false positives and missed detections.

Proposed Methodology

Retrieve and Copy (RAC)

To address the latency bottleneck, we propose the Retrieve and Copy (RAC) mechanism. Instead of performing exhaustive attention across the entire catalog, we employ an efficient Approximate Nearest Neighbor (ANN) search using the FAISS library.

  1. Retrieval: For each audio frame, the system treats the acoustic features as a query to retrieve the top $k$ (where $k \ll N$, typically $k=10$) most relevant entities from the catalog.
  2. Copy: The attention-based biasing is then applied only to this restricted subset.

This approach transforms the latency profile from $O(N)$ to $O(k)$, ensuring that inference speed remains manageable regardless of catalog size. The researchers found that $k=10$ was the "sweet spot," providing nearly 100% retrieval accuracy while keeping latency low.

Hard Negative Fine-tuning (HNFT)

To mitigate phonetic confusion, we introduce Hard Negative Fine-tuning. Standard training regimes often use random entities as negative examples, which are usually acoustically distinct from the target. HNFT uses k-means clustering to identify phonetic neighbors within the embedding space. During fine-tuning, the model is presented with these "hard negatives," forcing it to learn the subtle acoustic nuances required to differentiate between similar-sounding entities in a large catalog. This bridges the gap between training and real-world inference environments.

Experimental Evaluation

We evaluated our framework on several datasets, including in-house conversational data (First Names, Last Names, and Occupations) and the public VoxPopuli benchmark. We tested catalog sizes ranging from 250 to 20,000 items to measure performance scaling.

Performance and Scalability

Our results demonstrate that RAC maintains high F1-scores even at the 20,000-item limit, where baseline methods typically degrade. Key metrics include:

  • Accuracy: A 6% reduction in Word Error Rate (WERR) and a 3.6% absolute improvement in F1-score compared to strong biasing baselines.
  • Efficiency: A minimum 20% inference speedup per acoustic frame. In specific tests, frame-level latency was reduced from approximately 76ms to 51ms.
  • Robustness: The HNFT strategy successfully addressed the phonetic confusion that typically plagues large-scale catalogs.

Discussion and Limitations

While RAC significantly improves scalability, we observed some regressions on longer audio segments (e.g., in the VoxPopuli dataset). In these cases, the Contextual Adapter occasionally over-biases, substituting common words with rare entities. Additionally, while a 20,000-item catalog is a major step forward, it may still lack the breadth needed for global inclusivity. Future work will focus on improving biasing precision for long-form audio and expanding catalog diversity to better represent underrepresented communities and accents.

Conclusion

The Retrieve and Copy framework, combined with Hard Negative Fine-tuning, provides a robust path for scaling ASR personalization. By decoupling the biasing mechanism from the total catalog size and sharpening the model's discriminative capabilities, we enable high-performance, real-time ASR for large-scale, entity-heavy applications.

Summary updated

Abstract

Personalization of automatic speech recognition (ASR) models is a widely studied topic because of its many practical applications. Most recently, attention-based contextual biasing techniques are used to improve the recognition of rare words and domain specific entities. However, due to performance constraints, the biasing is often limited to a few thousand entities, restricting real-world usability. To address this, we first propose a "Retrieve and Copy" mechanism to improve latency while retaining the accuracy even when scaled to a large catalog. We also propose a training strategy to overcome the degradation in recall at such scale due to an increased number of confusing entities. Overall, our approach achieves up to 6% more Word Error Rate reduction (WERR) and 3.6% absolute improvement in F1 when compared to a strong baseline. Our method also allows for large catalog sizes of up to 20K without significantly affecting WER and F1-scores, while achieving at least 20% inference speedup per acoustic frame.
BibTeX
@inproceedings{jayanthi2023retrieve,
  author = {Sai Muralidhar Jayanthi and Devang Kulshreshtha and Saket Dingliwal and Srikanth Ronanki and Sravan Bodapati},
  title = {{Retrieve and Copy: Scaling ASR Personalization to Large Catalogs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=NXDoxLhCpo},
}
Figure