Abstract
Improving the representation of contextual information is key to unlocking the potential of end-to-end (E2E) automatic speech recognition (ASR). In this work, we present a novel and simple approach for training an ASR context mechanism with difficult negative examples. The main idea is to focus on proper nouns (e.g., unique entities such as names of people and places) in the reference transcript, and use phonetically similar phrases as negative examples, encouraging the neural model to learn more discriminative representations. We apply our approach to an end-to-end contextual ASR model that jointly learns to transcribe and select the correct context items, and show that our proposed method gives up to 53.1% relative improvement in word error rate (WER) across several benchmarks.
Introduction
End-to-end (E2E) Automatic Speech Recognition (ASR) models have become the gold standard for voice search, but they face a significant "long tail" problem: they struggle with rare words and Out-of-Vocabulary (OoV) terms. This is particularly evident in user-specific contexts, such as names in a contact list or specific song titles. While previous work introduced "contextual biasing"—allowing a model to reference a list of relevant phrases—models often fail when that list contains phonetically similar words. If a contact list includes both "Joan" and "John," standard neural models may produce similar internal representations, leading to misidentifications. The challenge lies in teaching the model to distinguish these subtle phonetic differences.
Methodology: Hard Negative Augmentation
To address this, researchers from Google and Technion developed a specialized training pipeline for the Contextualized Listen, Attend, and Spell (CLAS) model. The core innovation shifts from using random distractor phrases to using "hard negatives."
- Detecting Proper Nouns: The system identifies proper nouns (NNP) in training transcripts using a Part-Of-Speech (POS) tagger.
- Building a Fuzzy Inventory: An unsupervised inventory is created by analyzing word pairs that frequently appear as alternatives in ASR decoding, filtered by the Hixon metric for phonetic similarity.
- Hard Negative Augmentation: During training, the model is provided with the correct phrase plus its "fuzzy" (phonetically similar) alternatives, forcing it to learn more discriminative representations.
- Balancing with $\alpha_{drop}$: To prevent over-reliance on context, a dropout parameter ($\alpha_{drop}$) is used to occasionally train the model without bias phrases, maintaining its general ASR capabilities.
Experimental Results
The proposed CLAS NNP+fuzzy model was evaluated on ~25,000 hours of anonymized voice search traffic and several contextualized test sets (Songs, Contacts, and Talk-To). The results showed dramatic improvements over the vanilla CLAS baseline:
- 53.1% relative WER improvement on the Contacts test set.
- 44.9% relative WER improvement on the Songs test set.
- 25.7% relative WER improvement on the Talk-To set, which contains over 3,000 distractors.
Qualitative analysis revealed that models trained with fuzzy negatives developed "sharper" attention. Instead of spreading focus across similar-sounding phrases, the model learned to decisively "lock on" to the correct one. By tuning $\alpha_{drop}$ to 30%, the researchers recovered general ASR performance while maintaining these massive contextual gains.
Conclusion and Future Work
This work demonstrates that training with difficult negative examples is a powerful tool for improving contextual ASR. By raising the bar during training, models move beyond "close enough" phonetic matches to highly discriminative recognition. Future research may explore generating fuzzy alternatives on-the-fly or extending these methods to tonal languages where phonetic density and similarity present different challenges.
Summary updated