Contextual Speech Recognition with Difficult Negative Training Examples

Uri Alon, Golan Pundak, Tara N. Sainath

TL;DR

Researchers at Google and Technion have developed a way to significantly improve how voice assistants recognize rare words and names by training them with "difficult negative examples." By intentionally showing the model phonetically similar distractors (like "Joan" vs. "John") during training, they achieved up to a 53.1% relative improvement in Word Error Rate (WER) for contextual tasks.

What Problem Does This Paper Address?

End-to-end (E2E) Automatic Speech Recognition (ASR) models have become the gold standard for voice search, but they have a "long tail" problem: they struggle with rare words and Out-of-Vocabulary (OoV) terms. This is particularly frustrating when it comes to user-specific context, such as names in a contact list or specific song titles.

While previous work introduced "contextual biasing"—allowing a model to "peek" at a list of relevant phrases (like your contacts)—the models often get confused when that list contains words that sound similar. If your contacts include both "Joan" and "John," a standard neural model might produce very similar internal representations for both, leading to frequent misidentifications. The challenge is teaching the model to distinguish between these subtle phonetic differences.

Methods

The authors build upon the Contextualized Listen, Attend, and Spell (CLAS) model. Their core innovation is a specialized training pipeline that moves away from using random "distractor" phrases and moves toward "hard negatives."

  1. Detecting Proper Nouns: Using a Part-Of-Speech (POS) tagger, the system identifies proper nouns (NNP) in the training transcripts. These are the words most likely to be rare or difficult to transcribe.
  2. Building a Fuzzy Inventory: The researchers created an unsupervised "fuzzy inventory" by analyzing an external ASR model's decoding hypotheses. They looked for word pairs that frequently appeared as alternatives for the same audio and filtered them using the Hixon metric to ensure they were phonetically similar.
  3. Hard Negative Augmentation: During training, instead of giving the model the correct phrase plus random words, they provided the correct phrase plus its "fuzzy" (phonetically similar) alternatives.
  4. Balancing with $\alpha_{drop}$: To ensure the model doesn't become over-reliant on context (which would hurt performance on general speech), they introduced a dropout parameter ($\alpha_{drop}$). This occasionally trains the model without any bias phrases, forcing it to maintain its "pure" ASR capabilities.

Datasets

  • Training: ~25,000 hours of anonymized Google voice search traffic (33 million utterances), augmented with simulated room acoustics and noise.
  • Contextualized Test Sets:
    • Songs: 15k utterances with an average of 303 bias phrases.
  • Contacts: 15k utterances with an average of 75 bias phrases.
  • Talk-To: 4k utterances with a massive average of 3,255 bias phrases.
  • Context-Free Test Sets: Voice Search and Dictation sets used to ensure the model still performs well on general tasks.

Key Insights and Ideas

  • Discriminative Learning: By providing "Joan" as a negative example when the target is "John," the model is forced to learn more discriminative representations. It can no longer rely on a "close enough" phonetic match.
  • The "Lazy" Model Problem: Standard training with random distractors is too easy. The model learns to identify the target simply because it's the only thing that sounds remotely like the audio. Hard negatives raise the bar.
  • Cleaner Attention: Qualitative analysis showed that models trained with fuzzy negatives developed "sharper" attention. Instead of spreading its focus across several similar-sounding phrases, the model learned to "lock on" to the correct one more decisively.
  • The Context Trade-off: There is a natural tension between being good at using context and being good at general ASR. The use of $\alpha_{drop}$ is a simple but effective "knob" to tune this balance.

Key Results

The proposed CLAS NNP+fuzzy model showed dramatic improvements over the "Vanilla" CLAS baseline:

  • 53.1% relative WER improvement on the Contacts test set.
  • 44.9% relative WER improvement on the Songs test set.
  • 25.7% relative WER improvement on the Talk-To set (which is particularly difficult due to the high number of distractors).
  • General Performance: By tuning $\alpha_{drop}$ to 30%, the researchers were able to recover almost all the performance lost on general (context-free) ASR tasks while still maintaining massive gains on contextual tasks.

Limitations and Future Work

  • Scalability of Fuzzy Inventory: The paper relies on a pre-built fuzzy inventory derived from a conventional ASR model. Future work might explore generating these alternatives on-the-fly or using purely neural methods to find phonetic neighbors.
  • Computational Overhead: While the training is more efficient in terms of accuracy, the paper does not deeply explore the added latency or memory requirements of managing large bias sets (like the 3,000+ phrases in Talk-To) in a real-time production environment.
  • Language Support: The study was limited to English; the effectiveness of phonetic "fuzzing" may vary significantly in tonal languages or languages with different phonetic densities.

Summary updated

Abstract

Improving the representation of contextual information is key to unlocking the potential of end-to-end (E2E) automatic speech recognition (ASR). In this work, we present a novel and simple approach for training an ASR context mechanism with difficult negative examples. The main idea is to focus on proper nouns (e.g., unique entities such as names of people and places) in the reference transcript and use phonetically similar phrases as negative examples, encouraging the neural model to learn more discriminative representations. We apply our approach to an end-to-end contextual ASR model that jointly learns to transcribe and select the correct context items. We show that our proposed method gives up to 53.1% relative improvement in word error rate (WER) across several benchmarks.
BibTeX
@inproceedings{alon2019contextual,
  author = {Uri Alon and Golan Pundak and Tara N. Sainath},
  title = {{Contextual Speech Recognition with Difficult Negative Training Examples}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=GWhRoeg6kSp},
}

Loading related content...

Figure