Abstract
Contextual biasing to a specific domain, including a user’s song names, app names and contact names, is an important component of any production-level automatic speech recognition (ASR) system. Contextual biasing is particularly challenging in end-to-end models because these models keep a small list of candidates during beam search, and also do poorly on proper nouns, which is the main source of biasing phrases. In this paper, we present various algorithmic and training improvements to shallow-fusion-based biasing for end-to-end models. We will show that the proposed approach obtains better performance than a state-of-the-art conventional model across a variety of tasks, the first time this has been demonstrated.
Introduction
Modern End-to-End (E2E) Automatic Speech Recognition (ASR) systems, such as RNN-Transducers, have revolutionized the field by folding acoustic, pronunciation, and language models into a single neural network. However, they consistently struggle with contextual biasing—the ability to recognize specific entities like contact names or song titles that are relevant to a specific user at a specific time.
E2E models face two primary hurdles in this area:
- Data Scarcity: These models are typically trained on less text than traditional language models, making them less proficient at recognizing rare proper nouns.
- The Pruning Problem: To maintain low latency, E2E systems use "beam search," keeping only the most likely candidates at each step. Rare names are often discarded (pruned) early in the decoding process before the system can accumulate enough evidence to recognize them.
Algorithmic Improvements to Shallow Fusion
The authors improve upon Shallow Fusion, a technique where a separate, small "contextual" Finite State Transducer (FST) containing biasing phrases adjusts the scores of the E2E model during decoding. Their novel contributions focus on ensuring relevant phrases survive the beam search process.
- Early Biasing: Traditionally, biasing scores are applied after the beam is pruned. The authors propose applying these scores before pruning. This ensures that rare but contextually relevant words remain in the beam.
- Wordpiece Biasing: Instead of biasing at the character (grapheme) level, which can "flood" the beam with irrelevant words sharing the same starting letters, they use Wordpieces. This provides a sparser, more targeted match for biasing.
- Prefix Activation: To prevent the model from hallucinating names when they aren't present (the "anti-context" problem), biasing is only activated if the system detects a common prefix like "call," "text," or "play."
Data Augmentation Strategies
To further bolster the model's performance on proper nouns, the researchers leveraged large-scale data augmentation:
- Unsupervised Data: 100 million anonymized voice search utterances were automatically transcribed and filtered for high-confidence proper nouns.
- TTS Synthesis: 1 million synthetic utterances were generated using Text-to-Speech for categories like contacts and apps to expose the model to diverse name distributions.
Experimental Results
The models were trained on 35 million English utterances (approx. 27,500 hours) and evaluated across specialized test sets including Voice Search (VS), Songs, Communication (Cnt-Real/TTS), and Apps.
As shown in the results, the proposed "Early Biasing" with wordpieces (Experiment E5) significantly reduced Word Error Rate (WER). Most notably, for the first time in published literature, the E2E model outperformed a state-of-the-art conventional model of comparable size by 20-40% relative across nearly all contextual domains.
Conclusion
This work demonstrates that with the right algorithmic adjustments—specifically early biasing and wordpiece-level integration—end-to-end models can overcome their inherent weaknesses in contextual recognition. By treating biasing as a "survival game" within the beam search and leveraging synthetic data, the RNN-T model finally surpasses traditional hybrid systems in handling the long tail of personalized speech recognition.
Summary updated