Personalization Strategies for End-to-End Speech Recognition Systems

Aditya Gourav, Linda Liu, Ankur Gandhe, Yile Gu, Guitang Lan, Xiangyang Huang, Shashank Kalmane, Gautam Tiwari, Denis Filimonov, Ariya Rastrow, Andreas Stolcke, Ivan Bulyko

Abstract

End-to-end (E2E) speech recognition models, such as RNN-Transducers, offer streamlined architectures but often struggle with personalizing vocabulary for specific users. This paper presents a two-stage approach to improve the recognition of rare personalized terms like contact names and smart home devices. By implementing a subword lookahead biasing technique in the first pass and a de-biasing optimization in the second pass, the researchers achieved a 30% Word Error Rate Reduction (WERR) on personalized content while simultaneously improving general recognition accuracy by 2.5%.

Introduction

Modern end-to-end (E2E) speech recognition systems have simplified the speech processing pipeline by replacing multi-component systems with a single neural network. However, this architectural simplicity introduces challenges in personalization. Specifically, these models often fail to recognize rare, user-specific terms like contact names ("Ariya") or unique device identifiers ("Kitchen-Left") because these terms are underrepresented in the general training data.

Researchers at Amazon Alexa have developed a novel two-pass strategy to address this. By combining a "subword lookahead" biasing technique with a second-pass "de-biasing" optimization, they achieved significant improvements in personalized recognition without sacrificing general performance.

The Personalization Challenge

The difficulty in personalizing E2E systems stems from two primary issues:

  1. The Beam Search Problem: Rare names are often discarded early in the decoding process (the "beam search") before the model has enough context to recognize them.
  2. The Over-Biasing Problem: Aggressively forcing the model to recognize rare terms can lead to "hallucinations," where common words are misidentified as personalized terms (e.g., hearing "Ariya" when the user said "Area").

Proposed Methodology

The authors introduced three key innovations to overcome these hurdles.

Subword Rescoring with Lookahead

E2E models typically operate on subword units rather than full words. Traditional biasing requires building subword-level models, which is difficult to maintain if the main model's vocabulary changes. The researchers proposed a word-level biasing approach that uses a "lookahead" algorithm. During decoding, the system checks if the current subword sequence matches a prefix of a word in the user's personalized list. If a match is found, the weight is applied immediately at the subword level, keeping the correct path alive in the search beam.

Comparison of word-level (a) and subword-level (b) biasing FSTs with lookahead.

Contextual Boosting

To minimize false positives, the system employs a contextual model that only activates when the user's likely intent involves specific categories, such as "contacts" or "smart home."

Second-Pass De-biasing

The most significant innovation is the second-pass de-biasing. While aggressive first-pass biasing ensures the correct term is included in the "N-best" list of candidates, it can distort the final rankings. In the second pass, the system re-weights scores from the E2E model, the biasing model, and a large neural language model. This "de-biasing" step re-balances the scores to accurately select the correct hypothesis from the candidate list, effectively cleaning up hallucinations introduced in the first pass.

Experimental Results

The system was evaluated using a baseline RNN-T trained on 200,000 hours of interactions, tested against 20,000 contact-related utterances and 20,000 general utterances.

The results demonstrate a powerful synergy between the two passes. First-pass subword biasing alone improved contact recognition by 14-16% Word Error Rate Reduction (WERR). Adding the second-pass de-biasing provided an additional 14% boost, bringing the total improvement to 27-30% WERR.

Word Error Rate Reduction (WERR) for personalized contacts and general use cases using second-pass de-biasing.

Crucially, this approach did not degrade performance on general tasks. In fact, the de-biasing step improved general accuracy by 2.5%, proving that the optimization effectively handles the trade-off between personalization and general robustness.

Conclusion

This research provides a scalable and flexible framework for personalizing E2E speech recognition. By decoupling the biasing list from the specific subword vocabulary and introducing a corrective second pass, the system achieves high accuracy for personalized terms while maintaining—and even improving—robustness on general speech. Future work may extend this de-biasing framework to other contextual cues, such as user preferences or conversational history.

Summary updated

BibTeX
@inproceedings{gourav2021personalization,
  author = {Aditya Gourav and Linda Liu and Ankur Gandhe and Yile Gu and Guitang Lan and Xiangyang Huang and Shashank Kalmane and Gautam Tiwari and Denis Filimonov and Ariya Rastrow and Andreas Stolcke and Ivan Bulyko},
  title = {{Personalization Strategies for End-to-End Speech Recognition Systems}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413962},
}
Figure