Towards Fast and Accurate Streaming End-to-End ASR

Bo Li, Shuo-Yiin Chang, Tara N. Sainath, Ruoming Pang, Yanzhang He, Trevor Strohman, Yonghui WuICASSP 2020

TL;DR

This paper tackles the challenge of building fast and accurate streaming Automatic Speech Recognition (ASR) systems for on-device applications. The authors propose an enhanced end-to-end (E2E) model, RNN-T EP, by introducing novel penalties for end-of-speech token prediction, incorporating Minimum Word Error Rate (MWER) training, and employing a Listen, Attend and Spell (LAS) rescorer. This integrated approach significantly reduces latency while maintaining or improving accuracy.

What Problem Does This Paper Address?

Traditional ASR systems are complex, comprising separate acoustic, pronunciation, and language models. End-to-end (E2E) models simplify this by integrating these components into a single neural network, making them more suitable for resource-constrained environments like mobile devices. However, for streaming ASR, two critical factors are accuracy (measured by Word Error Rate or WER) and latency (the time between speech ending and the system outputting a hypothesis). Existing E2E models, particularly the Recurrent Neural Network Transducer (RNN-T), show promise but struggle to achieve an optimal balance between these two metrics. A key challenge is accurately detecting when a user has finished speaking (endpointing) without cutting off speech or introducing unnecessary delays. This paper aims to bridge this gap by developing an E2E model that jointly optimizes for both speed and accuracy in a streaming setting.

Methods

The core of the proposed solution is an extension of the RNN-T Endpointer (RNN-T EP) model. Here's a breakdown of the key techniques:

  • Joint Endpointing and Decoding: The RNN-T EP model integrates an endpointer directly into the RNN-T architecture by introducing a special end-of-speech token (</s>). This allows the model to learn to predict when speech ends simultaneously with transcribing it.
  • Early and Late Penalties: To guide the model towards better latency-accuracy trade-offs, the authors introduce penalties during training for predicting the </s> token too early or too late. These penalties are applied based on a ground truth end-of-speech time derived from forced alignment.
  • Minimum Word Error Rate (MWER) Training: Traditional training optimizes for log-likelihood, which doesn't directly correlate with WER. MWER training, a sequence-level training technique, is employed to directly minimize WER. This is particularly beneficial for RNN-T EP because premature </s> predictions can lead to significant sequence-level losses.
  • Listen, Attend and Spell (LAS) Rescoring: A non-streaming LAS model is used as a second-pass rescorer. While it doesn't directly reduce the first-pass streaming latency, it can significantly improve WER. The paper highlights how the presence of the </s> token in the RNN-T EP model facilitates a more consistent score combination with the LAS rescorer.

Datasets

The authors use a multidomain dataset for training, incorporating multistyle training (MTR) for noise robustness. The test set consists of 14K Voice Search utterances, each less than 5.5 seconds long, which are anonymized and hand-transcribed. This test set is representative of Google's traffic.

Key Insights and Ideas

  • The </s> token is a double-edged sword: While crucial for enabling joint endpointing and reducing latency, its premature prediction can lead to deletion errors, and late prediction increases latency. The proposed penalties directly address this.
  • Sequence-level training (MWER) is vital for endpointing: Because incorrect endpointing decisions (especially premature </s>) have a cascading effect on the entire recognized sequence, optimizing for sequence-level loss (like WER) is more effective than token-level loss.
  • LAS rescoring complements streaming models: Even though LAS is not a streaming model, it can effectively improve the accuracy of a streaming RNN-T EP system, providing a way to achieve better overall performance without sacrificing the benefits of streaming.
  • Joint optimization unlocks better trade-offs: By jointly training the endpointing and recognition components, and further refining this with MWER and LAS rescoring, the authors demonstrate a significant improvement in the WER-latency trade-off.

Key Results

The proposed RNN-T EP model with early and late penalties, combined with MWER training and LAS rescoring, achieved substantial improvements:

  • Relative WER reduction: 18.7% compared to the original RNN-T EP model.
  • Latency reduction:
    • 40ms reduction in median latency (EP50).
  • 160ms reduction in 90-percentile latency (EP90).
  • The model with late penalty and MWER training (E5) showed an 8.0% relative WER reduction and 30ms EP50 / 130ms EP90 latency reductions compared to the baseline RNN-T (B1).
  • The final model (E8) achieved a WER of 6.1% and an EP90 of 550ms, a significant improvement over the baseline RNN-T EP (B2) which had 7.5% WER and 710ms EP90.

Limitations and Future Work

  • LAS rescoring doesn't affect first-pass latency: While LAS improves overall accuracy, it's a second-pass process and doesn't directly speed up the initial streaming output. The paper notes that the WER gains from LAS provide more room to trade WER for latency.
  • Hyperparameter tuning: The authors mention that penalties and thresholds (like tbuffer, early, late, α, β) are tuned experimentally, suggesting that optimal values might be dataset- or task-dependent.
  • Future work: The paper implicitly suggests further exploration of sequence-level training and more sophisticated rescoring techniques to push the boundaries of streaming ASR performance. The ROC curves presented in Figure 2 visually demonstrate the improved trade-off space achieved by their proposed methods, hinting at further opportunities for fine-tuning.

Summary updated

Abstract

End-to-end (E2E) models fold the acoustic, pronunciation and language models of a conventional speech recognition model into one neural network with a much smaller number of parameters than a conventional ASR system, thus making it suitable for on-device applications. For example, recurrent neural network transducer (RNN-T) as a streaming E2E model has shown promising potential for on-device ASR. For such applications, quality and latency are two critical factors. We propose to reduce E2E model's latency by extending the RNN-T endpointer (RNN-T EP) model with additional early and late penalties. By further applying the minimum word error rate (MWER) training technique, we achieved 8.0% relative word error rate (WER) reduction and 130ms 90-percentile latency reduction over on a Voice Search test set. We also experimented with a second-pass Listen, Attend and Spell (LAS) rescorer . Although it did not directly improve the first pass latency, the large WER reduction provides extra room to trade WER for latency. RNN-T EP+LAS, together with MWER training brings in 18.7% relative WER reduction and 160ms 90-percentile latency reductions compared to the original proposed RNN-T EP model.
BibTeX
@inproceedings{li2020towards,
  author = {Bo Li and Shuo-Yiin Chang and Tara N. Sainath and Ruoming Pang and Yanzhang He and Trevor Strohman and Yonghui Wu},
  title = {{Towards Fast and Accurate Streaming End-to-End ASR}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
}
Figure