Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer

Suyoun Kim, Yuan Shangguan, Jay Mahadeokar, Antoine Bruguier, Christian Fuegen, Michael L. Seltzer, Duc Le

TL;DR

This paper introduces an improved method for integrating external Neural Language Models (NNLMs) into streaming Recurrent Neural Network Transducer (RNN-T) speech recognition systems. By combining an upgraded "cold fusion" training technique with standard "shallow fusion," the authors achieve a 13-18% relative Word Error Rate (WER) reduction on Librispeech without adding latency, while uniquely enabling "plug-and-play" swapping of language models at inference time without retraining.

What Problem Does This Paper Address?

Recurrent Neural Network Transducers (RNN-Ts) are the go-to architecture for on-device, streaming Automatic Speech Recognition (ASR). Because they process audio and emit text on the fly, they are highly efficient. However, RNN-Ts rely on an implicit internal language model (the prediction network) that struggles to capture long-range linguistic context.

Furthermore, because RNN-Ts are trained end-to-end on paired audio-text data, it is notoriously difficult to teach them using massive datasets of unpaired text (text without accompanying audio). While techniques like "shallow fusion" allow an external language model to help out during inference, the ASR model itself remains blind to this rich text data during training. The authors tackle the challenge of deeply integrating external NNLMs into the RNN-T during training (cold fusion) while keeping the system fast, streamable, and flexible.

Methods

The authors combine two fusion techniques to inject external NNLM knowledge into the RNN-T:

  1. Shallow Fusion (SF): A standard inference-time technique where the scores of an external NNLM are interpolated with the RNN-T's scores during beam search decoding.
  2. Cold Fusion (CF) [Novel Extensions]: A method of fusing a pre-trained external NNLM directly into the RNN-T's network during training. The authors introduce three crucial modifications to traditional cold fusion:
  3. Iterative Training: Instead of training the whole massive network from scratch, they pre-train the RNN-T and the NNLM separately. They then freeze the NNLM and fine-tune the combined model.
  4. Logit-based Fusion: Instead of fusing the complex, high-dimensional hidden states of the NNLM into the RNN-T, they fuse the NNLM's output logits (its final probability predictions).
  5. Fine Gating Mechanism: They use a multiplicative gating mechanism to intelligently combine the RNN-T predictor's output with the NNLM's linguistic information.

Datasets

  • Librispeech: 960 hours of labeled speech used for training the acoustic/RNN-T models.
  • Librispeech Text-Only Corpus: 810 million words used to train the external NNLM.
  • Note: Standard data augmentations like SpecAugment and speed perturbation were applied.

Key Insights and Ideas

  • Iterative training is make-or-break: The authors discovered that training a cold-fusion RNN-T from scratch completely fails to converge. Because the external NNLM provides such a strong, accurate signal early on, the acoustic parts of the RNN-T become "lazy" and under-train. Bootstrapping from a pre-trained RNN-T solves this.
  • Logits enable "Plug-and-Play" LMs: By fusing the NNLM's logits rather than its hidden states, the RNN-T becomes agnostic to the specific architecture of the external language model. This means you can train the system once, and at inference time, swap out a massive, heavy NNLM for a lightweight one—or a highly specific domain-adapted one—without retraining the acoustic model.
  • Fixing the RNN-T's blind spots: Analysis showed that NNLM fusion disproportionately improves accuracy on long utterances (proving it fixes the RNN-T's short-sighted context window) and on rare/Out-Of-Vocabulary (OOV) words (proving the model successfully leverages the unpaired text data).

Key Results

  • Significant WER Reduction: Combining Shallow and Cold Fusion (SF+CF) yielded an overall 13-18% relative WER reduction over strong vanilla RNN-T baselines. For example, on the challenging Librispeech test-other split, WER dropped from 10.1% to 8.3% using an LSTM encoder.
  • Zero Added Latency: Because the fusion happens on-the-fly during first-pass decoding, the model incurs no extra algorithmic latency, preserving its streamability.
  • Successful LM Swapping: The authors successfully swapped a 53M-parameter NNLM for a much smaller 15M-parameter NNLM at inference time with almost no performance penalty (WER went from 9.4% to 9.6% on test-other). Swapping in an "oracle" domain-specific LM dropped WER massively (down to 5.8%), proving the viability of zero-shot domain adaptation.

Limitations and Future Work

  • Cold Fusion < Shallow Fusion: The authors honestly note that, surprisingly, cold fusion alone underperforms shallow fusion alone. They hypothesize this is because shallow fusion's direct interpolation distributes probability mass more evenly across vocabulary tokens, whereas the RNN-T's native scores (which cold fusion modifies) are notoriously "spiky."
  • Future Directions: The team plans to investigate ways to close this performance gap between cold and shallow fusion. They also intend to compare these first-pass streaming methods against heavier second-pass LM rescoring techniques, and further explore on-the-fly domain adaptation using their plug-and-play LM swapping capability.

Summary updated

Abstract

Recurrent Neural Network Transducer (RNN-T), like most end-to-end speech recognition model architectures, has an implicit neural network language model (NNLM) and cannot easily leverage unpaired text data during training. Previous work has proposed various fusion methods to incorporate external NNLMs into end-to-end ASR to address this weakness. In this paper, we propose extensions to these techniques that allow RNN-T to exploit external NNLMs during both training and inference time, resulting in 13-18% relative Word Error Rate improvement on Librispeech compared to strong baselines. Furthermore, our methods do not incur extra algorithmic latency and allow for flexible plug- and-play of different NNLMs without re-training. We also share in-depth analysis to better understand the benefits of the different NNLM fusion methods. Our work provides a reliable technique for leveraging unpaired text data to significantly improve RNN-T while keeping the system streamable, flexible, and lightweight.
BibTeX
@inproceedings{kim2021improved,
  author = {Suyoun Kim and Yuan Shangguan and Jay Mahadeokar and Antoine Bruguier and Christian Fuegen and Michael L. Seltzer and Duc Le},
  title = {{Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=TWSxSF8Udt},
}

Loading related content...

Figure