Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer

Suyoun Kim, Yuan Shangguan, Jay Mahadeokar, Antoine Bruguier, Christian Fuegen, Michael L. Seltzer, Duc Le

Abstract

Recurrent Neural Network Transducer (RNN-T) architectures are widely used for end-to-end speech recognition but struggle to leverage unpaired text data due to their implicit internal language models. This paper proposes extensions to fusion techniques that integrate external Neural Language Models (NNLMs) during both training and inference. Our approach achieves a 13-18% relative Word Error Rate (WER) improvement on Librispeech without increasing algorithmic latency. Crucially, our method enables "plug-and-play" flexibility, allowing different NNLMs to be swapped at inference time without retraining the acoustic model. We provide an in-depth analysis to understand the benefits of different NNLM fusion methods, offering a reliable technique for significantly improving RNN-T while keeping the system streamable and lightweight.

Introduction

Streaming Automatic Speech Recognition (ASR) systems often utilize RNN-Ts for their efficiency and low latency. However, these models are typically trained on paired audio-text data, missing out on the vast amounts of available text-only data. While "shallow fusion" (SF) can incorporate an external LM during decoding, the core RNN-T model remains unexposed to this linguistic knowledge during training. We address this by refining "cold fusion" (CF) techniques to deeply integrate external NNLMs into the RNN-T training process while maintaining a streamable and flexible architecture.

Methodology: Advanced Fusion Techniques

We combine two primary fusion methods to inject external linguistic knowledge into the RNN-T:

  1. Shallow Fusion (SF): A standard inference-time technique where external NNLM scores are interpolated with RNN-T scores during beam search decoding.
  2. Enhanced Cold Fusion (CF): We introduce three key modifications to traditional cold fusion to improve stability and flexibility:
  3. Iterative Training: We pre-train the RNN-T and NNLM separately, then freeze the NNLM while fine-tuning the combined system. This prevents the acoustic model from becoming "lazy" due to the strong NNLM signal provided early in training.
  4. Logit-based Fusion: Instead of fusing high-dimensional hidden states, we fuse the NNLM's output logits. This decouples the RNN-T from the specific NNLM architecture, enabling model swapping.
  5. Fine Gating Mechanism: A multiplicative gating mechanism is used to intelligently balance the RNN-T predictor's output with the NNLM's linguistic information.

Experimental Setup

Models were trained on the Librispeech 960-hour labeled dataset. The external NNLM was trained on the Librispeech Text-Only Corpus (810M words). We evaluated performance on the standard test-clean and test-other splits, employing standard data augmentations like SpecAugment and speed perturbation.

Key Insights and Analysis

Our analysis reveals several critical findings regarding the integration of NNLMs:

  • Convergence through Iteration: Training from scratch with cold fusion fails to converge because the NNLM provides such a strong signal that the acoustic components under-train. Iterative training is essential for a balanced model.
  • Plug-and-Play Flexibility: By fusing logits rather than hidden states, the system becomes LM-agnostic. We demonstrated that a model trained with a 53M-parameter LM can be swapped with a 15M-parameter LM at inference with minimal performance loss.
  • Addressing Structural Weaknesses: Analysis showed that NNLM fusion disproportionately improves accuracy on long utterances and rare/Out-Of-Vocabulary (OOV) words, proving it successfully extends the RNN-T's context window and leverages unpaired text data.

Results

The combination of Shallow and Cold Fusion (SF+CF) consistently outperformed baselines across different encoder architectures, as shown in the table below.

Table 1: Performance comparison of different encoder architectures and fusion methods on Librispeech.

As illustrated, the SF+CF approach achieved a relative WER reduction of 13-18%. For the LC-BLSTM model, WER on test-other improved from 8.0% (None) to 6.8% (SF+CF). Importantly, these gains are achieved with zero added algorithmic latency.

Furthermore, the "plug-and-play" capability was validated by swapping language models at inference time.

Table 2: Impact of swapping language models at inference time on WER.

Swapping the 53M LM for a 15M LM resulted in only a minor WER increase (9.4% to 9.6% on test-other), while an "oracle" LM showed massive potential for zero-shot domain adaptation, dropping WER to as low as 4.9%.

Conclusion

Our proposed fusion extensions provide a robust framework for leveraging unpaired text data in streaming RNN-T systems. The method is lightweight, adds no latency, and offers unique flexibility for on-the-fly LM adaptation. Future work will focus on closing the performance gap between standalone cold fusion and shallow fusion, and exploring further domain adaptation strategies using this plug-and-play capability.

Summary updated

Abstract

Recurrent Neural Network Transducer (RNN-T), like most end-to-end speech recognition model architectures, has an implicit neural network language model (NNLM) and cannot easily leverage unpaired text data during training. Previous work has proposed various fusion methods to incorporate external NNLMs into end-to-end ASR to address this weakness. In this paper, we propose extensions to these techniques that allow RNN-T to exploit external NNLMs during both training and inference time, resulting in 13-18% relative Word Error Rate improvement on Librispeech compared to strong baselines. Furthermore, our methods do not incur extra algorithmic latency and allow for flexible plug-and-play of different NNLMs without re-training. We also share in-depth analysis to better understand the benefits of the different NNLM fusion methods. Our work provides a reliable technique for leveraging unpaired text data to significantly improve RNN-T while keeping the system streamable, flexible, and lightweight.
BibTeX
@inproceedings{kim2021improved,
  author = {Suyoun Kim and Yuan Shangguan and Jay Mahadeokar and Antoine Bruguier and Christian Fuegen and Michael L. Seltzer and Duc Le},
  title = {{Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=TWSxSF8Udt},
}
Figure