Abstract
Recurrent Neural Network Transducer (RNN-T) architectures are widely used for end-to-end speech recognition but struggle to leverage unpaired text data due to their implicit internal language models. This paper proposes extensions to fusion techniques that integrate external Neural Language Models (NNLMs) during both training and inference. Our approach achieves a 13-18% relative Word Error Rate (WER) improvement on Librispeech without increasing algorithmic latency. Crucially, our method enables "plug-and-play" flexibility, allowing different NNLMs to be swapped at inference time without retraining the acoustic model. We provide an in-depth analysis to understand the benefits of different NNLM fusion methods, offering a reliable technique for significantly improving RNN-T while keeping the system streamable and lightweight.
Introduction
Streaming Automatic Speech Recognition (ASR) systems often utilize RNN-Ts for their efficiency and low latency. However, these models are typically trained on paired audio-text data, missing out on the vast amounts of available text-only data. While "shallow fusion" (SF) can incorporate an external LM during decoding, the core RNN-T model remains unexposed to this linguistic knowledge during training. We address this by refining "cold fusion" (CF) techniques to deeply integrate external NNLMs into the RNN-T training process while maintaining a streamable and flexible architecture.
Methodology: Advanced Fusion Techniques
We combine two primary fusion methods to inject external linguistic knowledge into the RNN-T:
- Shallow Fusion (SF): A standard inference-time technique where external NNLM scores are interpolated with RNN-T scores during beam search decoding.
- Enhanced Cold Fusion (CF): We introduce three key modifications to traditional cold fusion to improve stability and flexibility:
- Iterative Training: We pre-train the RNN-T and NNLM separately, then freeze the NNLM while fine-tuning the combined system. This prevents the acoustic model from becoming "lazy" due to the strong NNLM signal provided early in training.
- Logit-based Fusion: Instead of fusing high-dimensional hidden states, we fuse the NNLM's output logits. This decouples the RNN-T from the specific NNLM architecture, enabling model swapping.
- Fine Gating Mechanism: A multiplicative gating mechanism is used to intelligently balance the RNN-T predictor's output with the NNLM's linguistic information.
Experimental Setup
Models were trained on the Librispeech 960-hour labeled dataset. The external NNLM was trained on the Librispeech Text-Only Corpus (810M words). We evaluated performance on the standard test-clean and test-other splits, employing standard data augmentations like SpecAugment and speed perturbation.
Key Insights and Analysis
Our analysis reveals several critical findings regarding the integration of NNLMs:
- Convergence through Iteration: Training from scratch with cold fusion fails to converge because the NNLM provides such a strong signal that the acoustic components under-train. Iterative training is essential for a balanced model.
- Plug-and-Play Flexibility: By fusing logits rather than hidden states, the system becomes LM-agnostic. We demonstrated that a model trained with a 53M-parameter LM can be swapped with a 15M-parameter LM at inference with minimal performance loss.
- Addressing Structural Weaknesses: Analysis showed that NNLM fusion disproportionately improves accuracy on long utterances and rare/Out-Of-Vocabulary (OOV) words, proving it successfully extends the RNN-T's context window and leverages unpaired text data.
Results
The combination of Shallow and Cold Fusion (SF+CF) consistently outperformed baselines across different encoder architectures, as shown in the table below.
As illustrated, the SF+CF approach achieved a relative WER reduction of 13-18%. For the LC-BLSTM model, WER on test-other improved from 8.0% (None) to 6.8% (SF+CF). Importantly, these gains are achieved with zero added algorithmic latency.
Furthermore, the "plug-and-play" capability was validated by swapping language models at inference time.
Swapping the 53M LM for a 15M LM resulted in only a minor WER increase (9.4% to 9.6% on test-other), while an "oracle" LM showed massive potential for zero-shot domain adaptation, dropping WER to as low as 4.9%.
Conclusion
Our proposed fusion extensions provide a robust framework for leveraging unpaired text data in streaming RNN-T systems. The method is lightweight, adds no latency, and offers unique flexibility for on-the-fly LM adaptation. Future work will focus on closing the performance gap between standalone cold fusion and shallow fusion, and exploring further domain adaptation strategies using this plug-and-play capability.
Summary updated