Language model fusion for streaming end to end speech recognition

Rodrigo Cabrera, Xiaofeng Liu, Mohammadreza Ghodsi, Zebulun Matteson, Eugene Weinstein, Anjuli Kannan

Abstract

Streaming processing of speech audio is essential for modern speech recognition tasks. However, end-to-end (E2E) models often struggle with the "long tail" of linguistic content due to limited paired audio-transcript data. This paper explores enhancing streaming Recurrent Neural Network Transducers (RNNT) using external language models (LM) trained on massive text-only datasets. We evaluate shallow fusion, cold fusion, and two new variants: early shallow fusion and early cold fusion. Our results demonstrate that cold fusion consistently outperforms other methods, providing up to an 8.5% relative improvement in Word Error Rate (WER).

The Challenge: Data Scarcity and the "Long Tail"

Modern E2E speech recognition models simplify the traditional pipeline into a single neural network. While elegant, these models require "paired" data—audio matched with transcripts. High-quality transcribed speech is far scarcer than the billions of sentences available in text-only corpora. This leads to the "long tail" problem, where models fail on rare words, proper nouns, and technical jargon that appear infrequently in speech training sets.

For streaming applications like voice search, the model must transcribe audio in real-time. This research investigates how to integrate massive text-only knowledge into streaming RNNT architectures to improve recognition of infrequent linguistic entities without compromising real-time performance.

Fusion Methodologies

The study focuses on four distinct ways to integrate an external Recurrent Neural Network Language Model (RNN-LM) into the RNNT architecture:

Comparison of the four fusion architectures: (a) Shallow fusion, (b) Early shallow fusion, (c) Cold fusion, and (d) Early cold fusion.
  1. Shallow Fusion (SF): The RNNT and LM are trained independently. During inference, their scores are combined via a weighted sum.
  2. Cold Fusion (CF): The RNNT is trained while connected to a pre-trained, fixed LM. A gating mechanism allows the model to dynamically balance acoustic features and linguistic priors.
  3. Early Shallow Fusion (ESF): A novel approach where the LM is fused with the RNNT’s internal prediction network before the final joint network.
  4. Early Cold Fusion (ECF): Similar to CF, but fusion occurs earlier in the architecture, passing LM outputs through a dedicated neural layer before gating.

Experimental Setup

The methods were evaluated across three languages with varying data availability:

  • Greek: ~6,700 hours (High resource).
  • Norwegian: ~3,500 hours (Medium resource).
  • Sinhala: ~160 hours (Low resource).

External LMs were trained on over a billion sentences from Google Search queries and web documents for Greek and Norwegian.

Key Insights

  • RNNT vs. LAS: Unlike non-streaming "Attention" models (like LAS) where Shallow Fusion often wins, Cold Fusion is superior for RNNTs. This suggests RNNTs are better suited for integrated training with language models.
  • The Blank Symbol: RNNTs use a <blank> symbol for frames with no output. The authors synchronized the models so the LM only updates when a non-blank token is predicted.
  • Architectural Flexibility: "Early" fusion variants showed that the fusion point can be moved within the architecture with minimal impact on quality, offering flexibility for hardware-efficient designs.

Results and Performance

Cold Fusion emerged as the most effective strategy, particularly for medium-resource languages.

Detailed architecture of the Cold Fusion (CF) method, which achieved the best performance.
  • Cold Fusion (CF) achieved an 8.5% relative WER reduction for Norwegian and a 3.4% reduction for Greek compared to the baseline.
  • Shallow Fusion (SF) yielded smaller gains (~1.2% for Norwegian) and required sensitive hyperparameter tuning.
  • Low-Resource Limitations: In Sinhala, gains were negligible due to domain mismatch between the text data and the voice search test sets, leading to overfitting.

Conclusion

Integrating external language models via Cold Fusion significantly enhances streaming RNNT performance by addressing the linguistic long tail. While highly effective for medium and high-resource languages, future work is needed to improve domain adaptation and fusion strategies for extremely low-resource scenarios.

Summary updated

Abstract

Streaming processing of speech audio is required for many contemporary practical speech recognition tasks. Even with the large corpora of manually transcribed speech data available today, it is impossible for such corpora to cover adequately the long tail of linguistic content that's important for tasks such as open-ended dictation and voice search. We seek to address both the streaming and the tail recognition challenges by using a language model (LM) trained on unpaired text data to enhance the end-to-end (E2E) model. We extend shallow fusion and cold fusion approaches to streaming Recurrent Neural Network Transducer (RNNT), and also propose two new competitive fusion approaches that further enhance the RNNT architecture. Our results on multiple languages with varying training set sizes show that these fusion methods improve streaming RNNT performance through introducing extra linguistic features. Cold fusion works consistently better on streaming RNNT with up to a 8.5% WER improvement.
BibTeX
@misc{cabrera2021language,
  author = {Rodrigo Cabrera and Xiaofeng Liu and Mohammadreza Ghodsi and Zebulun Matteson and Eugene Weinstein and Anjuli Kannan},
  title = {{Language model fusion for streaming end to end speech recognition}},
  year = {2021},
  eprint = {2104.04487},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.04487},
}
Figure