Abstract
Streaming processing of speech audio is essential for modern speech recognition tasks. However, end-to-end (E2E) models often struggle with the "long tail" of linguistic content due to limited paired audio-transcript data. This paper explores enhancing streaming Recurrent Neural Network Transducers (RNNT) using external language models (LM) trained on massive text-only datasets. We evaluate shallow fusion, cold fusion, and two new variants: early shallow fusion and early cold fusion. Our results demonstrate that cold fusion consistently outperforms other methods, providing up to an 8.5% relative improvement in Word Error Rate (WER).
The Challenge: Data Scarcity and the "Long Tail"
Modern E2E speech recognition models simplify the traditional pipeline into a single neural network. While elegant, these models require "paired" data—audio matched with transcripts. High-quality transcribed speech is far scarcer than the billions of sentences available in text-only corpora. This leads to the "long tail" problem, where models fail on rare words, proper nouns, and technical jargon that appear infrequently in speech training sets.
For streaming applications like voice search, the model must transcribe audio in real-time. This research investigates how to integrate massive text-only knowledge into streaming RNNT architectures to improve recognition of infrequent linguistic entities without compromising real-time performance.
Fusion Methodologies
The study focuses on four distinct ways to integrate an external Recurrent Neural Network Language Model (RNN-LM) into the RNNT architecture:
- Shallow Fusion (SF): The RNNT and LM are trained independently. During inference, their scores are combined via a weighted sum.
- Cold Fusion (CF): The RNNT is trained while connected to a pre-trained, fixed LM. A gating mechanism allows the model to dynamically balance acoustic features and linguistic priors.
- Early Shallow Fusion (ESF): A novel approach where the LM is fused with the RNNT’s internal prediction network before the final joint network.
- Early Cold Fusion (ECF): Similar to CF, but fusion occurs earlier in the architecture, passing LM outputs through a dedicated neural layer before gating.
Experimental Setup
The methods were evaluated across three languages with varying data availability:
- Greek: ~6,700 hours (High resource).
- Norwegian: ~3,500 hours (Medium resource).
- Sinhala: ~160 hours (Low resource).
External LMs were trained on over a billion sentences from Google Search queries and web documents for Greek and Norwegian.
Key Insights
- RNNT vs. LAS: Unlike non-streaming "Attention" models (like LAS) where Shallow Fusion often wins, Cold Fusion is superior for RNNTs. This suggests RNNTs are better suited for integrated training with language models.
- The Blank Symbol: RNNTs use a
<blank>symbol for frames with no output. The authors synchronized the models so the LM only updates when a non-blank token is predicted. - Architectural Flexibility: "Early" fusion variants showed that the fusion point can be moved within the architecture with minimal impact on quality, offering flexibility for hardware-efficient designs.
Results and Performance
Cold Fusion emerged as the most effective strategy, particularly for medium-resource languages.
- Cold Fusion (CF) achieved an 8.5% relative WER reduction for Norwegian and a 3.4% reduction for Greek compared to the baseline.
- Shallow Fusion (SF) yielded smaller gains (~1.2% for Norwegian) and required sensitive hyperparameter tuning.
- Low-Resource Limitations: In Sinhala, gains were negligible due to domain mismatch between the text data and the voice search test sets, leading to overfitting.
Conclusion
Integrating external language models via Cold Fusion significantly enhances streaming RNNT performance by addressing the linguistic long tail. While highly effective for medium and high-resource languages, future work is needed to improve domain adaptation and fusion strategies for extremely low-resource scenarios.
Summary updated