On Language Model Integration for RNN Transducer Based Speech Recognition

Wei Zhou, Wei Zhou 0043, Zuoyun Zheng, Ralf Schlüter, Hermann NeyICASSP 2022 · DOI

TL;DR

This paper tackles the performance bottleneck in end-to-end speech recognition systems caused by the mismatch between the system's internal language model (ILM) and an external language model (LM). They propose and validate methods for correcting this ILM, leading to improved speech recognition accuracy, and introduce a novel "exact-ILM training" framework for better theoretical grounding.

What Problem Does This Paper Address?

End-to-end (E2E) speech recognition models, like RNN-Transducers (RNN-T), learn an internal language model (ILM) implicitly from the acoustic training data. While powerful, this ILM often has a different statistical distribution than external LMs trained on much larger text corpora. This mismatch limits the effectiveness of common LM integration techniques, such as shallow fusion, which simply combine the probabilities from the E2E model and the external LM. The paper aims to bridge this gap by exploring and improving methods for correcting the ILM to better align with the external LM, thereby boosting overall speech recognition performance.

Methods

The authors investigate ILM correction-based methods within a unified RNN-T framework. Their approach involves:

  • Decoding Interpretation: They provide a theoretical explanation for why ILM correction improves performance, identifying two key reasons: rebalancing the RNN-T's label distribution to rely more on the external LM, and boosting label probabilities against the blank symbol to increase the external LM's influence without excessive deletion errors.
  • Exact-ILM Training: They propose a novel training framework that extends previous work (Hybrid Autoregressive Transducer - HAT) to provide a theoretical justification for ILM approaches. This "exact-ILM training" aims to make the ILM estimation more precise.
  • Systematic Comparison: They evaluate various ILM correction methods, including density ratio, h_zero, h_avg, and h_mini-LSTM, under different training regimes (ILM training - ILMT, and exact-ILM training - L_exact).

Datasets

  • Librispeech Corpus: Used for in-domain evaluation.
  • TED-LIUM Release 2 (TLv2) Corpus: Used for out-of-domain evaluation.

The paper also mentions using 5k acoustic data-driven subword modeling (ADSM) units derived from the Librispeech corpus.

Key Insights and Ideas

  • ILM Mismatch is a Real Bottleneck: The paper strongly validates that the mismatch between the internal and external LMs is a significant factor limiting performance in E2E ASR.
  • Two Key Benefits of ILM Correction: The decoding interpretation reveals that ILM correction helps by (1) rebalancing the model's reliance on the external LM and (2) boosting label probabilities to mitigate blank symbol dominance, thereby increasing the external LM's impact without introducing too many errors.
  • h_prime Methods Outperform Density Ratio: Methods that leverage parts of the RNN-T network to estimate the ILM (like h_zero, h_avg, h_mini-LSTM) generally perform better than simply training a separate density ratio model.
  • Exact-ILM Training Offers Theoretical Grounding: The proposed L_exact training framework provides a more robust theoretical basis for ILM integration, leading to further performance gains.
  • h_mini-LSTM with L_exact is State-of-the-Art: The combination of the h_mini-LSTM approach with the proposed L_exact training yields the best results across both in-domain and out-of-domain evaluations.

Key Results

  • All ILM correction approaches significantly improve upon the baseline shallow fusion (SF) on both Librispeech and TLv2 datasets.
  • The h_prime-based methods consistently outperform the density ratio approach.
  • The h_mini-LSTM approach, when trained with L_exact, achieves the best word error rate (WER) on both in-domain (Librispeech) and out-of-domain (TLv2) tasks. For example, on Librispeech dev, it achieves a WER of 1.8% compared to 3.3% for no LM and 2.0% for SF.
  • Joint training with ILM (LILMT) showed benefits on the cross-domain task (TLv2) but not on the in-domain task (Librispeech).

Limitations and Future Work

  • Tuning Scales: The paper notes that the scales (lambda1, lambda2) for ILM integration require careful tuning, and while they optimize them on dev sets, this tuning process can be sensitive.
  • Complexity of J' Design: The paper mentions that the exact-ILM training might require careful design of the J' function, which was not fully explored in this work. Future work could investigate optimal designs for J'.
  • Further Exploration of Joint Training: While LILMT was explored, the paper suggests that further investigation into joint training strategies, especially with the exact-ILM objective, could be beneficial.

Summary updated

BibTeX
@inproceedings{zhou2022language,
  author = {Wei Zhou and Wei Zhou and Zuoyun Zheng and Ralf Schlüter and Hermann Ney},
  title = {{On Language Model Integration for RNN Transducer Based Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746948},
}
Figure