On the Relation Between Internal Language Model and Sequence Discriminative Training for Neural Transducers

Wei Zhou, Zijian Yang, Wei Zhou 0043, Ralf Schlüter, Hermann NeyICASSP 2024 · DOI

TL;DR

This paper reveals a strong theoretical and empirical connection between internal language model (ILM) subtraction and sequence discriminative training (like MMI/MBR) for neural transducers. They show that sequence discriminative training effectively achieves similar goals to ILM subtraction, particularly by jointly reshaping the output probabilities of both the encoder and prediction networks.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems, especially neural transducers, often benefit from integrating external language models (ELMs) to improve accuracy. A common technique is "Shallow Fusion" (SF), which combines the transducer's output with an ELM. However, transducers implicitly learn their own "internal language model" (ILM). To avoid redundancy and potential conflicts, researchers often subtract this ILM from the ELM's contribution during decoding. This ILM subtraction has been shown to improve performance.

Simultaneously, sequence discriminative training methods like Maximum Mutual Information (MMI) and Minimum Bayes Risk (MBR) have also demonstrated significant gains in ASR. The authors noticed that both ILM subtraction and sequence discriminative training seemed to achieve similar improvements, leading to the central question: What is the precise relationship between these two techniques, and how does sequence discriminative training achieve its performance gains?

Methods

The paper explores this relationship through both theoretical derivations and extensive empirical experiments.

  • Theoretical Analysis: The authors derive the global optimum of the MMI training criterion. They show that this optimum has a mathematical form that is analogous to ILM subtraction, suggesting a fundamental connection.
  • Empirical Evaluation: They conduct experiments on the Librispeech dataset using RNN-Transducer models. They compare:
  • Baseline models trained with cross-entropy (CE).
  • CE models with ILM subtraction using various ILM estimation methods (zero-encoder, mini-LSTM, density ratio).
  • Models trained directly with MMI and MBR criteria, with and without ELM integration during training.
  • They also investigate the impact of sequence discriminative training on different parts of the neural transducer (encoder vs. prediction network) by swapping components between models trained with different criteria.

Datasets

  • Librispeech (LBS) [24]: This is the primary dataset used for both training and evaluation. The experiments are conducted on the 960-hour version of the dataset.

Key Insights and Ideas

  • Theoretical Equivalence: The global optimum of MMI training mathematically resembles ILM subtraction, providing a theoretical basis for their observed similarities.
  • Sequence Discriminative Training as ILM Suppression: Sequence discriminative training (MMI/MBR) effectively suppresses the transducer's internal language model, similar to explicit ILM subtraction. This is evident because applying ILM subtraction to already sequence-discriminative trained models yields much smaller improvements than applying it to CE-trained models.
  • Joint Effect on Encoder and Prediction Network: Sequence discriminative training doesn't just affect the prediction network (where the ILM is more directly modeled). It has a joint effect on both the encoder and the prediction network, leading to a more holistic reshaping of the output probabilities. This joint reshaping contributes to both ILM suppression and blank probability reduction.
  • ILM Estimation Nuances: The paper highlights that simpler ILM estimation methods (like the zero-encoder approach) might not fully capture the benefits of sequence discriminative training, as they might not account for the encoder's contribution to the ILM.

Key Results

  • Performance Gains: Both ILM subtraction and sequence discriminative training lead to significant improvements in Word Error Rate (WER) over baseline CE-trained models, especially when using external language models.
  • Similar Effects: Across various settings (different context sizes for transducers and LMs, MMI/MBR criteria), sequence discriminative training and ILM subtraction yield comparable performance improvements.
  • Reduced Benefit of ILM Subtraction After SDT: When sequence discriminative training is applied, the additional benefit of performing ILM subtraction during decoding becomes much smaller, reinforcing the idea that SDT already handles ILM suppression.
  • Full Network Improvement: Swapping components showed that using the full network (encoder + prediction) from MMI/MBR-trained models resulted in the best performance, demonstrating the joint effect. For example, using MMI/MBR models for both components yielded around an 11% relative improvement over the CE baseline.

Limitations and Future Work

  • Global Optimum Not Always Achieved: The paper acknowledges that in practice, the theoretical global optimum for MMI/MBR training is not always guaranteed due to factors like limited data, model interdependencies, regularization, and approximations in hypothesis space generation.
  • ILM Estimation Impact: The authors note that the effectiveness of ILM estimation methods can vary, and simpler methods might not fully capture the benefits of sequence discriminative training. They suggest that the ILM estimated from the prediction network is correlated with the encoder's contribution.
  • Future Work: The paper implicitly suggests further exploration into how to best leverage the joint effect of sequence discriminative training on both encoder and prediction networks for optimal LM integration. Understanding the interplay between different ILM estimation techniques and sequence discriminative training could also be a fruitful area.

Summary updated

BibTeX
@inproceedings{zhou2024relation,
  author = {Wei Zhou and Zijian Yang and Wei Zhou and Ralf Schlüter and Hermann Ney},
  title = {{On the Relation Between Internal Language Model and Sequence Discriminative Training for Neural Transducers}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447468},
}
Figure