Lattice-Free Sequence Discriminative Training for Phoneme-Based Neural Transducers

Wei Zhou, Zijian Yang, Wei Zhou 0043, Ralf Schlüter, Hermann NeyICASSP 2023 · DOI

TL;DR

This paper introduces novel "lattice-free" training objectives for phoneme-based Neural Transducer (RNN-T) models. By eliminating the computationally expensive decoding step typically required for discriminative training, these methods achieve significant speedups (40-70%) while maintaining competitive performance, and can even improve convergence speed when replacing standard cross-entropy training.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems have seen remarkable progress with sequence-to-sequence models like RNN-Transducers. While discriminative training methods (like MMI, MPE, MBR) are known to boost performance in traditional hybrid ASR systems, applying them to RNN-Ts has been challenging. These methods typically require generating lattices or N-best lists of possible output sequences during training to compute discriminative objectives. This decoding step is computationally intensive and resource-heavy, making training slow and expensive. Previous "lattice-free" approaches have been explored for hybrid and CTC models, but their direct application to the final output layer of RNN-Ts, especially those with limited context dependency, was not well-established. This paper aims to bridge this gap by proposing and evaluating lattice-free discriminative training objectives specifically for phoneme-based RNN-Ts.

Methods

The core contribution is the proposal of three lattice-free (LF) discriminative training objectives:

  • LF-MMI (Lattice-Free Maximum Mutual Information): Adapts the MMI objective to avoid explicit lattice generation.
  • LF-SegMBR (Lattice-Free Segment-Level Minimum Bayes Risk): A novel approach for MBR that computes costs locally per frame, making it compatible with lattice-free methods. It introduces a cost function that considers both label emission penalties and a smoothed cost over a window to allow for minor alignment shifts.
  • LF-MBR (Lattice-Free Label-based Minimum Bayes Risk): Another MBR variant that operates at the output label sequence level. It uses a phoneme-level Hamming distance with a smoothing window as the risk function, avoiding the need for word-level alignments which are difficult to obtain locally.

All these LF methods leverage dynamic programming (DP) to compute the necessary summations for the denominator of the objective functions, eliminating the need for an explicit decoding step. This allows for more efficient training. The paper also explores integrating these LF objectives with a phoneme-based LM.

Datasets

The experiments were conducted on the Librispeech (LBS) dataset, a widely used benchmark for ASR. The paper uses the 960-hour version of Librispeech for training and evaluation.

Key Insights and Ideas

  • Lattice-free discriminative training is feasible and beneficial for RNN-Ts: The paper demonstrates that it's possible to apply powerful discriminative training techniques to the final output layer of phoneme-based RNN-Ts without the computational overhead of lattice generation.
  • Significant training speedups: Eliminating the decoding step drastically reduces training time, making discriminative training more accessible.
  • LF-MMI can mitigate the "dominant blank" issue: Unlike N-best list methods, LF-MMI, when integrated with an LM, tends to favor longer sequences in the denominator, which helps to reduce the over-reliance on blank emissions.
  • LF-SegMBR and LF-MBR offer flexible cost functions: The proposed cost functions for MBR are designed to be compatible with lattice-free computation and address specific issues like multiple emissions of the same label within a segment.
  • LF-MMI can replace standard CE training: The paper shows that LF-MMI can be used as an alternative to cross-entropy (CE) training, leading to faster convergence and slightly better performance.

Key Results

  • Performance Gains: Lattice-free methods achieved up to 6.5% relative improvement in Word Error Rate (WER) compared to a sequence-level cross-entropy trained model.
  • Training Speedup: LF methods provided a significant training time speedup of 40% - 70% compared to N-best list based minimum Bayes risk objectives, with only a small degradation in performance.
  • Convergence: When LF-MMI replaced standard CE-FS training, the model converged with fewer epochs.
  • Specific Improvements: LF-MMI showed fewer deletion errors compared to CE-FS. LF-SegMBR's risk function effectively assigns higher probabilities to correct labels. LF-MBR, despite using Hamming distance, provided useful cost information.

Limitations and Future Work

  • Hamming Distance Approximation: The paper acknowledges that Hamming distance used in LF-MBR is not a perfect approximation of Levenshtein distance, which is the standard metric for performance measurement.
  • Pruning Factor: The use of a pruning factor (γ) in LF-MBR is a heuristic to manage computational complexity and memory constraints.
  • Future Work: The paper suggests that further exploration of cost functions for MBR-based objectives and investigating the impact of different LM integration strategies could be beneficial. The authors also implicitly suggest that these LF methods could be applied to other sequence-to-sequence architectures.

Summary updated

BibTeX
@inproceedings{zhou2023lattice,
  author = {Wei Zhou and Zijian Yang and Wei Zhou and Ralf Schlüter and Hermann Ney},
  title = {{Lattice-Free Sequence Discriminative Training for Phoneme-Based Neural Transducers}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10094609},
}

Loading related content...

Figure