Posterior-scaled MPE: novel discriminative training criteria

Markus Nussbaum-Thom, Zoltan Tuske, Georg Heigold, Ralf Schlüter, Hermann NeyInterspeech 2012

TL;DR

This paper introduces a novel discriminative training criterion for Automatic Speech Recognition (ASR) called Posterior-Scaled Minimum Phoneme Error (K-MPE). By deriving this criterion from theoretical error bounds, the authors provide a stronger justification for its effectiveness and demonstrate that it outperforms existing state-of-the-art methods like MPE on the TIMIT dataset.

What Problem Does This Paper Address?

In Automatic Speech Recognition (ASR), achieving high accuracy often requires moving beyond standard Maximum Likelihood (ML) training. Discriminative training criteria aim to directly optimize for performance metrics like word error rate, which are more relevant to the end goal than simply maximizing the probability of the training data. Existing discriminative training methods, such as Maximum Mutual Information (MMI) and Minimum Phoneme Error (MPE), have been successful but lack a direct theoretical link to fundamental error bounds. This paper addresses this gap by developing new discriminative training criteria, including K-MPE, that are rigorously derived from theoretical error bounds on the mismatch between the model's decision rule and the optimal Bayes decision rule, especially for non-trivial loss functions like the Levenshtein distance used in ASR.

Methods

The core contribution lies in the theoretical derivation of novel discriminative training criteria from error bounds on the global error for pattern classification tasks. Specifically, the authors derive upper bounds on the mismatch between the model-based decision rule's error and the Bayes error. They then introduce the Posterior-Scaled Minimum Phoneme Error (K-MPE) criterion as an approximation to one of these novel criteria, providing a theoretical link to error bounds for MPE for the first time.

For practical implementation, the paper details how the K-MPE criterion is integrated into a transducer-based framework. This involves extending the Eisner expectation semiring to handle the K-scaled expectation semiring, which allows for efficient calculation of the criterion's gradient. The training criteria are then optimized using the RPROP algorithm.

Datasets

The experiments were conducted on the TIMIT dataset for the ASR task. The paper provides statistics for both the training and test sets, including duration, number of utterances, words, and frames. No new benchmarks were introduced; the standard TIMIT dataset was used for evaluation.

Key Insights and Ideas

  • Theoretical Justification for MPE: The paper provides the first theoretical justification for the effectiveness of the MPE criterion by linking it to fundamental error bounds. This moves beyond empirical observation to a principled understanding.
  • Novel Criteria from Error Bounds: The derivation of new discriminative training criteria (WMMI, WSE, and K-MPE) directly from error bounds offers a more robust theoretical foundation for improving ASR performance.
  • K-MPE as a Superior Alternative: The K-MPE criterion, particularly for specific values of κ (e.g., 0.6), demonstrates superior performance compared to standard MPE and other discriminative training methods.
  • Transducer Framework Efficiency: The implementation of K-MPE within a transducer-based framework using an extended expectation semiring highlights the practical feasibility and computational efficiency of these novel criteria.

Key Results

The experimental results on the TIMIT test set show that the novel discriminative training criteria offer improvements over the baseline ML model. Specifically:

  • The modified MMI and MPE criteria achieved a relative improvement of 5% in Word Error Rate (WER) compared to the ML model.
  • The 0.6-MPE criterion performed even better, achieving a relative improvement of 6% over the modified MPE criterion.
  • Figure 1 illustrates the WER progress for different κ values, showing that 0.6-MPE and 0.8-MPE significantly outperform standard MPE (1.0-MPE), while 0.4-MPE performs worse, suggesting a sensitivity to the choice of κ.
  • Table 2 summarizes the WER results, with ML at 32.0%, MMI at 30.6%, MPE at 30.6%, WSE at 31.2%, WMMI at 31.7%, and the best performing 0.6-MPE at 30.1%.

Limitations and Future Work

  • Parameter Sensitivity (κ): The performance of the K-MPE criterion is sensitive to the choice of the scaling parameter κ. While 0.6-MPE showed the best results, the paper notes that κ=0.4 performed worse, possibly due to being closer to the numerically unstable WMMI criterion (κ=0). Further research could explore more robust methods for selecting or adapting κ.
  • Margin Term Integration: The paper briefly mentions that modified criteria can be derived by augmenting the joint distribution with a margin term, similar to support vector machines. This suggests a promising avenue for future work to further enhance performance by incorporating margin-based regularization.
  • Computational Cost: While the paper discusses efficient implementation, the computational cost of training with these novel criteria, especially for very large datasets, might still be a consideration.

Summary updated

BibTeX
@inproceedings{nussbaumthom2012posterior,
  author = {Markus Nussbaum-Thom and Zoltan Tuske and Georg Heigold and Ralf Schlüter and Hermann Ney},
  title = {{Posterior-scaled MPE: novel discriminative training criteria}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure