Optimization-based control for the extended baum-welch algorithm

Janne Pylkkönen, Mikko KurimoInterspeech 2012

TL;DR

This paper introduces a novel optimization-based method for controlling the Extended Baum-Welch (EBW) algorithm, a crucial component in discriminative training for speech recognition. By optimizing smoothing parameters on a held-out control set, the authors demonstrate significant improvements in speech recognition accuracy, particularly in challenging mismatched conditions, compared to traditional heuristic approaches.

What Problem Does This Paper Address?

Discriminative training of acoustic models for speech recognition aims to optimize models directly for recognition accuracy, rather than just likelihood. The Extended Baum-Welch (EBW) algorithm is a popular method for this, but it relies on several "smoothing" or "control" parameters (often denoted as D constants). Historically, these parameters have been set using heuristic rules, which are often ad-hoc and may not yield optimal performance. The challenge is to find a principled, data-driven way to set these parameters that leads to better acoustic models.

Methods

The core contribution is an optimization-based control procedure for EBW. Instead of relying on heuristics, the authors propose to optimize the EBW control parameters (D constants) numerically using a held-out control set. The process involves:

  1. Collecting Statistics: Gathering discriminative statistics from the training data.
  2. Model Estimation: Using the current EBW constants and training data statistics to estimate a new acoustic model.
  3. Control Set Statistics: Computing discriminative statistics over a separate, held-out control set using the newly estimated model.
  4. Parameter Optimization: Updating the EBW constants to maximize the discriminative criterion (e.g., SNFE) evaluated on the control set. This is done using a gradient-based optimizer (Quickprop).
  5. Iteration: Repeating steps 2-4 until convergence or a maximum number of iterations is reached.

To manage complexity and prevent over-training, the authors explore different levels of clustering for the D constants (global, mixture-level, Gaussian-level). They also introduce a logarithmic transformation for the parameters to ensure positive definiteness of covariance matrices and a continuous objective function. A prior is added to the derivative to encourage smaller control parameter values, which can lead to larger step sizes in the acoustic model updates.

Datasets

The experiments were conducted using the Speecon corpus.

  • Training Data: Clean sentences from 310 speakers (approx. 15h of speech).
  • Development/Evaluation Sets: 40 speakers each, approx. 1100 utterances (1.9h of speech).
  • Control Set: A subset of the training data (approx. 12% of training data, about 1100 utterances) was used as a held-out set for optimization. For mismatched scenarios, a small amount of target data was used for both training and as the control set.

Key Insights and Ideas

  • Data-Driven Control is Superior: The paper strongly suggests that optimizing EBW control parameters directly on a held-out set yields better results than relying on traditional heuristic rules.
  • Benefits in Mismatched Conditions: The proposed method shows particularly significant gains when there's a mismatch between training and target data, especially when a small amount of target data is available for training. This highlights the robustness of the optimization approach.
  • Clustering Matters: Different levels of clustering for the D constants (global, mixture, Gaussian) offer trade-offs between performance and computational cost. Gaussian-level optimization, while potentially powerful, can be prone to over-training with small control sets.
  • Logarithmic Transformation for Stability: Using a logarithmic transformation for the D parameters helps ensure numerical stability and proper handling of covariance matrix constraints.
  • Prior for Step Size Control: Adding a prior to the optimization objective encourages larger acoustic model updates, which can be beneficial for convergence.

Key Results

The paper reports significant improvements in Letter Error Rate (LER) on both development and evaluation sets.

  • Clean/Matched Task: Small improvements were observed compared to heuristics, with methods like "Fixed D" and "Mixture clustering" showing benefits.
  • Noisy/Mismatched Task: More substantial gains were seen. For MMI on the noisy task, global parameter clustering showed a 25% LER reduction compared to heuristics and a 5% reduction compared to a fixed D approach. The mixture-clustered control method achieved the best performance with a 9% LER reduction compared to heuristics.
  • Statistical Significance: Many of the observed improvements were statistically significant according to the Wilcoxon signed rank test.

Limitations and Future Work

  • Computational Cost: The optimization-based control procedure roughly doubles the computation required for discriminative training due to the iterative optimization of control parameters.
  • Overtraining Risk: Using a small control set, especially in mismatched scenarios, increases the risk of over-training.
  • Future Work: The authors suggest exploring optimal clustering strategies and constraint settings. They also aim to reduce the computational overhead associated with the control set optimization.

Summary updated

BibTeX
@inproceedings{pylkkonen2012optimization,
  author = {Janne Pylkkönen and Mikko Kurimo},
  title = {{Optimization-based control for the extended baum-welch algorithm}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure