Dynamic threshold setting via Bayesian information criterion (BIC) in HMM training

Ying Jia, Yonghong Yan, Baosheng YuanInterspeech 2000 · OpenReview

TL;DR

This paper introduces a novel approach to automatically tune crucial thresholds in Hidden Markov Model (HMM) training using the Bayesian Information Criterion (BIC). By dynamically setting these thresholds, the authors achieve significant improvements in speech recognition accuracy for Chinese dictation tasks, outperforming traditional fixed-threshold methods.

What Problem Does This Paper Address?

Training HMMs for speech recognition involves several critical decisions that traditionally rely on manually set, heuristic thresholds. Two key areas are:

  1. Decision Tree State Tying: HMMs are often built using decision trees to share parameters among similar phonetic states, especially for triphone models. Growing these trees involves deciding whether to split a node. A fixed threshold determines if the potential gain in model likelihood from splitting outweighs the increased complexity (more parameters). This heuristic approach can be suboptimal.
  2. Gaussian Mixture Splitting (State Mixing-Up): Each HMM state can be modeled by multiple Gaussian distributions to capture acoustic variations. Deciding how many Gaussians to use for each state is crucial; too few limit model expressiveness, while too many lead to overfitting. Again, this is typically managed with a fixed threshold based on the number of training samples assigned to a state.

The problem is that these fixed thresholds are not adaptive to the underlying complexity of the data or the specific phonetic context, potentially leading to suboptimal model structures and performance.

Methods

The core innovation is the application of the Bayesian Information Criterion (BIC) as a dynamic thresholding mechanism in two key HMM training stages:

  • Decision Tree State Tying: Instead of a constant threshold, BIC is used to decide whether to split a node in the decision tree. The criterion balances the increase in log-likelihood achieved by splitting against the increased number of parameters, penalized by the BIC formula. This allows the tree to adapt its structure based on the data's complexity at each node.
  • Gaussian Mixture Splitting (State Mixing-Up): BIC is also employed to determine the optimal number of Gaussian mixtures for each HMM state. It compares the log-likelihood of a state with M' mixtures versus M mixtures, using a BIC-derived penalty that accounts for the number of parameters and data points. This ensures that the number of Gaussians is appropriately chosen for each state's acoustic variability.

The paper leverages existing HMM training frameworks but replaces the fixed thresholding logic with BIC-based decision-making.

Datasets

  • LVCSR Chinese Dictation Task: This is the primary dataset used for evaluation.
  • Training Set: 204,153 utterances from 516 speakers, totaling 65,547,305 frames.
  • Test Set: 110 utterances from 11 speakers (5 female, 6 male).
  • Acoustic Features: 12 mel-cepstral coefficients plus their 1st and 2nd order time derivatives.
  • Language Model: Trigram.
  • Vocabulary: 51k words.

No new benchmarks were introduced; the evaluation was performed on an existing LVCSR Chinese dictation task.

Key Insights and Ideas

  • BIC as an Adaptive Regulator: BIC, a well-established model selection criterion, proves effective in dynamically controlling model complexity during HMM training, moving beyond static, heuristic thresholds.
  • Context-Aware Model Structure: By using BIC for decision tree splitting, the HMM structure becomes more sensitive to the local complexity of phonetic contexts, leading to more refined state tying.
  • Optimal Gaussian Mixture Allocation: BIC helps avoid both under-modeling (too few Gaussians) and over-modeling (too many Gaussians) for individual HMM states, leading to better acoustic representation.
  • Performance Gains are Real: The dynamic thresholding approach demonstrably leads to significant improvements in speech recognition accuracy, validating the theoretical benefits of BIC.

Key Results

  • Baseline System: Achieved a Word Error Rate (WER) of 10.7% using constant thresholds for decision tree tying and Gaussian mixture splitting.
  • BIC for Decision Tree State Tying: Using BIC for decision tree splitting (with specific penalty values of 3.0 for splitting and 9.6 for merging) resulted in a slightly improved average WER of 10.1% on the test set, with a comparable number of states (6005 vs. 6007 in baseline). This suggests BIC can achieve similar or better state tying with potentially fewer states or without performance degradation.
  • BIC for Gaussian Mixture Splitting: The paper observed that increasing Gaussian mixtures beyond 12 for states in the baseline system did not yield further improvements with constant thresholds. However, when BIC was applied to determine the number of mixtures, it achieved a minimum WER of 9.8% at 32 mixtures per state, representing a significant improvement (nearly 9% relative reduction in WER compared to the baseline's 10.7%).

Limitations and Future Work

  • Parameter Tuning for BIC: While BIC provides a principled way to set thresholds, the specific penalty factors (e.g., $\lambda$ in the BIC formula, or the explicit values used for splitting/merging penalties) might still require some tuning, though less so than entirely heuristic constants. The paper mentions a splitting penalty of 2.0 and merger penalty of 9.6 for one experiment, and 3.0/9.6 for another, suggesting some exploration was done.
  • Computational Cost: The paper doesn't explicitly detail the computational overhead introduced by using BIC for dynamic thresholding, though it's implied to be manageable as it's applied within standard training iterations.
  • Future Work: The authors suggest that this dynamic thresholding approach can be extended to other aspects of HMM training and model selection. The success on the LVCSR Chinese dictation task implies potential for broader application in other speech recognition systems and languages.

Summary updated

Abstract

In this paper, an approach of dynamic threshold setting via Bayesian Information Criterion (BIC) in HMM training is described. The BIC threshold setting is applied to two important applications. Firstly, it is used to set the thresholds for decision tree based state tying, in place of the conventional approach of using a heuristic constant threshold. Secondly, it is applied to choosing the number of Gaussian mixture at state mixing-up stage. Experimental results on LVCSR Chinese dictation task indicate that BIC can dynamically set thresholds for cluster splitting according to the underlying complexity of the cluster parameters. Also significant performance improvement is achieved with the dynamic BIC threshold setting.
BibTeX
@inproceedings{jia2000dynamic,
  author = {Ying Jia and Yonghong Yan and Baosheng Yuan},
  title = {{Dynamic threshold setting via Bayesian information criterion (BIC) in HMM training}},
  booktitle = {Interspeech},
  year = {2000},
  url = {https://openreview.net/forum?id=JKCCzkJ6pJ},
}
Figure