Acoustic Modeling with Bootstrap and Restructuring Based on Full Covariance

Bowen Zhou, Xiaodong Cui, Xin Chen, Jian Xue, Peder A. Olsen, John R. Hershey, Bowen Zhou 0006Interspeech 2011 · DOI

Abstract

Acoustic modeling for low-resource languages often faces a trade-off between model complexity and data availability. This paper presents an enhanced "Bootstrap and Restructuring" (BSRS) framework that utilizes full covariance matrices during the model-building phase. By preserving richer structural information during intermediate clustering and refinement steps before converting back to efficient diagonal covariances, the proposed method achieves significant accuracy gains in low-resource scenarios without increasing the final computational cost of decoding.

Introduction

Acoustic modeling for low-resource languages is notoriously difficult because there is often insufficient data to train robust, complex models. A previously established technique, Bootstrap and Restructuring (BSRS), addresses this by training multiple models on data subsets (bootstrapping), aggregating them into a single large model, and then "restructuring" (shrinking) that model back to a manageable size.

However, earlier iterations of BSRS relied on diagonal covariance matrices for simplicity and speed. Diagonal matrices assume that different acoustic features are independent, an assumption that rarely holds true in real-world speech data. This paper addresses the resulting loss of "structural information" by performing the restructuring process in a full covariance space, capturing feature correlations to build a more accurate representative model before simplifying it for production use.

Methodology

The proposed pipeline transitions from a high-capacity aggregate model to an efficient diagonal model through several stages of full-covariance processing.

1. Bootstrap & Aggregation

The training data is split into $N$ subsets using "subbagging" (sampling without replacement). An HMM is trained for each subset, with all models sharing a single decision tree. These are averaged to create a massive "aggregated" HMM with a very large number of Gaussian components.

2. Full Covariance Restructuring

This stage is the core of the innovation. The massive model is scaled down using clustering and refinement within a full covariance framework.

The full covariance model restructuring pipeline, illustrating the transition from the initial FC HMM to the final DC HMM.
  • Gaussian Clustering: The authors compared three distance metrics to decide which Gaussians to merge: KL divergence, Entropy, and Bayes Error (approximated via the Chernoff function and Newton-based optimization).
  • Model Refinement (MCKLGMM): A Monte Carlo-based KL minimization is used to fine-tune the reduced model so it closely mimics the original large aggregate model.

3. Conversion and Final Refinement

The full covariance model is converted to a diagonal covariance model to meet the requirements for efficient real-time decoding. A final refinement step (MCKLHMM) is performed in the diagonal space to ensure optimal performance.

Experimental Setup

The approach was evaluated on Pashto, a low-resource language spoken in Afghanistan. The training set consisted of 135 hours of speech from 116 speakers, while the test set included 10 hours from 22 speakers. Features were 24-dimensional PLP features projected to 40 dimensions using LDA and Semi-Tied Covariance (STC).

Results and Discussion

The proposed full-covariance BSRS consistently outperformed both the baseline and the previous diagonal-only BSRS.

Comparison of Word Error Rates (WER) across different modeling spaces and restructuring methods.

As shown in the results, the baseline Maximum Likelihood model achieved a 39.6% Word Error Rate (WER). The previous Diagonal BSRS improved this to 38.6%. The proposed Full Covariance BSRS reached 38.1% WER, representing a 1.5% absolute improvement over the baseline.

When combined with advanced discriminative training techniques like FMMI and BMMI, the proposed method reached 33.2% WER, compared to the baseline's 34.1%. These results demonstrate that even after converting back to diagonal covariance, the "memory" of the full covariance structure during training leads to a superior final model.

Conclusion

The core insight of this work is that diagonal covariances act as an information bottleneck during training. By using full covariances during restructuring, the algorithm makes better decisions about which acoustic components are truly redundant. Because the model is converted back to diagonal covariance at the end, the final speech recognizer remains as fast as a standard one, shifting the complexity entirely to the offline training phase. Future work will explore if these gains scale further for extremely low-resource languages with less than 10 hours of data.

Summary updated

BibTeX
@inproceedings{zhou2011acoustic,
  author = {Bowen Zhou and Xiaodong Cui and Xin Chen and Jian Xue and Peder A. Olsen and John R. Hershey and Bowen Zhou},
  title = {{Acoustic Modeling with Bootstrap and Restructuring Based on Full Covariance}},
  booktitle = {Interspeech},
  year = {2011},
  doi = {10.21437/interspeech.2011-185},
}
Figure