Abstract
Acoustic modeling for low-resource languages often faces a trade-off between model complexity and data availability. This paper presents an enhanced "Bootstrap and Restructuring" (BSRS) framework that utilizes full covariance matrices during the model-building phase. By preserving richer structural information during intermediate clustering and refinement steps before converting back to efficient diagonal covariances, the proposed method achieves significant accuracy gains in low-resource scenarios without increasing the final computational cost of decoding.
Introduction
Acoustic modeling for low-resource languages is notoriously difficult because there is often insufficient data to train robust, complex models. A previously established technique, Bootstrap and Restructuring (BSRS), addresses this by training multiple models on data subsets (bootstrapping), aggregating them into a single large model, and then "restructuring" (shrinking) that model back to a manageable size.
However, earlier iterations of BSRS relied on diagonal covariance matrices for simplicity and speed. Diagonal matrices assume that different acoustic features are independent, an assumption that rarely holds true in real-world speech data. This paper addresses the resulting loss of "structural information" by performing the restructuring process in a full covariance space, capturing feature correlations to build a more accurate representative model before simplifying it for production use.
Methodology
The proposed pipeline transitions from a high-capacity aggregate model to an efficient diagonal model through several stages of full-covariance processing.
1. Bootstrap & Aggregation
The training data is split into $N$ subsets using "subbagging" (sampling without replacement). An HMM is trained for each subset, with all models sharing a single decision tree. These are averaged to create a massive "aggregated" HMM with a very large number of Gaussian components.
2. Full Covariance Restructuring
This stage is the core of the innovation. The massive model is scaled down using clustering and refinement within a full covariance framework.
- Gaussian Clustering: The authors compared three distance metrics to decide which Gaussians to merge: KL divergence, Entropy, and Bayes Error (approximated via the Chernoff function and Newton-based optimization).
- Model Refinement (MCKLGMM): A Monte Carlo-based KL minimization is used to fine-tune the reduced model so it closely mimics the original large aggregate model.
3. Conversion and Final Refinement
The full covariance model is converted to a diagonal covariance model to meet the requirements for efficient real-time decoding. A final refinement step (MCKLHMM) is performed in the diagonal space to ensure optimal performance.
Experimental Setup
The approach was evaluated on Pashto, a low-resource language spoken in Afghanistan. The training set consisted of 135 hours of speech from 116 speakers, while the test set included 10 hours from 22 speakers. Features were 24-dimensional PLP features projected to 40 dimensions using LDA and Semi-Tied Covariance (STC).
Results and Discussion
The proposed full-covariance BSRS consistently outperformed both the baseline and the previous diagonal-only BSRS.
As shown in the results, the baseline Maximum Likelihood model achieved a 39.6% Word Error Rate (WER). The previous Diagonal BSRS improved this to 38.6%. The proposed Full Covariance BSRS reached 38.1% WER, representing a 1.5% absolute improvement over the baseline.
When combined with advanced discriminative training techniques like FMMI and BMMI, the proposed method reached 33.2% WER, compared to the baseline's 34.1%. These results demonstrate that even after converting back to diagonal covariance, the "memory" of the full covariance structure during training leads to a superior final model.
Conclusion
The core insight of this work is that diagonal covariances act as an information bottleneck during training. By using full covariances during restructuring, the algorithm makes better decisions about which acoustic components are truly redundant. Because the model is converted back to diagonal covariance at the end, the final speech recognizer remains as fast as a standard one, shifting the complexity entirely to the offline training phase. Future work will explore if these gains scale further for extremely low-resource languages with less than 10 hours of data.
Summary updated