Refactoring acoustic models using variational expectation-maximization

Pierre L. Dognin, John R. Hershey, Vaibhava Goel, Peder A. OlsenInterspeech 2009 · OpenReview

In modern speech recognition, acoustic models often rely on thousands of Gaussian Mixture Models (GMMs). While increasing the number of Gaussian components generally improves accuracy, it also increases computational costs and memory requirements—critical constraints for mobile and embedded systems. This research from IBM presents a "refactoring" method that allows for significant model compression without sacrificing performance.

The Challenge of Model Refactoring

Refactoring involves taking a high-performing, large "reference" model and creating a smaller version that mimics its behavior. A common goal is to minimize the Kullback-Leibler (KL) divergence between the large and small models. However, calculating KL divergence between two GMMs is mathematically intractable, making direct optimization difficult. Simply training a smaller model from scratch on raw data often fails to capture the nuances learned by the larger reference model.

Methodology: varEM and wLML

The researchers address this intractability through a two-pronged approach:

  1. Variational Expectation-Maximization (varEM): By maximizing a variational lower bound instead of the exact KL divergence, the authors use an EM algorithm to iteratively update the smaller model's parameters to match the reference distribution.
  2. Weighted Local Maximum Likelihood (wLML): This is the core innovation. Standard EM is highly sensitive to initialization. The authors propose a greedy clustering approach using a generalized KL divergence that accounts for the "weight" (prior probability) of each Gaussian component. This ensures that the most statistically significant parts of the original distribution are preserved during the initial clustering phase.
Figure 3: WER (%) vs. Number of Gaussians (K) comparing various refactoring methods against training from data.

Experimental Results

The method was tested on an internal IBM database of 786 hours of US English in-car speech. The reference model contained 100,000 Gaussian components.

Key Findings

  • Superior Initialization: The proposed wLML initialization provided a 24% relative improvement over standard LML. At the 5,000 component level, wLML achieved a 23% improvement in Word Error Rate (WER) over standard LML.
  • Effective Compression: The authors successfully reduced the 100K model to 50K components while maintaining the same recognition performance as a 50K model trained directly from data.
  • Refactoring vs. Training: For mid-range model sizes (40K–95K Gaussians), refactoring actually yielded better results than training a model of that size from scratch. Refactoring is also an order of magnitude faster than full training, taking hours instead of days.
Figure 5: Detailed WER (%) comparison between the baseline (trained from data) and the refactored 100K-STC model across various sizes.

Conclusion

This variational framework demonstrates that large acoustic models can be effectively "shrunk" by 50% with negligible impact on accuracy. While a performance gap remains for extremely small models (below 25K components), the varEM and wLML approach provides a robust and efficient tool for rapid model iteration and deployment in resource-constrained environments. Future work may explore more global optimization methods to further bridge the gap for ultra-compact models.

Summary updated

BibTeX
@inproceedings{dognin2009refactoring,
  author = {Pierre L. Dognin and John R. Hershey and Vaibhava Goel and Peder A. Olsen},
  title = {{Refactoring acoustic models using variational expectation-maximization}},
  booktitle = {Interspeech},
  year = {2009},
  url = {https://openreview.net/forum?id=qmvLkMbjfLw},
}
Figure