Restructuring exponential family mixture models

Pierre L. Dognin, John R. Hershey, Vaibhava Goel, Peder A. OlsenInterspeech 2010 · OpenReview

Abstract

This paper introduces a mathematical framework for the efficient restructuring of complex Automatic Speech Recognition (ASR) models without significant loss in accuracy. By extending variational techniques from standard Gaussian models to the broader Exponential Family Mixture Models (EMMs), the authors provide a method to cluster and approximate large "reference" models into smaller, more computationally efficient "restructured" models. The approach is particularly effective for models incorporating advanced features like phone posterior probabilities, demonstrating superior performance over traditional Gaussian Mixture Models (GMMs) even at reduced scales.

Introduction

In Automatic Speech Recognition (ASR), mixture models are essential for representing the complex distributions of speech sounds. While increasing the number of components typically improves accuracy, it simultaneously raises the computational overhead in terms of memory and processing speed. Model restructuring aims to approximate a large, high-accuracy reference model with a smaller, more efficient one. A primary challenge in this process is measuring the similarity between mixture models, as calculating the Kullback-Leibler (KL) divergence is mathematically intractable. While solutions exist for GMMs, modern ASR often utilizes non-Gaussian features, necessitating a more flexible framework within the Exponential Family.

Methods

The authors bridge the gap in model restructuring by deriving variational approximations specifically for EMMs. Their methodology consists of three core components:

  1. Variational KL (varKL): Utilizing Jensen’s inequality, the authors create a computable lower-bound approximation of the true KL divergence.
  2. Variational EM (varEM): An extension of the standard Expectation-Maximization algorithm to EMMs, allowing a smaller model to iteratively update its parameters to mimic a larger reference model.
  3. Weighted Local Maximum Likelihood (wLML): A greedy clustering algorithm that merges components in a large model based on minimizing the loss of variational likelihood.

These techniques were evaluated using a combined distribution of standard audio features (fMMI-PLP) and Sparse representation Phone Identification Features (SPIF), which represent log-probabilities of specific speech sounds.

Experimental Setup

The models were trained on 50 hours of English Broadcast News (BN) speech corpora (LDC97S44, LDC98S71). Evaluation was conducted on the EARS Dev-04f test set, comprising 3 hours of audio. The reference models initially contained 100,000 (100K) components and were systematically clustered down to sizes as small as 10,000 (10K) components.

Results and Discussion

The experimental results highlight the superiority of EMMs over standard GMM baselines. As shown in the performance data, EMMs utilizing SPIF features (+logP) consistently outperformed GMMs across all model sizes. At the 100K component level, the EMM achieved a Word Error Rate (WER) of 19.2%, compared to 20.4% for the GMM baseline.

Word Error Rate (WER) performance for various model restructuring techniques across different component counts.

One of the most significant findings is the "graceful degradation" of the EMMs. When the 100K EMM was restructured down to 10K components using the wLML approach, the WER only increased to 20.6%. In comparison, the baseline GMM at the same 10K size yielded a much higher WER of 23.0%. This indicates that the variational restructuring successfully preserves the critical information provided by phone posteriors even under heavy compression. Furthermore, the wLML approach generally outperformed simple symmetric KL divergence for model reduction, particularly at smaller target sizes.

Conclusion

This work demonstrates that Exponential Family Mixture Models provide a robust unified framework for modeling diverse data types in ASR. The proposed variational restructuring techniques, specifically wLML, allow for significant model size reduction with minimal impact on accuracy. Future research may explore optimizing component assignments to speech states during clustering and applying these techniques to models trained with discriminative criteria.

Summary updated

BibTeX
@inproceedings{dognin2010restructuring,
  author = {Pierre L. Dognin and John R. Hershey and Vaibhava Goel and Peder A. Olsen},
  title = {{Restructuring exponential family mixture models}},
  booktitle = {Interspeech},
  year = {2010},
  url = {https://openreview.net/forum?id=-1G51rta32z},
}
Figure