Semi-supervised training of Gaussian mixture models by conditional entropy minimization

Jui-Ting Huang, Mark Hasegawa-JohnsonInterspeech 2010 · DOI

Introduction

In the field of speech recognition, the availability of raw audio data far outstrips the resources available for manual transcription. Semi-supervised learning (SSL) offers a pathway to leverage this vast pool of unlabeled data alongside limited labeled sets. This paper presents a method for training Gaussian Mixture Models (GMMs) using a combination of Maximum Mutual Information (MMI) and conditional entropy minimization (CE), termed MMI-CE. By encouraging the model to produce high-confidence predictions on unlabeled data, the approach achieves notable performance gains in phonetic classification.

Methodology

The core of the proposed approach is a new objective function that integrates discriminative training with a regularizer for unlabeled data.

The MMI-CE Objective

The objective function $J$ is defined as: $$J = F_{MMI} - \alpha H_{emp}$$ where $F_{MMI}$ is the standard MMI criterion applied to labeled data, maximizing the log-posterior probability of correct labels. The second term, $H_{emp}$, represents the empirical conditional entropy of the model's predictions on the unlabeled data. The parameter $\alpha$ serves as a weighting factor to balance these two components.

Minimizing conditional entropy forces the model to assign high probability to a single class for each unlabeled sample, effectively pushing the decision boundaries into "low-density" regions of the feature space. This heuristic assumes that classes are typically separated by regions with fewer data points.

Optimization

To optimize this non-convex objective, the authors employ a preconditioned conjugate gradient method. Efficiency is achieved by approximating the Hessian using the GMM's covariance matrices, leading to stable convergence within approximately 50 iterations.

Experimental Setup

The method was evaluated on the TIMIT acoustic-phonetic corpus. The task involved phonetic classification into 39 categories using 61-dimensional feature vectors (PLP coefficients and duration). To simulate real-world data scarcity, the researchers used varying subsets of the training data as "labeled" (ranging from 10% to 100%), treating the remainder as "unlabeled."

Results and Discussion

Performance Gains

The MMI-CE framework consistently outperformed both standard Maximum Likelihood (ML) and MMI baselines. The benefit of the conditional entropy regularizer is most pronounced when labeled data is scarce.

Comparison of Phone Accuracy across different label percentages for ML, MMI, and MMI+CE.

As shown in the figure above, MMI+CE provides a consistent 1-2% accuracy boost over standard MMI when the percentage of labels used is between 10% and 60%. This demonstrates the effectiveness of utilizing unlabeled data to sharpen class distinctions.

Parameter Sensitivity

A critical finding for practical application is the robustness of the model to the tuning parameter $\alpha$.

Phone Accuracy vs. tuning parameter alpha for various labeled data proportions.

The experimental results indicate that performance remains stable across a wide range of $\alpha$ values (from 6 to 14). This lack of sensitivity simplifies the deployment process, as it reduces the need for extensive hyperparameter optimization.

Conclusion

This work demonstrates that conditional entropy minimization is a powerful tool for semi-supervised GMM training. By integrating this principle into a discriminative framework, the authors provide a theoretically grounded and effective method for improving phonetic classification when labeled data is limited. While the focus was on GMMs and phonetic classification, the underlying principle of entropy minimization remains a cornerstone of modern semi-supervised learning strategies.

Summary updated

BibTeX
@inproceedings{huang2010semi,
  author = {Jui-Ting Huang and Mark Hasegawa-Johnson},
  title = {{Semi-supervised training of Gaussian mixture models by conditional entropy minimization}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-19},
}
Figure