A Comparative Evaluation of GMM-Free State Tying Methods for ASR

Tamás Grósz, Gábor Gosztolya, László TóthInterspeech 2017

TL;DR

This paper evaluates four methods for creating context-dependent acoustic models in speech recognition systems that use deep neural networks (DNNs) instead of traditional Gaussian Mixture Models (GMMs). They found that using decision criteria specifically designed for DNNs, rather than adapting GMM-based criteria, significantly improves speech recognition accuracy.

What Problem Does This Paper Address?

Traditional speech recognition systems relied on Gaussian Mixture Models (GMMs) to model acoustic sounds. More recently, Deep Neural Networks (DNNs) have become the state-of-the-art, offering superior performance. However, many techniques developed for GMM-based systems, particularly for creating "context-dependent tied states" (which help the model understand how sounds change based on their neighbors), are not directly optimal for DNNs. This is because these techniques often rely on assumptions about the data distribution (like Gaussians) and use metrics (like likelihood gain) that are better suited for GMMs than for the more complex, non-linear outputs of DNNs. While several new methods have been proposed to adapt state tying for DNNs, they have often been evaluated on different datasets, making direct comparison difficult. This paper aims to provide a fair, comparative evaluation of these GMM-free state tying methods on the same task and dataset.

Methods

The paper compares four approaches for state tying, all of which use the output of a context-independent (CI) neural network as input for the clustering process. The key differences lie in how this input is used:

  • DNN + Likelihood (Senior et al.): This method uses the output probabilities of the CI DNN directly as input to the standard decision tree-based state tying algorithm. This algorithm's decision criterion is based on maximizing likelihood gain, originally designed for GMMs.
  • DNN (hidden) + Likelihood (Bacchiani & Rybach): Instead of using the final output layer of the CI DNN, this approach uses the activations from the last hidden layer as input to the same standard likelihood-based state tying algorithm.
  • Kullback-Leibler Divergence (Gosztolya et al.): This method replaces the standard likelihood-based decision criterion with the Kullback-Leibler (KL) divergence. KL divergence is a measure of how one probability distribution differs from another, making it more suitable for comparing probability distributions generated by DNNs.
  • Entropy (Zhu et al.): This approach also uses a novel decision criterion, this time based on entropy. Entropy measures the uncertainty or randomness of a probability distribution, and this criterion aims to split states in a way that minimizes the entropy of the resulting clusters.

The paper also highlights their use of a "GMM-free flat start" training method for the CI DNN, which avoids any reliance on GMMs for initial alignments, ensuring a truly GMM-free pipeline.

Datasets

  • Training: The Wall Street Journal (WSJ) read speech corpus (specifically the si-284 set), an 81-hour dataset.
  • Evaluation: The WSJ eval92 and eval93 test sets were used. The eval93 set served as the development set for tuning parameters, and eval92 was the final test set.

Key Insights and Ideas

  • GMM assumptions are suboptimal for DNNs: The core insight is that directly applying GMM-based state tying methods (even with DNN outputs as input) is not ideal because GMMs have specific assumptions about data distributions and use metrics that don't align well with DNN characteristics.
  • DNN-specific criteria are crucial: Methods that replace the traditional likelihood-based decision criterion with metrics like KL divergence or entropy consistently outperform those that stick to the original GMM-based criterion. This suggests that how we measure the "goodness" of a split is as important as the input data used.
  • Input matters, but criterion matters more: While using DNN outputs (or hidden activations) as input is a necessary step, the real performance gains come from adapting the decision criterion itself to be GMM-free.
  • Balanced class distribution is beneficial: The paper notes that the Senior et al. method (DNN output + Likelihood) resulted in the most balanced distribution of tied states, though it didn't yield the best WER. This hints at a potential trade-off between state distribution balance and overall recognition accuracy.

Key Results

  • The most basic approach (DNN outputs + standard likelihood criterion) yielded the highest Word Error Rate (WER) on both development (8.70%) and test (6.47%) sets.
  • Using hidden layer activations with the standard criterion improved test set performance slightly (6.04% WER).
  • The GMM-free methods, using KL divergence and entropy-based criteria, significantly outperformed the others. They achieved WERs of 5.72% (KL-divergence) and 5.92% (Entropy) on the test set.
  • The Kullback-Leibler divergence-based method achieved the lowest WER on the test set (5.72%), representing a substantial relative improvement of 12% compared to the baseline.
  • The paper also notes that the GMM-free methods achieved optimal performance with fewer tied states (around 2400) compared to the likelihood-based methods, which continued to improve with more states. This suggests greater efficiency.

Limitations and Future Work

  • Limited number of methods compared: While the paper compares four prominent methods, it acknowledges that other approaches exist and were not included in this specific comparative study.
  • Dataset specificity: The results are specific to the WSJ corpus and the chosen DNN architecture. Further validation on different datasets and architectures would be beneficial.
  • Future work: The authors suggest that further research could explore other GMM-free decision criteria and investigate how these methods interact with different DNN architectures and training schemes. The observation that the KL-divergence method achieved optimal performance with fewer states also opens avenues for research into more efficient state tying.

Summary updated

BibTeX
@inproceedings{grosz2017comparative,
  author = {Tamás Grósz and Gábor Gosztolya and László Tóth},
  title = {{A Comparative Evaluation of GMM-Free State Tying Methods for ASR}},
  booktitle = {Interspeech},
  year = {2017},
}
Figure