Integrate template matching and statistical modeling for speech recognition

Xie Sun, Yunxin ZhaoInterspeech 2010

TL;DR

This paper introduces a novel approach to speech recognition by integrating template matching with statistical models (HMMs). By representing speech frames with GMM indices and using DTW with a log-likelihood ratio distance, they achieve significant improvements in phone recognition accuracy over traditional HMM baselines, especially when using template representatives to reduce computational cost.

What Problem Does This Paper Address?

Traditional Hidden Markov Models (HMMs) in speech recognition excel at modeling spectral variations and time dynamics but struggle with capturing the fine temporal details of non-stationary speech sounds. While template-based methods, like Dynamic Time Warping (DTW), can better capture these temporal evolutions, they traditionally require storing vast amounts of training data, making them impractical for large-scale tasks. This paper aims to bridge this gap by combining the strengths of both approaches: leveraging the statistical power of HMMs and the temporal modeling capabilities of template matching, while mitigating the storage and computational burden of pure template-based methods.

Methods

The core innovation lies in integrating template matching with statistical modeling. Here's a breakdown of their key methods:

  • Template Representation: Instead of storing raw feature vectors, speech frames are represented by their top n Gaussian Mixture Model (GMM) indices and associated weights. This significantly reduces the storage footprint.
  • Template Construction: Tied triphone units (a concept from HMMs) are used as template units. HMMs are used to align training data and obtain phone boundaries, which then define the phone templates.
  • Hierarchical Clustering for Representatives: To further reduce storage and computation, hierarchical agglomerative clustering is employed to group similar templates within each tied triphone cluster. This process generates "template representatives," which are a smaller, representative subset of the original templates.
  • Log-Likelihood Ratio (LLR) Distance: A novel local distance measure for DTW is introduced. Instead of directly comparing feature vectors, the LLR distance compares how well a test frame fits its best GMM model versus how well it fits the best GMM model of a template frame. This provides an indirect but effective measure of frame similarity.
  • DTW Matching and Lattice Rescoring: Dynamic Time Warping (DTW) is used with the LLR distance to match test speech segments against template representatives. The system then uses this matching score to rescore phone lattices generated by a baseline HMM system, improving the overall recognition accuracy.

Datasets

  • TIMIT: The experiments were conducted on the TIMIT phone recognition task.
  • Training Set: 4620 sentences.
  • Core Test Set: 192 sentences from 24 speakers.

Key Insights and Ideas

  • GMM Indices as Compact Representations: Using GMM indices to represent speech frames is a highly effective way to compress template data without sacrificing too much information, making template-based methods more scalable.
  • LLR Distance for Robust Matching: The log-likelihood ratio distance measure proves more effective than a simple negative log-likelihood distance. It indirectly compares frames through their GMM fits, offering a more robust way to handle variations.
  • Template Representatives are Key: Clustering templates into representatives drastically reduces computational overhead and storage space, making the approach practical for real-world applications, even with a slight trade-off in accuracy compared to using all templates.
  • Synergy of HMMs and Templates: The paper demonstrates that combining statistical models (HMMs) with template matching is more powerful than either approach alone, inheriting the strengths of both.

Key Results

  • Consistent Accuracy Gains: The proposed method consistently improved phone recognition accuracy over HMM baselines.
  • Using all training templates, absolute accuracy gains ranged from 1.69% to 1.83%.
  • Using template representatives (12,000 of them), absolute gains ranged from 1.29% to 1.37%.
  • Reduced Computational Cost: Using 12,000 template representatives reduced memory storage to about 6.6% of that required for all templates. The computational overhead for lattice rescoring was reduced from 135% (with all templates) to just 8% (with representatives) compared to the baseline.
  • Significance: Statistical significance tests confirmed that the improvements were not due to chance.
  • GMM Indices Benefit: Using the top 5 GMM indices for frame labeling yielded the best results, outperforming using only 1, 3, or 7 indices.

Limitations and Future Work

  • Accuracy vs. Efficiency Trade-off: While template representatives significantly reduce overhead, they result in a slightly smaller accuracy gain compared to using all templates.
  • Future Work: The authors suggest investigating longer templates to capture more speech dynamics and evaluating the technique on different speech recognition tasks. They also aim to optimize their codes for speed.

Summary updated

BibTeX
@inproceedings{sun2010integrate,
  author = {Xie Sun and Yunxin Zhao},
  title = {{Integrate template matching and statistical modeling for speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
}

Loading related content...

Figure