Incorporating sparse representation phone identification features in automatic speech recognition using exponential families

Vaibhava Goel, Tara N. Sainath, Bhuvana Ramabhadran, Peder Olsen, David Nahamoo, Dimitri KanevskyInterspeech 2010

TL;DR

This paper introduces a novel way to improve automatic speech recognition (ASR) by combining existing acoustic features with new "sparse representation phone identification features" (SPIF). By using exponential family models, they achieve performance comparable to complex system combination methods but with a simpler, more unified approach.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems rely on accurately modeling the acoustic signals to identify spoken words. A common approach is to use Hidden Markov Models (HMMs) with Gaussian Mixture Models (GMMs) to represent the likelihood of observing certain acoustic features given a particular phonetic state. While these models are effective, there's a continuous drive to improve their accuracy, especially for large vocabulary continuous speech recognition (LVCSR) tasks.

One way to boost performance is through system combination, where outputs from multiple ASR systems or feature streams are merged. However, these methods can be complex, requiring the training of multiple systems or careful tuning of combination weights. This paper aims to find a more integrated and efficient way to leverage complementary information from different feature sources, specifically combining standard acoustic features with a new type of phone-level probability estimate.

Methods

The core of this paper lies in how they combine two types of features:

  • fMMI features: These are standard acoustic features derived from a boosted Maximum Mutual Information (MMI) training process, which are used to train GMMs for HMM states.
  • SPIF features: These are novel features based on "sparse representation." The idea is to represent a given acoustic frame as a sparse linear combination of training acoustic vectors. The coefficients of this sparse representation are then used to estimate phone posterior probabilities. This is a new approach for generating phone-level information directly from acoustic frames.

The authors propose using exponential family models to combine these two feature streams. Instead of treating them as separate systems to be combined later, they are integrated as components of a single, more powerful acoustic model. They explore two specific exponential families, "modell" and "model2," which differ in how they handle the phone posterior probabilities. The key idea is that these exponential models can naturally learn the relationship between the fMMI features and the SPIF phone posteriors, allowing for a more unified acoustic representation.

They also discuss parameter estimation for these models using an Expectation-Maximization (EM) procedure and a modified Maximum Likelihood Linear Regression (MLLR) for adaptation.

Datasets

  • Training Data: 50 hours of English Broadcast News speech corpora (from 1996 and 1997).
  • Testing Data: EARS Dev-04f set (3 hours of audio from 6 shows collected in November 2003).
  • Language Model Data: A 54M 4-gram model trained on 335M words.

No new benchmarks were introduced; the focus was on evaluating their method on existing datasets.

Key Insights and Ideas

  • SPIF as a Complementary Feature: The paper demonstrates that SPIF features, derived from sparse representations, provide valuable phone-level information that complements traditional acoustic features.
  • Exponential Families for Unified Modeling: Using exponential families offers a principled and elegant way to integrate diverse feature streams (like fMMI and SPIF) into a single acoustic model, avoiding the complexities of late-stage system combination.
  • Simplicity and Efficiency: The proposed method achieves performance comparable to complex system combination techniques (like ROVER or model combination) but without the overhead of training multiple systems or tuning combination weights.
  • The Power of Sparse Representation: The success of SPIF highlights the potential of sparse representation techniques for extracting meaningful phonetic information from raw acoustic data.

Key Results

  • The baseline fMMI+bMMI acoustic model achieved a Word Error Rate (WER) of 19.4%.
  • The baseline SPIF model achieved a WER of 19.3%.
  • When combining fMMI and SPIF features using their proposed exponential models (modell and model2) and applying speaker adaptation (eMLLR), they achieved a word error rate of 18.2%.
  • This 18.2% WER is comparable to the best results obtained from traditional system combination methods like model combination and N-best ROVER, which achieved 18.1% and 18.3% respectively.

Limitations and Future Work

  • Overtraining: The authors note that the exponential models can be susceptible to overtraining, especially when the dimension of the phone posterior vector is large. They address this by constraining the model parameters.
  • Parameter Tuning: While simpler than system combination, some parameter tuning (like thresholding for the exponential model parameters) is still required.
  • Future Work: The paper suggests that SPIF can be extended to other classes like sub-phones or context-dependent sub-phones, implying further avenues for feature development. The success of exponential families also opens doors for exploring other feature combinations and model architectures.

Summary updated

BibTeX
@inproceedings{goel2010incorporating,
  author = {Vaibhava Goel and Tara N. Sainath and Bhuvana Ramabhadran and Peder Olsen and David Nahamoo and Dimitri Kanevsky},
  title = {{Incorporating sparse representation phone identification features in automatic speech recognition using exponential families}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure