Sparse representation features for speech recognition

Tara N. Sainath, Bhuvana Ramabhadran, David Nahamoo, Dimitri Kanevsky, Abhinav SethyInterspeech 2010

TL;DR

This paper introduces a novel feature extraction method for speech recognition using sparse representations (SRs). By mapping test speech features into the linear span of training examples, they achieve significant reductions in phonetic and word error rates on both small and large vocabulary tasks, outperforming traditional Gaussian Mixture Models (GMMs).

What Problem Does This Paper Address?

Traditional speech recognition systems often rely on Gaussian Mixture Models (GMMs) to represent speech features. While GMMs are computationally efficient for training and scoring, they can lose valuable information by pooling training samples. Exemplar-based methods, like sparse representations (SRs), aim to preserve this information by representing test data as a linear combination of actual training examples. However, applying these methods directly to speech recognition has been challenging due to computational costs and difficulties in integrating them with standard Hidden Markov Model (HMM) frameworks, which require frame-level probabilities. This paper addresses this gap by proposing a way to leverage SRs to create new, more discriminative features that can be effectively used within an HMM-based recognition system.

Methods

The core of this paper is the use of Sparse Representation (SR) features. Given a test feature vector y and a dictionary H composed of training examples from various classes, the goal is to find a sparse coefficient vector β such that y ≈ Hβ. This essentially projects the test feature y into the "linear span" of the training data in H. The sparsity constraint on β is crucial, as it encourages the representation to use only a few training examples, ideally those belonging to the correct class.

The authors use the Approximate Bayesian Compressive Sensing (ABCS) method to solve for β, which incorporates both L1 and L2 regularization to promote sparsity and prevent overfitting. The resulting SR feature for a given frame is the product Hβ.

A key challenge addressed is how to construct the dictionary H. The paper explores several strategies for "seeding" H with relevant training data:

  • Nearest Neighbors (kNN): Selecting training examples closest to the test sample.
  • Language Model (LM) based seeding: Using trigram or unigram LMs to identify the most likely acoustic models (Gaussians) for a given frame and then selecting training data associated with those Gaussians and their neighbors.
  • Gaussian Means: Using the means of selected Gaussians instead of actual training examples to reduce computational cost.

Once the SR features (Hβ) are computed for all frames, a new HMM is trained on these enhanced features for both training and testing.

Datasets

  • TIMIT phonetic corpus: Used for small vocabulary recognition experiments.
  • English broadcast news transcription task (50 hours): Used for large vocabulary recognition experiments.

Key Insights and Ideas

  • SRs move features closer to training and the correct class: The paper demonstrates that SR features not only map test data into the space of training data but also push it towards the correct class, leading to improved classification accuracy.
  • SR features enhance existing discriminative models: Applying SR features on top of already discriminatively trained features (fBMMI) yields further improvements, suggesting SRs capture complementary information.
  • Language Model seeding is crucial for large vocabulary tasks: Seeding the SR dictionary H using language model information (especially trigram LM) significantly outperforms using only nearest neighbors, which is attributed to the computational infeasibility and potentially lower frame accuracy of kNN for large datasets.
  • Gaussian means offer a computationally efficient alternative: Using Gaussian means to seed H provides a good trade-off between computational cost and recognition performance, making SRs more practical for large-scale applications.
  • Sparsity in β coefficients is a form of discrimination: The sparse nature of the β vector visually confirms that only a few training exemplars are needed to represent a test sample, acting as an implicit selection of "good" examples.

Key Results

  • TIMIT (Small Vocabulary):
    • SR features achieved a 0.7% absolute reduction in Phonetic Error Rate (PER) over the best discriminatively trained system (from 19.9% to 19.2%).
  • After MLLR adaptation, the PER was reduced to 19.0%, reported as the best results on TIMIT to date.
  • Frame classification accuracy for SRs was 71.7%, compared to 70.4% for GMMs.
  • Broadcast News (Large Vocabulary):
    • SR features achieved a 0.3% absolute reduction in Word Error Rate (WER) over the baseline system (from 19.0% to 18.7%).
  • The best performing SR configuration for broadcast news involved seeding H using a trigram LM and random sampling of top 5 Gaussians, or using Gaussian Means.

Limitations and Future Work

  • Computational Cost: While the paper explores methods to mitigate computational cost (e.g., Gaussian means, LM-based seeding), SR feature extraction can still be more computationally intensive than GMMs, especially for very large training sets.
  • Choice of Dictionary H: The performance is highly dependent on the strategy used to construct the dictionary H. The paper explores various seeding methods but acknowledges that further optimization might be possible.
  • Future Work: The authors suggest exploring mapping SR features from the fBMMI space before applying model-space discriminative training and MLLR to potentially further improve performance. They also implicitly suggest continued exploration of dictionary construction and sampling strategies.

Summary updated

BibTeX
@inproceedings{sainath2010sparse,
  author = {Tara N. Sainath and Bhuvana Ramabhadran and David Nahamoo and Dimitri Kanevsky and Abhinav Sethy},
  title = {{Sparse representation features for speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure