Discriminative feature-space transforms using deep neural networks

George Saon, Brian KingsburyInterspeech 2012 · OpenReview

Presented at INTERSPEECH 2012 by researchers from IBM, this paper marks a pivotal moment in the transition from traditional Gaussian Mixture Models (GMMs) to Deep Learning in speech recognition. It explores how to use Deep Neural Networks (DNNs) not as standalone acoustic models, but as sophisticated non-linear filters to improve the features fed into existing GMM-based systems.

Problem Statement

For decades, the Hidden Markov Model - Gaussian Mixture Model (HMM-GMM) framework was the industry standard for speech recognition. To enhance accuracy, researchers developed feature-space transforms like Feature-space Maximum Mutual Information (FMMI), which apply offsets to raw audio features to make different sounds easier for the model to distinguish.

However, these traditional transforms were primarily linear. While DNNs were showing massive potential, they were difficult to integrate into existing infrastructures that relied on GMM-specific techniques like speaker adaptation. The authors sought to achieve the "best of both worlds": the non-linear modeling power of DNNs and the established reliability of GMM-HMM systems.

Methodology: Neural Network FMMI (NN-FMMI)

The researchers developed Neural Network FMMI (NN-FMMI), replacing simple linear projections with a multi-layer feed-forward network to calculate feature offsets. A key technical challenge was optimization; training a deep network directly on a complex speech recognition objective like Boosted MMI (BMMI) often leads to poor local optima.

To solve this, they employed a three-step training strategy:

  1. MSE Pretraining: The DNN was first trained to mimic the outputs of a standard linear FMMI transform using Mean Squared Error (MSE) loss. This "teacher-student" phase provided a sensible starting point.
  2. Layer Stacking: The network was grown incrementally, initializing each new layer with weights from the previous version.
  3. Discriminative Fine-tuning: Finally, the entire network was updated using the BMMI objective function to maximize the probability of correct word sequences.
The impact of MSE pretraining on the BMMI objective function compared to random initialization.

Experimental Setup

The experiments were conducted on a 50-hour English broadcast news transcription task.

  • Training: 50 hours of transcribed audio from various sources.
  • Evaluation: The DEV04f test set, containing roughly 2 hours of audio and 22.6K words.
  • Features: 13-dimensional PLP (Perceptual Linear Prediction) features, expanded with temporal context and reduced via LDA/STC to 40 dimensions.

Key Results and Insights

The NN-FMMI approach consistently outperformed linear baselines, demonstrating that DNNs can act as a non-linear "front-end" that simplifies the task for traditional GMMs. A major advantage of this approach is that it preserves the rest of the ASR pipeline, such as speaker adaptation (VTLN, MLLR), making it easier to deploy in real-world systems.

Performance comparisons showed significant gains:

  • Baseline (ML): 23.6% Word Error Rate (WER).
  • Baseline (Linear FMMI + BMMI): 18.7% WER.
  • NN-FMMI (6-layer) + Model-space training: 17.9% WER.

This represents a 4% relative improvement over a highly optimized state-of-the-art system. The results also indicated that increasing the number of layers generally improved the discriminative objective.

BMMI objective function values across training epochs for networks with 3, 4, and 5 layers.

Conclusion and Future Work

The primary limitation of this approach is its dependency on a linear transform to provide targets for the initial MSE pretraining phase. Future directions include finding ways to train the network directly from scratch and exploring advanced optimization techniques, such as Hessian-free (second-order) optimization, to better navigate the complex objective function space.

Summary updated

BibTeX
@inproceedings{saon2012discriminative,
  author = {George Saon and Brian Kingsbury},
  title = {{Discriminative feature-space transforms using deep neural networks}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=B0g6P-AR8eq},
}
Figure