Context-dependent MLPs for LVCSR: TANDEM, hybrid or both?

Zoltán Tüske, Martin Sundermeyer, Ralf Schlüter, Hermann NeyInterspeech 2012

TL;DR

This paper investigates the effectiveness of Multi-Layer Perceptron (MLP) based acoustic models for Large Vocabulary Continuous Speech Recognition (LVCSR) in French, comparing them against traditional Gaussian Mixture Model (GMM) approaches. The authors find that MLP-based models, especially when combined with context-dependent "bottle-neck" features and deep network architectures, significantly outperform GMMs, particularly when using more complex, transformed features.

What Problem Does This Paper Address?

The field of Automatic Speech Recognition (ASR) has seen a significant shift towards using neural networks, specifically Multi-Layer Perceptrons (MLPs), to model acoustic information. However, there's an ongoing debate about the best way to integrate MLPs into existing Hidden Markov Model (HMM) frameworks. Two primary approaches exist: TANDEM, where MLP posteriors are used as features for a GMM-based system, and hybrid MLP-HMM, where MLPs directly replace GMMs for estimating emission probabilities. This paper aims to systematically compare these two MLP integration strategies, as well as explore the impact of different feature types (standard MFCCs, speaker-adaptive features, and novel context-dependent "bottle-neck" features) and network architectures (shallow vs. deep MLPs) on recognition performance in a French LVCSR task. The goal is to determine the optimal configuration for achieving state-of-the-art results.

Methods

The researchers compare two main acoustic modeling paradigms:

  • GMM-HMM: This is the traditional baseline, using Gaussian Mixture Models within an HMM framework. They employ standard techniques like speaker adaptive training (SAT) and linear discriminant analysis (LDA) for feature transformation.
  • MLP-HMM (Hybrid): Here, MLPs directly estimate the posterior probabilities of HMM states, replacing the GMM emission probabilities. They experiment with both shallow (3-layer) and deep (multiple hidden layers) MLPs.
  • TANDEM: This approach uses MLP-generated posterior probabilities as input features for a GMM-HMM system. They specifically investigate two types of TANDEM features:
  • Probabilistic TANDEM (pTANDEM): Uses MLP posteriors directly.
  • Bottle-neck TANDEM (BN): Uses the output of a narrow "bottle-neck" layer within a deeper MLP, which is then used as features.

They also explore various feature transformations, including LDA and Principal Component Analysis (PCA), and investigate the effect of concatenating different feature types (e.g., MFCCs with TANDEM features). The impact of the number of input frames (context) fed to the MLP is also a key experimental variable.

Datasets

The experiments are conducted on a French broadcast news and conversations task, using data collected within the QUAERO project.

  • Training Data: Approximately 250 hours of transcribed French speech.
  • Development Data (Dev10): 3.7 hours, used for tuning system parameters.
  • Evaluation Data (Eval10): 2.9 hours, used for initial performance evaluation.
  • Evaluation Data (Eval11): 3.1 hours, used for final conclusions, with a new language model tuned on Dev11.

Key Insights and Ideas

  • MLPs Outperform GMMs: Across various feature types, MLP-based acoustic models consistently achieve better results than GMM-based models, especially when using transformed features.
  • Context-Dependent Bottle-neck Features Shine: The "bottle-neck" TANDEM features, which capture context-dependent information, significantly outperform standard probabilistic TANDEM features.
  • Deep MLPs Matter: Deep MLP architectures (multiple hidden layers) provide a substantial performance boost compared to shallow MLPs, particularly when combined with advanced features.
  • Feature Concatenation is Key: Combining standard cepstral features (like MFCCs) with context-dependent bottle-neck features leads to the best overall performance, suggesting a synergistic effect.
  • Context is Crucial for MLPs: MLPs benefit significantly from a wider temporal context (more input frames) compared to GMMs, where performance can degrade without explicit feature reduction techniques like LDA.

Key Results

  • Hybrid MLP vs. GMM: For standard MFCC features, hybrid MLP models achieve over 1.5% absolute improvement in Word Error Rate (WER) over GMMs. This gap widens to over 10% relative when using linearly transformed features (derivatives, LDA, CMLLR).
  • Bottle-neck TANDEM: Bottle-neck TANDEM features, when concatenated with MFCCs, yield a 3% relative gain over phone-posterior based BN features and a 9% relative improvement over standalone MFCC-SAT features.
  • Best Performance: The highest recognition performance is achieved by a deep MLP acoustic model trained on speaker-normalized, concatenated MFCC and context-dependent bottle-neck features (MFCCLDA+BN)SAT. This configuration resulted in a WER of 18.4% on the Eval11 test set.
  • Impact of Context: For MLPs, using 9 consecutive frames of acoustic features as input leads to better performance than using a single frame, a trend that is less pronounced or even detrimental for GMMs without feature reduction.

Limitations and Future Work

  • Short-term Features: The study was primarily limited to short-term TANDEM features. The authors suggest future work should explore long-term features like MRASTA.
  • Deeper MLPs and Discriminative Features: They also propose investigating even deeper MLP architectures and other discriminative features (e.g., fMMI) as avenues for further improvement.
  • Generalization: While the results are promising, the study focuses on a specific French LVCSR task. Further validation on different languages and tasks would be beneficial.

Summary updated

BibTeX
@inproceedings{tuske2012context,
  author = {Zoltán Tüske and Martin Sundermeyer and Ralf Schlüter and Hermann Ney},
  title = {{Context-dependent MLPs for LVCSR: TANDEM, hybrid or both?}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure