Jointly optimized discriminative features for speech recognition

Tim Ng, Bing Zhang, Long NguyenInterspeech 2010

TL;DR

This paper introduces a novel method called Region Dependent Transform (RDT) to jointly optimize acoustic features extracted from both traditional Perceptual Linear Predictive (PLP) and modern Multi-Layer Perceptron (MLP) models. By applying RDT under a discriminative training criterion, they achieve a significant reduction in character error rate for Mandarin speech recognition, outperforming conventional feature augmentation techniques.

What Problem Does This Paper Address?

Extracting effective acoustic features is crucial for accurate speech recognition. While Multi-Layer Perceptrons (MLPs) have shown promise in capturing long-term acoustic information, they often don't outperform traditional features like Perceptual Linear Predictive (PLP) on their own. Instead, MLP features are typically used as complementary information, augmenting existing feature sets. A challenge arises when trying to reduce the dimensionality of these combined features for efficiency. Standard linear dimension reduction methods like Linear Discriminative Analysis (LDA) are not ideal because they don't directly optimize for speech recognition errors. Furthermore, applying these methods to combined PLP and MLP features can be problematic, as they might discard valuable information or fail to leverage the complementary nature of these different feature types. This paper tackles the problem of effectively combining and optimizing these diverse acoustic features for improved speech recognition performance.

Methods

The core innovation in this paper is the application of Region Dependent Transform (RDT) for jointly optimizing feature extraction. RDT is a discriminatively trained piecewise linear transformation. Unlike traditional methods that might process features independently, RDT takes a combined feature vector (in this case, from PLP and MLP) and transforms it based on "regions" defined by the MLP's phone posterior probabilities. This allows the transform to adapt differently to various acoustic contexts, effectively learning to leverage the strengths of both PLP and MLP features while discarding redundant information.

The authors also employ a discriminative training criterion, specifically the Minimum Phone Frame Error (MPFE) criterion, to estimate the RDT transforms. This is a key departure from standard Maximum Likelihood (ML) training, as it directly optimizes for reducing recognition errors rather than just modeling the data distribution.

For initialization, the RDT transforms are set up such that they initially represent the standard Speaker Adaptive Training (SAT) features without MLP augmentation. This allows for initial HMM training using ML before the discriminative MPFE training refines the features.

Datasets

  • Training Data: 1900 hours of audio data released by LDC for the GALE Phase 4 evaluation.
  • Language Models: Trained on 4.6 billion characters for ML experiments and 6.8 billion characters for MPFE experiments, with a vocabulary size of 64K.
  • Evaluation Data: Four test sets were used: dev07 (2.6 hours), dev08 (1.0 hour), eval08 (1.5 hours), and dev09sub (3.0 hours).

Key Insights and Ideas

  • Joint Optimization is Key: Simply augmenting features with MLP outputs isn't enough. Jointly optimizing the transformation of combined PLP and MLP features under a discriminative criterion yields significant gains.
  • Discriminative Training Matters: Using MPFE for training RDT transforms directly targets error reduction, leading to better performance than ML-based approaches, especially after speaker adaptation.
  • MLP as Context for RDT: Using MLP phone posterior probabilities to define the "regions" for RDT is an effective way to partition the acoustic space and guide the transform, leveraging the MLP's contextual understanding.
  • Robustness to Adaptation: While conventional MLP augmentation benefits diminish after speaker adaptation, RDT-based features maintain their advantage, suggesting better robustness.

Key Results

  • Mandarin Speech Recognition: The proposed RDT method achieved a 3% to 6% relative reduction in Character Error Rate (CER) for Mandarin speech recognition after speaker adaptation, compared to conventional feature augmentation.
  • Unadapted Systems: Conventional MLP augmentation showed a 6% relative CER reduction on dev07 (from 12.8% to 12.0%) and smaller improvements on other sets.
  • Speaker Adapted Systems (ML): The benefits of conventional MLP augmentation largely vanished after Speaker Adaptive Training (SAT), with only minor improvements or even degradations.
  • Speaker Adapted Systems (MPFE): With MPFE training, the RDT approach consistently outperformed the baseline and conventional MLP augmentation, showing 3% to 6% relative CER reduction. For example, on dev07, the RDT system achieved a CER of 9.0% compared to 9.3% for +MLP and 9.4% for the baseline.

Limitations and Future Work

  • Language Specificity: The current evaluation is focused on a Mandarin speech recognition system. The authors plan to apply the method to other languages like Arabic and English.
  • Fair Comparison: The authors acknowledge that comparing a discriminatively trained system (RDT) with a purely ML-trained baseline might not be entirely fair, as the RDT system is optimized for error reduction.
  • Initialization: While the initialization strategy allows for initial ML training, further exploration into optimal initialization for RDT could be beneficial.

Summary updated

BibTeX
@inproceedings{ng2010jointly,
  author = {Tim Ng and Bing Zhang and Long Nguyen},
  title = {{Jointly optimized discriminative features for speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure