Advances in noise robust digit recognition using hybrid exemplar-based techniques

Jort F. Gemmeke, Hugo Van hammeInterspeech 2012

TL;DR

This paper presents a hybrid system that significantly improves noise-robust digit recognition by combining exemplar-based sparse classification (SC) and feature enhancement (FE) techniques. By integrating several recent advances in SC and employing multi-condition training for acoustic models, the proposed system achieves state-of-the-art results on the challenging AURORA-2 benchmark, demonstrating remarkable robustness against both matched and mismatched noise.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems struggle significantly in noisy environments, often performing worse than humans even on simple tasks like digit recognition. While various noise robustness techniques exist, such as spectral subtraction and missing data approaches, a promising direction involves representing noisy speech as a linear combination of clean speech and noise "exemplars" (segments of audio). This paper tackles the challenge of effectively integrating and advancing these exemplar-based methods, specifically focusing on improving Sparse Classification (SC) and Feature Enhancement (FE) techniques, and combining them into a unified, high-performing system. The goal is to push the boundaries of noise robustness, especially in scenarios with diverse and unseen noise types.

Methods

The core of the proposed system relies on a compositional model for noise-robust ASR. This model represents noisy speech spectrograms as a sparse linear combination of basis atoms (exemplars) from separate speech and noise dictionaries.

  • Sparse Classification (SC): In this approach, the weights of the selected speech exemplars are directly used to estimate speech state probabilities. The paper incorporates several recent advances to improve SC:
  • Tuned Sparsity: Jointly optimizing sparsity for both speech and noise exemplars.
  • Increased Iterations: Running the sparse representation algorithm for more iterations (from 200 to 600).
  • Larger Speech Dictionary: Increasing the number of speech exemplars from 4000 to 10000.
  • Artificial Noise Dictionary: Using an artificial dictionary to better match unseen noise types.
  • Noise Sniffing: Extracting noise exemplars from the initial frames of noisy speech and introducing shifted copies for invariance.
  • Feature Enhancement (FE): Alternatively, the activation matrix from the sparse representation can be used to reconstruct a cleaner speech estimate. This enhanced feature set is then fed into a conventional Gaussian Mixture Model (GMM)-based recognizer. The paper investigates retraining GMM models with FE-processed features and using multi-condition training data.
  • Hybrid Multi-Stream Decoding: The paper proposes a novel hybrid system that combines the SC and FE approaches at the state probability level. The state probabilities from the SC stream and the GMM-based FE stream are merged, treating them as independent streams. This allows the system to leverage the strengths of both approaches.

Datasets

  • AURORA-2 Corpus: This is the primary dataset used for both training and evaluation. It's a standard benchmark for noise-robust ASR, featuring digit recognition tasks corrupted by various noise types at different Signal-to-Noise Ratios (SNRs).
  • Test Sets: The experiments utilize Test Set A (matched noise types) and Test Set B (mismatched noise types).
  • Training Data: Both clean and multi-condition training sets are used.

Key Insights and Ideas

  • Synergy of SC Advances: Individual advances in SC (like increased iterations, larger dictionaries, and artificial noise dictionaries) cumulatively lead to significant improvements in noise robustness, especially for mismatched noise.
  • SC vs. FE Trade-offs: SC tends to perform better at low SNRs and on mismatched noise, while FE excels at high SNRs and matched noise. This highlights the complementary nature of the two techniques.
  • FE for Acoustic Model Training: Applying FE to features during acoustic model training (not just testing) can surprisingly improve performance even on clean speech, suggesting it helps the model learn a more robust representation of the speech manifold.
  • Multi-Condition Training is Key: Using multi-condition training data for the GMM-based FE recognizer significantly boosts noise robustness, even at the cost of a slight degradation on clean speech.
  • Hybrid System Dominance: Combining SC and FE in a multi-stream architecture at the probability level yields the best overall performance, outperforming either individual system and achieving state-of-the-art results.

Key Results

The paper reports impressive Word Error Rate (WER) reductions on the AURORA-2 dataset:

  • Best Hybrid System: The proposed multi-condition trained hybrid SC-FE system achieves an average WER of 3.1% on matched noise types (Test Set A) and 5.0% on mismatched noise types (Test Set B).
  • These results are stated to be among the best ever published on the AURORA-2 dataset, significantly outperforming previous baseline systems and even competitive with other state-of-the-art methods like ESSEM and CMLLR.
  • Table 1 and Table 2 provide detailed WER breakdowns across different SNRs and noise types, illustrating the impact of each individual SC advance and the FE training strategies.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the hybrid system is highly effective, there's still room for improvement, particularly in further enhancing robustness on mismatched noise types. The current work focuses on digit recognition, which is a relatively simple task.
  • Future Work: The authors suggest several promising directions:
  • Further improving noise robustness on mismatched noise types.
  • Evaluating the proposed techniques on larger vocabulary speech databases.
  • Addressing non-additive signal corruptions such as channel effects and reverberation.

Summary updated

BibTeX
@inproceedings{gemmeke2012advances,
  author = {Jort F. Gemmeke and Hugo Van hamme},
  title = {{Advances in noise robust digit recognition using hybrid exemplar-based techniques}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure