Adaptive training with noisy constrained maximum likelihood linear regression for noise robust speech recognition

D. K. Kim, M. J. F. GalesInterspeech 2009

TL;DR

This paper introduces "Noisy CMLLR" (NCMLLR), a novel adaptive training technique for speech recognition systems that explicitly models the uncertainty introduced by background noise. By extending the popular CMLLR method, NCMLLR significantly improves robustness in low signal-to-noise ratio (SNR) conditions, outperforming traditional approaches.

What Problem Does This Paper Address?

Building robust speech recognition systems is a significant challenge, especially when dealing with data collected under varying and noisy conditions. Traditional adaptive training methods, like CMLLR, aim to create a "clean" acoustic model from noisy data. However, these methods often don't adequately account for the increased uncertainty in the acoustic observations when the signal-to-noise ratio (SNR) is low. This can lead to suboptimal performance in real-world scenarios where noise is prevalent. The paper addresses this gap by proposing a method that explicitly models this noise-induced uncertainty, leading to more accurate and robust acoustic models.

Methods

The core contribution is the Noisy CMLLR (NCMLLR) technique. It builds upon the existing Constrained Maximum Likelihood Linear Regression (CMLLR) framework.

  • Generative Model Extension: NCMLLR introduces a modified generative model that relates clean speech ($s_t$) to noisy observations ($o_t$). This model explicitly includes a noise term and a bias on the variance, effectively capturing the additional uncertainty from noise. This can be seen as combining CMLLR with a variance bias, drawing inspiration from factor analysis.
  • NCMLLR Transform: The NCMLLR transform is represented by parameters $A$, $b$, and $\Sigma_b$, which are derived from the underlying linear transform ($H$), bias ($g$), and noise covariance ($\Psi$).
  • Adaptive Training: The paper details how to estimate NCMLLR parameters and use them within an adaptive training framework. This involves an Expectation-Maximization (EM) algorithm to iteratively refine both the NCMLLR transform parameters and the canonical acoustic model parameters.
  • Discriminative Training: The authors also explore using NCMLLR with discriminative training criteria, specifically Minimum Phone Error (MPE), to further enhance performance.

Datasets

The proposed NCMLLR method was evaluated on two distinct datasets:

  • Resource Management (RM): A noise-corrupted version of the standard RM dataset was used. This involved artificially adding "Operations Room" noise to create test sets with varying SNRs (20dB and 14dB). A multi-style training model was built from data with noise added at different SNRs.
  • In-car Data (Toshiba): This dataset consists of noisy speech collected in cars under different driving conditions (engine on and highway driving). The evaluation focused on phone number recognition tasks.

Key Insights and Ideas

  • Explicit Noise Uncertainty Modeling is Crucial: The most significant insight is that simply trying to "clean" noisy speech isn't enough. Explicitly modeling the uncertainty introduced by noise, especially at low SNRs, is key to achieving robust speech recognition.
  • NCMLLR Offers a Principled Way to Model Noise Uncertainty: By extending CMLLR with a variance bias, NCMLLR provides a mathematically sound framework to incorporate this noise-induced uncertainty into the adaptive training process.
  • NCMLLR Outperforms CMLLR at Low SNRs: The experiments clearly demonstrate that NCMLLR provides substantial benefits when the signal-to-noise ratio is low, a common scenario in real-world applications.
  • Adaptive Training with NCMLLR is Effective: The paper shows that NCMLLR is not just a feature enhancement but a powerful tool for adaptive training, leading to improved performance over standard CMLLR.
  • Discriminative Training Further Boosts Performance: Combining NCMLLR with discriminative training (MPE) yields even greater performance gains, particularly for adaptive training scenarios.

Key Results

  • RM Task:
    • At a low SNR of 14dB, NCMLLR significantly outperformed CMLLR for both multi-style and adaptive training, with error rates dropping from 13.0% to 11.6% (full transform) and 10.5% to 9.1% (adaptive training).
  • At a higher SNR of 20dB, CMLLR sometimes performed better for multi-style systems, suggesting NCMLLR's primary advantage lies in low-SNR conditions.
  • Toshiba In-car Task:
    • For the highway driving (HWY) condition, NCMLLR consistently outperformed CMLLR across both ML and MPE trained models, with statistically significant differences.
  • For adaptive training, NCMLLR either matched or outperformed CMLLR in all conditions.
  • MPE training provided substantial gains over ML training, with adaptive training showing particularly large improvements (over 30% relative reduction).

Limitations and Future Work

  • Limitations:
    • The paper notes that at high SNRs, CMLLR can sometimes outperform NCMLLR for multi-style systems, suggesting NCMLLR's benefits are most pronounced in low-SNR scenarios.
  • The authors mention that using full linear transforms for NCMLLR requires storing more complex statistics, though this is manageable.
  • A potential issue with the magnitude of transform matrices in low SNR regions is addressed by limiting the variance bias.
  • Future Work:
    • The authors suggest exploring the use of NCMLLR within a joint adaptive training framework, given its formal similarity to Joint Uncertainty Decoding (JUD).
  • Further investigation into the interaction between NCMLLR and different types of discriminative training criteria is also a promising direction.

Summary updated

BibTeX
@inproceedings{kim2009adaptive,
  author = {D. K. Kim and M. J. F. Gales},
  title = {{Adaptive training with noisy constrained maximum likelihood linear regression for noise robust speech recognition}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure