Reconstructing clean speech from noisy MFCC vectors

Ben Milner, Jonathan Darch, Ibrahim AlmajaiInterspeech 2009

TL;DR

This paper tackles the challenge of reconstructing clean speech from noisy Mel-Frequency Cepstral Coefficients (MFCCs) alone, a crucial step for distributed speech recognition systems. They introduce noise compensation techniques for predicting fundamental frequency and voicing, significantly improving reconstruction quality in noisy environments.

What Problem Does This Paper Address?

In distributed speech recognition (DSR), speech is often processed into MFCCs at a terminal device and sent to a remote server for decoding. This architecture typically doesn't allow for reconstructing the original speech waveform at the server. This capability is valuable for tasks like error checking or avoiding complex audio codecs on the terminal. While a standard exists (ETSI Aurora) that allows for speech reconstruction by transmitting additional information like fundamental frequency (F0) and voicing, this paper aims to achieve reconstruction solely from MFCCs, even when they are corrupted by noise. The core problem is that noise distorts MFCCs, making it difficult to accurately predict the necessary F0 and voicing information for reconstruction.

Methods

The authors build upon existing work that predicts F0 and voicing from MFCCs using a Maximum A Posteriori (MAP) approach based on a joint density model of MFCCs and F0. They extend this by:

  • Phoneme-Specific Modeling: Instead of a single global model, they train separate models for each phoneme, leading to more accurate F0 and voicing predictions.
  • Noise Compensation: Two primary methods are explored to mitigate the impact of noise on MFCCs and improve prediction accuracy:
  • Spectral Subtraction: This feature-domain method involves transforming MFCCs to the filterbank domain, estimating and subtracting noise, and then transforming back.
  • Model Adaptation: This model-domain approach adapts the statistics (mean and covariance) of the clean-trained Gaussian Mixture Models (GMMs) to better match the noisy MFCC vectors.

Datasets

  • Training and Testing: A speaker-dependent database from a female US English speaker was used, comprising 579 sentences for training and 246 for testing.
  • Reference Data: Fundamental frequency and voicing information were obtained from laryngograph recordings.
  • Noise: Car noise was used to contaminate the clean speech at various Signal-to-Noise Ratios (SNRs).

Key Insights and Ideas

  • Noise is a major hurdle for F0/voicing prediction: Just like in speech recognition, noise corrupts MFCCs, shifting their statistics away from clean models and degrading prediction accuracy.
  • Phoneme-specific models are better: Localizing prediction to individual phonemes significantly improves accuracy compared to a global model.
  • Model adaptation shines in noise: The model adaptation technique proved to be the most effective noise compensation method, achieving performance close to matched training/testing conditions, even at low SNRs.
  • Reconstruction quality is surprisingly good: Even with predicted F0/voicing from noisy MFCCs, the reconstructed speech quality is decent and significantly improves with noise compensation.
  • ETSI standard is robust but requires more data: The ETSI Aurora standard, which uses time-domain waveform information, provides good reconstruction but requires transmitting extra data, whereas this method aims for MFCC-only reconstruction.

Key Results

  • Voicing Prediction: Model adaptation significantly reduced voicing classification errors, especially at lower SNRs (e.g., 10dB and 0dB), outperforming spectral subtraction and no compensation.
  • Fundamental Frequency Prediction: Model adaptation also improved fundamental frequency prediction accuracy, particularly at 20dB and 10dB SNRs.
  • Speech Reconstruction Quality (MOS):
    • In clean speech, reconstructed speech using predicted F0/voicing (PRED) had a Mean Opinion Score (MOS) of 3.6, while the ETSI standard (ETSI) achieved 4.1, and original speech (ORIG) was the benchmark.
  • With noise compensation (PREDA), the MOS improved significantly compared to PRED, approaching the quality of the ETSI method, especially at lower SNRs. For instance, at 20dB, PREDA's MOS was only about 0.5 points lower than ETSI.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that even with model adaptation, performance at very low SNRs (0dB) still lags behind matched training/testing conditions.
  • The ETSI method, while requiring more data transmission, still achieves slightly better reconstruction quality in clean conditions.
  • Future Work: The paper suggests that the noise compensation methods developed for F0 and voicing prediction could be further explored and integrated into more advanced speech recognition systems. The success of model adaptation points towards its potential for robust DSR.

Summary updated

BibTeX
@inproceedings{milner2009reconstructing,
  author = {Ben Milner and Jonathan Darch and Ibrahim Almajai},
  title = {{Reconstructing clean speech from noisy MFCC vectors}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure