A corpus-based approach to speech enhancement from nonstationary noise

Ji Ming, Ramji Srinivasan, Danny CrookesInterspeech 2010

TL;DR

This paper introduces a novel corpus-based approach to speech enhancement that tackles the challenge of nonstationary noise by treating speech segments as whole units. By leveraging a large speech corpus to model temporal dynamics and speaker characteristics, the method achieves improved noise immunity without requiring prior knowledge of the noise statistics.

What Problem Does This Paper Address?

Speech enhancement aims to clean up noisy audio signals, a crucial task for applications like voice assistants and telecommunications. Traditional methods often struggle with nonstationary noise (like music, song, or crosstalk) because they rely on estimating noise statistics, which is difficult when the noise changes rapidly. These methods also typically process audio frame-by-frame, ignoring the natural temporal flow and speaker-specific characteristics of speech. This paper addresses the limitations of existing techniques by proposing a method that can effectively enhance speech even in the presence of highly variable, unpredictable noise, by focusing on the inherent structure of speech itself.

Methods

The core of the proposed approach is to recognize and utilize longer segments of speech as complete units, rather than processing individual frames. This is achieved by:

  • Corpus-Based Modeling: A large speech corpus (like TIMIT) is used to build models that capture the temporal dynamics and speaker characteristics of clean speech.
  • Example-Based Matching: The system learns to identify segments of noisy speech by comparing them to these corpus-based models. The idea is that longer speech segments are more distinctive and thus easier to distinguish from noise.
  • Gaussian Mixture Models (GMMs): A GMM is trained on the entire corpus to represent the general characteristics of speech.
  • Sentence-Specific Models: For each sentence in the corpus, a model is built that represents its complete temporal dynamics, essentially creating a template.
  • Noise Compensation: To improve robustness against noise, the system simulates noisy versions of the corpus sentences at various noise levels and SNRs. It then employs a "missing-feature" theory, where it selectively uses the most reliable parts of the noisy signal for matching, effectively ignoring corrupted local features.
  • Reconstruction: Once matching segments are identified, the clean speech is reconstructed by averaging the corresponding corpus segments, weighted by their posterior probability (confidence score).

Datasets

  • TIMIT Database: Used for both training and evaluation.
  • Training: 3696 sentences from 462 speakers (326 male, 136 female) were used to build the GMMs and corpus models. Male and female data were modeled separately.
  • Testing: 192 sentences from 24 speakers (16 male, 8 female) were used for evaluation.

Key Insights and Ideas

  • Speech as Whole Units: The fundamental insight is that treating speech as longer, coherent segments (rather than isolated frames) significantly improves its distinguishability from noise, especially when noise statistics are unknown.
  • Leveraging Corpus for Structure: A large speech corpus acts as a rich source of "clean" examples, providing templates for temporal dynamics and speaker characteristics that can be matched against noisy input.
  • Robustness through Noise Simulation: Simulating various noise conditions during training, combined with a missing-feature approach, makes the system resilient to noise types and levels not explicitly seen during training.
  • No Prior Noise Knowledge Needed: The method's strength lies in its ability to perform well without needing to estimate noise statistics, a major bottleneck for traditional methods dealing with nonstationary noise.

Key Results

  • Improved Objective Measures: The new approach consistently outperformed conventional algorithms (KLT, Log MMSE, MBand, Wiener filtering) in terms of segmental SNR and log-spectral distance, particularly for faster-varying noises like music and pop songs.
  • Subjective Preference: Subjective listening tests showed that listeners preferred the enhanced speech from the new algorithm, citing reduced noise intrusion and improved overall quality.
  • Phone Identification: The algorithm also demonstrated improved phone recognition accuracy in the presence of crosstalk speech noise.
  • Competitive Performance: Even for slowly varying noise (babble), the new algorithm was competitive with established methods.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed, corpus-based matching and GMM training can be computationally intensive.
  • Model Complexity: The performance is tied to the quality and size of the training corpus.
  • Future Work: The paper suggests that the approach can be extended to more complex scenarios, such as separating multiple speakers or dealing with even more challenging acoustic environments. The authors also mention the use of state-space models for more realistic temporal dynamics, hinting at potential future improvements.

Summary updated

BibTeX
@inproceedings{ming2010corpus,
  author = {Ji Ming and Ramji Srinivasan and Danny Crookes},
  title = {{A corpus-based approach to speech enhancement from nonstationary noise}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure