A framework for estimation of clean speech by fusion of outputs from multiple speech enhancement systems

Venkatesh Krishnan, Phil S. Whitehead, David V. Anderson, Mark A. ClementsInterspeech 2005

TL;DR

This paper introduces a novel framework that combines the outputs of multiple speech enhancement systems to produce a cleaner speech signal than any individual system can achieve. By using a multiple-input Kalman filter (MIKF) and modeling speech and noise with autoregressive processes, the system effectively "fuses" information, leading to significant improvements in both objective and subjective speech quality.

What Problem Does This Paper Address?

Speech enhancement aims to remove noise from audio signals, improving intelligibility and the performance of downstream applications like speech recognition. However, existing speech enhancement systems often leave behind residual noise and artifacts, and different systems excel in different noise conditions or with different types of speech. The challenge is to leverage the strengths of multiple enhancement systems to create a more robust and effective overall solution. This paper addresses the gap of how to optimally combine the diverse outputs of these individual systems into a single, superior clean speech estimate.

Methods

The core of the proposed solution is a Multiple-Input Kalman Filter (MIKF). This framework treats the outputs of individual speech enhancement systems as noisy observations of the underlying clean speech signal.

Key methodological components include:

  • Autoregressive (AR) Modeling: Both the clean speech signal and the residual noise in each enhancement system's output are modeled as Gaussian AR processes. This allows for the representation of temporal dependencies in the signals.
  • Dynamic Noise Modeling: The AR parameters for the noise are dynamically updated, enabling the system to handle non-stationary noise conditions.
  • Codebook-Constrained AR Parameters for Speech: The AR parameters for the speech signal are constrained to a pre-designed codebook of prototype AR parameters. This improves the quality of the speech estimate and facilitates integration with speech coders.
  • Expectation-Maximization (EM) Algorithm: An iterative EM algorithm is used to estimate the AR model parameters for both speech and noise from the available input data.
  • Heuristic Weighting: The framework allows for user-defined, heuristic weights to be applied to the inputs. This enables external control over the influence of each enhancement system's output, potentially based on perceptual quality or intelligibility measures.

Datasets

  • TIMIT Database: Used for training the AR model codebook for clean speech. Approximately 100,000 AR parameter training vectors were obtained from both male and female utterances across eight North American English dialects.
  • NOISEX-92 Database: Used to generate noisy speech samples by adding five different noise environments to clean utterances.

Key Insights and Ideas

  • Fusion is More Than the Sum of its Parts: The fundamental idea is that combining multiple imperfect estimates can yield a better result than any single estimate, a concept akin to "wisdom of the crowd" applied to signal processing.
  • Kalman Filter as an Optimal Combiner: The Kalman filter's ability to provide a Minimum Mean-Square Error (MMSE) estimate makes it a natural choice for fusing these signals.
  • Flexibility in Input Handling: The MIKF can handle any number of inputs and allows for flexible weighting, making it adaptable to various combinations of enhancement systems.
  • Codebook Constraint Improves Speech Quality: Constraining speech AR parameters to a codebook, rather than estimating them freely, significantly improves the perceptual quality of the enhanced speech.

Key Results

  • Objective Improvement (SSNR): The MIKF system demonstrated significant improvements in Segmental Signal-to-Noise Ratio (SSNR) over individual enhancement systems and the original noisy speech. Gains of over 15 dB were observed in adverse conditions (e.g., -5 dB SNR with M109 tank noise).
  • Subjective Improvement (Perceptual Quality): Category Comparison Rating (CCR) listening tests showed significant perceptual quality improvements. The MIKF output was rated higher than the outputs of the individual enhancement systems (AWF and NPP) in both tested noise conditions (M109 and Buccaneer1 at 0 dB SNR). The improvement was more pronounced compared to the AWF system.
  • Performance with Non-Stationary Noise: The dynamic noise modeling and AR parameter updates allow the system to perform well even with non-stationary noise.

Limitations and Future Work

  • Heuristic Weighting Optimization: While heuristic weighting is supported, the paper suggests further investigation into optimal methods for choosing these weights, potentially based on segmentation or other intelligibility/quality metrics.
  • Codebook Design: The paper notes that codebook size is critical for spectral resolution and that a codebook of 214 sets was found adequate. Further research could explore more sophisticated codebook design strategies.
  • Integration with Speech Coders: The authors mention that the framework is well-suited for integration with model-based speech coders and that work is in progress to evaluate the subjective quality and intelligibility of decoded speech from the MIKF output.
  • Class-Specific Codebooks: Designing codebooks tailored to different phonemes or speech classes is suggested as a future direction for further improvement.

Summary updated

BibTeX
@inproceedings{krishnan2005framework,
  author = {Venkatesh Krishnan and Phil S. Whitehead and David V. Anderson and Mark A. Clements},
  title = {{A framework for estimation of clean speech by fusion of outputs from multiple speech enhancement systems}},
  booktitle = {Interspeech},
  year = {2005},
}
Figure