Speech Enhancement In Multiple-Noise Conditions using Deep Neural Networks

Anurag Kumar, Dinei FlorêncioarXiv 2016 · OpenReview

Abstract

In this paper, we consider the problem of speech enhancement in real-world conditions where multiple noises can simultaneously corrupt speech. Most current literature focuses primarily on the presence of a single noise source, which does not accurately reflect real-world environments. Specifically, we address improving speech quality in office environments where multiple stationary and non-stationary noises may be present. We propose several strategies based on Deep Neural Networks (DNN) for speech enhancement in these scenarios and investigate a training strategy based on psychoacoustic models from speech coding.

1. Introduction

Traditional speech enhancement (SE) research often operates under the assumption of a single, isolated noise source. However, in practical settings like an office, speech is typically corrupted by a mixture of stationary noises (e.g., fans, air conditioning) and non-stationary noises (e.g., keyboard clicks, background chatter). Standard algorithms frequently struggle with such overlapping, unpredictable interference. This work moves SE into these "messy" realities, utilizing an environment-specific paradigm to develop more robust and practical models.

2. Methodology

The proposed approach utilizes a multilayer feed-forward DNN to map noisy log-power spectra to clean log-power spectra. Two primary innovations are introduced to handle complex noise environments:

2.1 Dynamic Feature Expansion (Noise-Aware Training)

Rather than relying on static noise assumptions, the model incorporates a running noise estimate that updates frame-by-frame. This dynamic feature expansion allows the network to track and adapt to changing background conditions more effectively than models trained with stationary noise estimates.

2.2 Psychoacoustic Error Weighting

To align the model's performance with human perception, the standard Mean Squared Error (MSE) loss function is replaced with a Weighted Squared Error (WSE). This weighting is derived from two psychoacoustic principles:

  • Absolute Threshold of Hearing (ATH): Frequencies where the human ear is most sensitive receive higher weights.
  • Masking Thresholds: The model prioritizes errors in frequency regions where noise is most likely to be audible, ignoring errors in regions masked by louder sounds.

3. Experimental Setup

The models were trained using clean speech from the TIMIT dataset mixed with a custom collection of 95 office-environment noise samples. To simulate real-world complexity, clean speech was mixed with up to four simultaneous noises at Signal-to-Noise Ratios (SNR) ranging from -5 dB to 20 dB. The study also compared the impact of dataset scale, evaluating models trained on 25 hours versus 100 hours of data.

4. Results and Analysis

The DNN-based approaches consistently outperformed the classical Log-MMSE baseline across all evaluated metrics.

4.1 Intelligibility and Quality

As shown in the STOI performance chart, the proposed DNN strategies significantly improve the Short-Time Objective Intelligibility (STOI) score, particularly in high-noise scenarios. At -5 dB SNR, the DNN models achieved a 15-16% improvement in STOI over noisy speech, whereas the Log-MMSE baseline actually degraded intelligibility compared to the unprocessed signal.

STOI performance across different SNR levels for various DNN strategies and the Log-MMSE baseline.

In terms of speech quality, the best-performing DNN model (utilizing running noise estimates) demonstrated a 30% improvement in PESQ at low SNRs (-5 to 5 dB).

4.2 Noise Reduction and Distortion

Quantitative analysis of Noise Reduction (NR) and Speech Distortion (SD) further validates the superiority of the DNN approach. The psychoacoustic-weighted model (BSWD) achieves higher noise reduction and lower speech distortion than the baseline, especially at the most challenging noise levels.

Comparative analysis of Noise Reduction (NR) and Speech Distortion (SD) metrics for different enhancement models.

5. Conclusion

This research demonstrates that combining deep learning with dynamic noise estimation and human-centric psychoacoustic models significantly enhances speech in complex, multi-noise environments. While psychoacoustic weighting provides the most substantial benefits at very low SNRs, the overall robustness of the system scales effectively with increased training data. Future work will explore further architectural optimizations and generalization to a broader range of acoustic environments.

Summary updated

Abstract

In this paper we consider the problem of speech enhancement in real-world like conditions where multiple noises can simultaneously corrupt speech. Most of the current literature on speech enhancement focus primarily on presence of single noise in corrupted speech which is far from real-world environments. Specifically, we deal with improving speech quality in office environment where multiple stationary as well as non-stationary noises can be simultaneously present in speech. We propose several strategies based on Deep Neural Networks (DNN) for speech enhancement in these scenarios. We also investigate a DNN training strategy based on psychoacoustic models from speech coding for enhancement of noisy speech
BibTeX
@misc{kumar2016speech,
  author = {Anurag Kumar and Dinei Florêncio},
  title = {{Speech Enhancement In Multiple-Noise Conditions using Deep Neural Networks}},
  year = {2016},
  eprint = {1605.02427},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1605.02427},
}
Figure