A Deep Neural Network Based Harmonic Noise Model for Speech Enhancement

Zhiheng Ouyang, Hongjiang Yu, Wei-Ping Zhu, Benoit ChampagneInterspeech 2018

TL;DR

This paper introduces a novel deep neural network (DNN) approach for speech enhancement that leverages a Harmonic Noise Model (HNM) to better capture the underlying structure of clean speech. By focusing on estimating the harmonic and residual amplitudes of clean speech, their method achieves superior speech quality and intelligibility, especially in challenging low-SNR environments, outperforming existing DNN-based techniques.

What Problem Does This Paper Address?

In real-world scenarios, speech is often contaminated by various types of background noise, significantly degrading its quality and intelligibility. This poses a challenge for numerous applications like voice communication and automatic speech recognition. While traditional methods like Wiener filtering and MMSE estimators have been used, they often introduce speech distortion or leave residual noise. More recent DNN-based methods have shown promise, but some either struggle with complex acoustic environments or, in the case of masking-based approaches, can inadvertently harm the clean speech signal. A key limitation of many existing methods is their failure to adequately exploit the inherent harmonic structure of voiced speech, which is crucial for perceptual quality and intelligibility, particularly at low signal-to-noise ratios (SNRs).

Methods

The core of this work is a novel speech enhancement system that combines a Harmonic Noise Model (HNM) with a Deep Neural Network (DNN). The HNM decomposes speech into a harmonic component (representing voiced sounds) and a residual component (representing unvoiced sounds and noise). The authors propose to use a DNN to learn a mapping from features extracted from noisy speech to the amplitudes of the HNM components of the clean speech.

Here's a breakdown of their approach:

  • Harmonic Noise Model (HNM): Speech s(n) is modeled as s(n) = h(n) + r(n), where h(n) is the harmonic component and r(n) is the residual. The harmonic component is further modeled as a sum of sinusoids at the fundamental frequency and its harmonics.
  • DNN Architecture: A DNN with one input layer, three hidden layers (each with 512 ReLU units), and one output layer is employed.
  • Feature Extraction: A rich set of features is used, including Amplitude Modulation Spectrum (AMS), RASTA-PLP, MFCC, and Gammatone filterbank energies. To capture temporal context, features from adjacent frames are concatenated into a single input vector.
  • Training Target: The DNN is trained to predict the amplitudes of the harmonic (Ahx) and residual (Arx) components of the clean speech HNM.
  • Residual Scaling: A key innovation is the introduction of a scaling factor (α) for the residual amplitude. This factor is adaptively determined based on the ratio of clean to noisy speech amplitude. The idea is to suppress the residual noise more aggressively when the speech is heavily degraded, while preserving it when the speech is relatively clean. This leads to three variations of their method:
  • bHNM-DNN: Basic target with α=1 (no residual suppression).
  • sHNM-DNN: Scaled target using the adaptive α.
  • HM-DNN: Full harmonic target with α=0 (residual completely discarded).
  • Reconstruction: The enhanced speech is reconstructed by combining the DNN-estimated clean speech HNM amplitudes with the noisy speech phase. The authors justify using the noisy phase due to the difficulty and computational cost of accurately estimating clean speech phase.

Datasets

  • Training: The IEEE corpus was used for clean speech.
  • Noises: Eight types of noises from the NOISEX-92 database were used, categorized into "seen" (babble, white, buccaneer, factory) and "unseen" (pink, hfchannel, destroyerops, f16) for evaluation.
  • Data Generation: Noisy speech was created by mixing clean training speech with seen noises at various SNRs (-10dB, -5dB, 0dB, 5dB), resulting in 4800 training utterances. For testing, both seen and unseen noises were mixed with clean test speech at the same SNRs, creating 1280 utterances for each noise type.

Key Insights and Ideas

  • Exploiting Harmonic Structure is Crucial: The paper strongly advocates for the importance of modeling the harmonic structure of speech, arguing that many prior methods overlook this, leading to performance degradation, especially in low-SNR conditions.
  • DNNs as Amplitude Predictors: Instead of directly predicting clean speech amplitude or a masking function, the DNN is used to predict the amplitudes of the HNM components. This nuanced approach seems to be a key to their success.
  • Adaptive Residual Scaling: The introduction of an adaptive scaling factor for the residual component is a significant contribution. It allows for a more balanced enhancement, suppressing noise where necessary without overly damaging the speech signal, leading to better perceptual quality.
  • Noisy Phase is Sufficient (and Practical): The decision to use the noisy speech phase for reconstruction, rather than attempting to estimate clean speech phase, is a pragmatic choice that simplifies the system while still yielding good results.

Key Results

The proposed HNM-DNN method consistently outperformed two other DNN-based methods (LPS-DNN and IRM-DNN) in terms of objective metrics like PESQ (Perceptual Evaluation of Speech Quality) and SSNR (Segmental SNR) across various SNRs and noise types.

  • Overall Superiority: HNM-DNN generally achieved better objective scores than LPS-DNN and IRM-DNN, particularly at low SNRs.
  • Impact of Residual Scaling:
    • The sHNM-DNN (scaled residual) variant generally achieved the best overall performance, showing a good balance between speech quality (PESQ) and intelligibility (STOI), especially when considering all objective metrics.
  • HM-DNN (full harmonic, no residual) performed best in terms of SSNR at low SNRs, suggesting its focus on voiced speech is beneficial.
  • bHNM-DNN (unscaled residual) showed the best STOI metric, indicating its preservation of the full speech amplitude is good for intelligibility.
  • Unseen Noises: The HNM-DNN method also demonstrated strong performance on unseen noise types, indicating good generalization capabilities.
  • Qualitative Findings: While not explicitly detailed with subjective listening test scores, the authors state that their method provides better speech quality and intelligibility, especially in low SNR environments.

Limitations and Future Work

  • Phase Estimation: The paper acknowledges that using the noisy speech phase is a practical choice, but suggests that synthesizing enhanced speech with the clean speech phase could potentially lead to even better performance. However, accurately estimating clean speech phase is a significant challenge.
  • Computational Complexity: While not explicitly detailed, the use of DNNs and STFT implies a certain level of computational cost. Future work could explore more efficient architectures or processing techniques.
  • Focus on Amplitude: The current method focuses on estimating HNM amplitudes. Future work could explore incorporating phase estimation or more sophisticated modeling of the HNM components.

Summary updated

BibTeX
@inproceedings{ouyang2018deep,
  author = {Zhiheng Ouyang and Hongjiang Yu and Wei-Ping Zhu and Benoit Champagne},
  title = {{A Deep Neural Network Based Harmonic Noise Model for Speech Enhancement}},
  booktitle = {Interspeech},
  year = {2018},
}
Figure