A Wavenet for Speech Denoising.

Dario Rethage, Jordi Pons, Xavier Serra

TL;DR

This paper adapts the Wavenet architecture for speech denoising by making it non-causal and non-autoregressive, significantly improving its efficiency. The modified Wavenet predicts entire "target fields" of audio samples at once, leading to state-of-the-art denoising performance that outperforms traditional Wiener filtering.

What Problem Does This Paper Address?

Traditional speech processing, including denoising, often relies on magnitude spectrograms as a front-end. This approach discards the phase information of the audio signal, which can be crucial for reconstructing high-quality audio. While neural networks have shown promise in handling raw audio, existing models often suffer from high computational complexity due to their autoregressive nature (predicting one sample at a time based on previous ones). This paper tackles the challenge of developing an efficient and effective speech denoising system that operates directly on raw audio waveforms, preserving phase information and overcoming the computational bottlenecks of previous methods.

Methods

The core of this work is the adaptation of the Wavenet architecture for speech denoising. Key modifications include:

  • Non-Causality: Unlike the original Wavenet, which is causal (only uses past information), this adapted model uses non-causal, dilated convolutions. This allows the model to consider both past and future audio samples, which is beneficial for denoising.
  • Non-Autoregressive Prediction: Instead of predicting one sample at a time, the model predicts a "target field" of multiple samples in a single forward pass. This drastically reduces computational complexity and enables parallel processing.
  • Discriminative Learning: The model is trained in a supervised, discriminative manner by minimizing a regression loss, rather than modeling probability distributions as in generative Wavenet.
  • Real-valued Predictions: The model predicts real-valued audio samples directly, avoiding the issues associated with discrete softmax outputs and µ-law quantization, which could amplify noise.
  • Energy-Conserving Loss: A novel two-term loss function is introduced to enforce energy conservation between the input and output signals, helping to maintain the overall signal energy.
  • Speaker Conditioning: The model is conditioned on the speaker's identity, which acts as a bias term in the convolutions, improving performance and acting as a form of data augmentation.
  • 3x1 Filters for Continuity: To mitigate potential temporal discontinuities introduced by removing autoregression, the final two layers use 3x1 filters, which helps maintain adjacent audio continuity.

Datasets

  • Voice Bank Corpus: Used for speech data, featuring 30 native English speakers.
  • Diverse Environments Multichannel Acoustic Noise Database (DEMAND): Used for environmental sounds (background noise).
  • Synthetically Mixed Data: Training samples were created by mixing speech from the Voice Bank with noise from DEMAND at various Signal-to-Noise Ratios (SNRs: 0, 5, 10, 15 dB) and different noise types. Test samples were mixed with unseen speakers and noise conditions at different SNRs (2.5, 7.5, 12.5, 17.5 dB).

Key Insights and Ideas

  • Phase Matters: Operating directly on raw audio and preserving phase information is crucial for high-quality speech denoising, outperforming methods that rely solely on magnitude spectrograms.
  • Non-Causality is Key for Denoising: For tasks like denoising, where some latency is acceptable, leveraging future information through non-causal convolutions significantly improves prediction quality.
  • Parallelism for Efficiency: Predicting target fields instead of single samples is a major breakthrough for reducing computational complexity, making the model practical for real-world applications.
  • Discriminative Models for Specific Tasks: Adapting a generative model like Wavenet into a discriminative one for a specific task (denoising) can yield better performance and efficiency.
  • Data Augmentation for Silence: Noise-only data augmentation was essential for enabling the model to generate silence effectively.

Key Results

  • Superior Denoising: The proposed Wavenet adaptation significantly outperforms Wiener filtering in terms of background-noise suppression (OVL and BAK measures).
  • High Perceptual Quality: Subjective listening tests (MOS scores) showed that participants significantly preferred the proposed method over Wiener filtering (3.60 vs. 2.92).
  • Efficiency Gains: The non-autoregressive, target-field prediction approach leads to a denoising time of approximately 0.56 seconds per second of audio on a GPU.
  • Effective Conditioning: Speaker conditioning improved denoising performance, albeit marginally.
  • Impact of Target Field Size: Longer target fields (1601 samples) were crucial for achieving significant denoising and avoiding fuzzy audio.

Limitations and Future Work

  • Temporal Continuity: While 3x1 filters help, the removal of autoregression means temporal continuity is not strictly enforced, which can still lead to sporadic discontinuities.
  • Sudden Interferences: The model struggles with sudden, transient noises like honks in city traffic.
  • Future Work: The authors suggest exploring alternative loss functions motivated by domain knowledge and investigating the impact of their proposed loss in more complex scenarios. They also note that their model inherently estimates both speech and background noise, hinting at potential for source separation tasks.

Summary updated

Abstract

Most speech processing techniques use magnitude spectrograms as front-end and are therefore by default discarding part of the signal: the phase. In order to overcome this limitation' we propose an end-to-end learning method for speech denoising based on Wavenet. The proposed model adaptation retains Wavenet's powerful acoustic modeling capabilities, while significantly reducing its time-complexity by eliminating its autoregressive nature. Specifically, the model makes use of non-causal, dilated convolutions and predicts target fields instead of a single target sample. The discriminative adaptation of the model we propose, learns in a supervised fashion via minimizing a regression loss. These modifications make the model highly parallelizable during both training and inference. Both quantitative and qualitative evaluations indicate that the proposed method is preferred over Wiener filtering, a common method based on processing the magnitude spectrogram.
BibTeX
@inproceedings{rethage2018wavenet,
  author = {Dario Rethage and Jordi Pons and Xavier Serra},
  title = {{A Wavenet for Speech Denoising.}},
  booktitle = {ICASSP},
  year = {2018},
  url = {https://openreview.net/forum?id=ZMVY7ALR_Fi},
}
Figure