Speech Enhancement for Noise-Robust Speech Synthesis Using Wasserstein GAN

Nagaraj Adiga, Yannis Pantazis, Vassilis Tsiaras, Yannis StylianouInterspeech 2019

TL;DR

This paper introduces a novel speech enhancement technique using a Wasserstein Generative Adversarial Network (WGAN) with gradient penalty and gated linear units (GLUs). This method significantly improves the quality of synthesized speech in noisy environments, outperforming existing methods like SEGAN, especially in mid-range signal-to-noise ratios (SNRs).

What Problem Does This Paper Address?

Speech synthesis systems typically require clean audio recordings to produce high-quality output. However, real-world scenarios, such as mobile phone recordings or the need to synthesize voices without access to pristine recordings, often introduce background noise. While existing speech enhancement (SE) techniques can reduce noise, they are often designed for tasks like automatic speech recognition (ASR) and may not perform optimally for speech synthesis (SS), sometimes even degrading quality. This paper addresses the need for an SE method specifically tailored for SS under moderate noise conditions (15-30 dB SNR), where traditional SE methods struggle and the noise energy is relatively low.

Methods

The core of the proposed method is a Speech Enhancement Generative Adversarial Network (SEGAN) architecture, but with key modifications. Instead of the standard GAN loss, the authors employ a Wasserstein GAN with Gradient Penalty (WGAN-GP). WGAN-GP is known for its more stable training and improved performance compared to traditional GANs. Additionally, they replace the standard activation functions with Gated Linear Units (GLUs), which have shown promise in capturing speech-specific temporal dependencies and improving speech quality.

The generator network is an autoencoder with skip connections, designed to operate directly on the waveform. The discriminator network is similarly structured. The WGAN-GP loss function encourages the generator to produce enhanced speech that is indistinguishable from clean speech by the discriminator, while the gradient penalty ensures stable training. The GLUs are integrated within the convolutional layers of both the generator and discriminator.

Datasets

  • Training Data: Voice bank corpus [18] (30 speakers: 15 male, 15 female, ~400 sentences per speaker).
  • Noise Data: BUTReverbDB database [19] (recordings from silent office and conference rooms).
  • Speech Synthesis Training: CMU-ARCTIC database [17] (SLT, BDL, CLB, RMS speakers) and additional speakers from the Voice bank corpus.

The authors created noisy speech by contaminating clean utterances from the Voice bank corpus with noise from BUTReverbDB at three different SNR levels: 15, 20, and 25 dB.

Key Insights and Ideas

  • WGAN-GP for Speech Enhancement: Applying WGAN-GP, known for its stability in image generation, proves beneficial for waveform-based speech enhancement, leading to more robust and higher-quality results than standard GANs.
  • GLUs for Speech Synthesis: Gated Linear Units are more effective than standard ReLU activations for speech tasks, likely due to their ability to model temporal dependencies crucial for natural-sounding speech.
  • Tailoring SE for SS: SEGAN, while effective for ASR, doesn't directly translate to SS. The proposed WGAN-GLU approach demonstrates that specific SE techniques can significantly boost SS quality, especially in moderate noise.
  • WaveNet's Sensitivity to Noise: Training a WaveNet vocoder on noisy data severely degrades speech synthesis quality. This highlights the critical need for effective pre-processing when using such powerful generative models.

Key Results

  • Objective Metrics: The proposed WGAN method showed significant improvements over SEGAN across various objective metrics:
  • PESQ: Increased by 0.2.
  • STOI: Increased by 0.02.
  • SSNR: Increased by approximately 1 dB.
  • CSIG: Improved from 3.57 to 3.69.
  • CBAK: Slightly better (3.46 to 3.48).
  • COVL: Improved by 0.10.
  • Subjective Evaluation (MOS):
    • For SE alone, WGAN achieved a MOS of 4.06 ± 0.17, compared to SEGAN's 3.08 ± 0.19.
  • For SS with the WGAN-enhanced speech, the MOS improved to 2.86 ± 0.19, a substantial gain over using noisy speech (2.01 ± 0.18) and a notable improvement over SEGAN-enhanced speech (2.20 ± 0.20).
  • Spectrogram Analysis: Visual inspection of spectrograms showed the WGAN-enhanced speech was closer to the clean speech than SEGAN-enhanced speech, with better noise suppression.

Limitations and Future Work

  • Limitations:
    • The current work focuses on additive noise in mid-SNR conditions.
  • The MOS for SS with WGAN-enhanced speech (2.86) is still below that of clean speech (3.57), indicating room for further improvement.
  • The study uses a parallel dataset setup for SE.
  • Future Work:
    • Extend the approach to non-parallel datasets for speech enhancement.
  • Investigate performance in more challenging noise conditions and types.
  • Further explore architectural and loss function improvements to bridge the gap between enhanced and clean speech synthesis quality.

Summary updated

BibTeX
@inproceedings{adiga2019speech,
  author = {Nagaraj Adiga and Yannis Pantazis and Vassilis Tsiaras and Yannis Stylianou},
  title = {{Speech Enhancement for Noise-Robust Speech Synthesis Using Wasserstein GAN}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure