Speech Enhancement Using Forked Generative Adversarial Networks with Spectral Subtraction

Ju Lin, Sufeng Niu, Zice Wei, Xiang Lan, A. van Wijngaarden, Melissa C. Smith, Kuang‐Ching Wang2019 · DOI

TL;DR

This paper introduces S-ForkGAN, a novel speech enhancement method that leverages a forked Generative Adversarial Network (GAN) operating on log-power spectra (LPS) to simultaneously extract speech and noise information. By combining spectral processing with a dual-decoder GAN architecture and spectral subtraction, S-ForkGAN achieves state-of-the-art performance in improving automatic speech recognition (ASR) accuracy, outperforming existing GAN-based methods.

What Problem Does This Paper Address?

Speech enhancement aims to clean up noisy audio to improve its intelligibility and perceptual quality, which is crucial for applications like automatic speech recognition (ASR), hearing aids, and mobile communication. Traditional methods like spectral subtraction and Wiener filters have limitations, while recent deep learning approaches, particularly Generative Adversarial Networks (GANs), have shown promise. However, many GAN-based methods operate directly on high-dimensional time-domain waveforms, demanding significant computational resources. Furthermore, effectively separating speech from noise and leveraging this separation for enhancement remains a challenge. This paper addresses the need for more efficient and effective speech enhancement techniques that can seamlessly integrate with ASR systems, which typically rely on spectral features.

Methods

The core innovation of this paper is the S-ForkGAN framework. Unlike previous GANs that might process raw audio or single-feature outputs, S-ForkGAN operates on log-power spectra (LPS). This choice offers several advantages:

  • Reduced dimensionality: LPS features are typically lower-dimensional than raw waveforms.
  • Seamless ASR integration: ASR systems often use spectral features, making direct integration easier.
  • Reinforced robustness: Processing in the frequency domain can inherently improve robustness.

The S-ForkGAN architecture features a forked GAN generator. This generator has a shared encoder that maps the input LPS to a latent representation. This latent representation is then split into two paths by a linear transformation: one for speech information and one for noise information. Two separate decoders then process these latent representations to estimate the clean speech and the noise, respectively.

Crucially, S-ForkGAN incorporates two novel auxiliary loss functions to guide the training:

  • Margin-based Loss: This loss encourages the learned speech and noise embeddings to be maximally dissimilar, ensuring a clear separation.
  • Spectral Subtraction Loss: This loss integrates traditional spectral subtraction principles by using the estimated noise to directly subtract from the noisy LPS, aiding in noise reduction.

The generator's overall loss is a weighted sum of the adversarial loss, the L1 regression loss for clean speech reconstruction, and these two new auxiliary losses.

Datasets

  • Clean Speech References: DARPA TIMIT corpus. This corpus contains American-English speech from 630 speakers, with a focus on phonetic richness.
  • Noise: NOISEX-92 corpus. This corpus includes 15 different types of noise.

For training, noise from NOISEX-92 was added to TIMIT speech segments at various Signal-to-Noise Ratios (SNRs) of -5 dB, 0 dB, 5 dB, and 10 dB. The test set was generated similarly.

Key Insights and Ideas

  • Spectral Processing is Key: Operating on log-power spectra (LPS) rather than raw waveforms offers significant advantages in terms of computational efficiency and direct compatibility with ASR systems.
  • Forked GAN for Dual Extraction: The "forked" architecture, with separate decoders for speech and noise, is effective in simultaneously learning and separating these two components. This dual extraction is more powerful than single-output enhancement.
  • Auxiliary Losses Enhance Performance: The margin-based and spectral subtraction losses are critical for improving the separation and reconstruction quality, going beyond standard GAN objectives.
  • Noise Information is Valuable: Explicitly modeling and extracting noise information, as done with the second decoder and spectral subtraction loss, directly benefits the speech enhancement process.

Key Results

The paper demonstrates that S-ForkGAN achieves state-of-the-art performance in speech enhancement, as measured by the phone error rate (PER) on the TIMIT dataset for ASR.

  • At an SNR of 5 dB, S-ForkGAN achieved a PER of 30.5%, outperforming previous GAN-based methods like GAN-LSTM (34.1%) and GAN-DNN (39.4%).
  • At the most challenging SNR of -5 dB, S-ForkGAN achieved 45.1% PER, again showing superiority over other methods.
  • Qualitative results (spectrograms) show that S-ForkGAN produces cleaner speech signals with less residual noise compared to other GAN-based methods.
  • The paper also highlights that LPS-based input features generally outperform raw waveform inputs for ASR tasks.

Limitations and Future Work

  • Limitations: While S-ForkGAN shows strong performance, the paper doesn't extensively discuss computational complexity comparisons beyond stating spectral processing is generally less demanding. The specific trade-offs between the number of epochs, learning rate, and final performance are detailed for the setup but not explored as a broad limitation.
  • Future Work: The authors suggest that the forked GAN framework for simultaneous speech and noise extraction is a promising direction. Further exploration of different spectral processing techniques and GAN architectures could yield even better results. The paper also implies that the effectiveness of LPS over raw audio for ASR tasks warrants further investigation.

Summary updated

BibTeX
@misc{lin2019speech,
  author = {Ju Lin and Sufeng Niu and Zice Wei and Xiang Lan and A. van Wijngaarden and Melissa C. Smith and Kuang‐Ching Wang},
  title = {{Speech Enhancement Using Forked Generative Adversarial Networks with Spectral Subtraction}},
  year = {2019},
  doi = {10.21437/interspeech.2019-2954},
}
Figure