Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder

Huajian Fang, Guillaume Carbajal, Stefan Wermter, Timo Gerkmann

Abstract

Standard Variational Autoencoders (VAEs) for speech enhancement often struggle in noisy environments because they are typically trained only on clean data. This paper introduces a Noise-Aware VAE (NA-VAE) that utilizes a supervised latent mapping strategy. By training a noise-aware encoder to minimize the Kullback–Leibler (KL) divergence between noisy and clean latent distributions, the model achieves significant robustness in low signal-to-noise ratio (SNR) conditions and superior generalization to unseen noise types compared to traditional supervised models.

Introduction

In speech enhancement, the goal is to isolate target speech from background noise. VAEs have become popular for this task due to their ability to model the complex manifold of clean speech. However, a standard VAE encoder is often presented with noisy speech during inference that it did not see during training. At low SNRs, the encoder maps the noisy input to incorrect regions of the latent space, causing the decoder to generate distorted speech. While supervised Deep Neural Networks (DNNs) can be trained on noisy-clean pairs, they frequently overfit to specific training noises and fail to generalize to new environments.

Methodology

The authors propose a two-step learning strategy to create the NA-VAE:

  1. Unsupervised Pre-training: A standard VAE is trained on a large corpus of clean speech (WSJ0) to learn a robust latent representation ($z$) and a decoder ($p(s|z)$).
  2. Supervised Latent Mapping: A new "noise-aware" encoder is introduced and trained using pairs of noisy and clean speech. Rather than direct reconstruction, the objective is to minimize the KL divergence between the latent distribution produced by the noisy input and that produced by the clean input. This teaches the encoder to "see through" the noise and locate the correct latent coordinates on the clean speech manifold.
The proposed NA-VAE training scheme using KL divergence to align noisy and clean latent representations.

For enhancement, the system uses the VAE-NMF framework, modeling speech with the VAE and noise with Non-negative Matrix Factorization (NMF), combined via a Wiener filter.

Experimental Results

The model was evaluated using the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) across various SNR levels (-10 dB to 10 dB) using both seen (QUT-NOISE) and unseen (DEMAND) noise datasets.

  • Superiority at Low SNR: In seen noise tests, the NA-VAE outperformed the standard VAE by nearly 1 dB at the difficult -10 dB SNR level.
  • Robustness to Unseen Noise: The NA-VAE demonstrated exceptional generalization. When tested on the DEMAND dataset (unseen noise), it achieved an average SI-SDR of 11.88 dB, while a standard supervised DNN-Wiener Filter (DNN-WF) plummeted to 2.93 dB.
  • Data Efficiency: The noise-aware encoder is highly efficient; using just 1% of the paired training data yielded noticeable improvements over the standard VAE.
Performance comparison (SI-SDR) on unseen noise (DEMAND dataset) across various SNR levels, highlighting the NA-VAE's superior generalization.

Conclusion and Future Work

The NA-VAE effectively uses the latent space as a bridge to improve speech enhancement robustness without increasing parameter overhead. By focusing on latent alignment rather than direct spectral mapping, it avoids the overfitting common in supervised DNNs. Future research will focus on integrating temporal dynamics (e.g., via Transformers) and reducing the computational intensity of the inference algorithm for real-time applications.

Summary updated

Abstract

Recently, a generative variational autoencoder (VAE) has been proposed for speech enhancement to model speech statistics. However, this approach only uses clean speech in the training phase, making the estimation particularly sensitive to noise presence, especially in low signal-to-noise ratios (SNRs). To increase the robustness of the VAE, we propose to include noise information in the training phase by using a noise-aware encoder trained on noisy-clean speech pairs. We evaluate our approach on real recordings of different noisy environments and acoustic conditions using two different noise datasets. We show that our proposed noise-aware VAE outperforms the standard VAE in terms of overall distortion without increasing the number of model parameters. At the same time, we demonstrate that our model is capable of generalizing to unseen noise conditions better than a supervised feedforward deep neural network (DNN). Furthermore, we demonstrate the robustness of the model performance to a reduction of the noisy-clean speech training data size.
BibTeX
@inproceedings{fang2021variational,
  author = {Huajian Fang and Guillaume Carbajal and Stefan Wermter and Timo Gerkmann},
  title = {{Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=9Jpt5bo1x6},
}
Figure