Guided Variational Autoencoder for Speech Enhancement With a Supervised Classifier

Guillaume Carbajal, Julius Richter, Timo Gerkmann

Abstract

This paper introduces the "Guided Variational Autoencoder" (VAE), a novel framework that bridges the gap between generative and supervised speech enhancement. By utilizing a separate supervised classifier to provide high-level structural "hints"—such as speech activity masks—to a VAE trained on clean speech, the system significantly improves speech reconstruction in highly noisy environments. Experimental results demonstrate that this approach outperforms both standard VAEs and traditional supervised deep learning models, particularly at low signal-to-noise ratios.

Introduction

Speech enhancement—the task of separating clean voice from background noise—has traditionally been approached through two distinct paradigms. Supervised Deep Neural Networks (DNNs) are trained on noisy-clean speech pairs; while effective at removing known noise, they often struggle to generalize to unseen environments. Conversely, Generative Models like Variational Autoencoders (VAEs) learn the distribution of clean speech in an unsupervised manner. While VAEs generalize better, they frequently become "confused" at low signal-to-noise ratios (SNRs), failing to distinguish speech from heavy background noise.

This research addresses this limitation by providing the VAE with a "guide": a high-level label that informs the model about the underlying speech structure even when noise is overwhelming.

Methodology: The Guided VAE Framework

The authors propose Model M2: The Labeled VAE. Unlike a standard VAE, the generative core of this model is conditioned on a categorical label ($y_n$).

The M2 model architecture

The Supervised Guide

A separate, supervised DNN classifier is trained to analyze noisy audio and predict these labels. Because this classifier is decoupled from the VAE, it can be specifically optimized for noise robustness. The researchers explored two types of labels:

  • Voice Activity Detection (VAD): A binary label indicating speech presence in a frame.
  • Ideal Binary Mask (IBM): A detailed label indicating which specific frequency bins contain speech.

Inference via MCEM

At test time, the system uses a Monte Carlo Expectation Maximization (MCEM) algorithm. It iteratively estimates the noise (using Non-negative Matrix Factorization) and the speech (using the Guided VAE) to reconstruct the clean signal. The supervised classifier provides the necessary guidance to keep the VAE focused on the speech components.

Inference process with supervised guidance

Experimental Evaluation

The model was trained on the WSJ0 (Wall Street Journal) corpus and the DEMAND noise dataset. Testing was conducted using the QUT-NOISE dataset, ensuring that test noise types and speakers were different from the training set to evaluate generalization. Performance was measured using the Scale-invariant signal-to-distortion ratio (SI-SDR).

Key Results and Insights

  • IBM as the Superior Guide: While VAD provides a slight performance boost, the IBM label is significantly more effective. It provides the VAE with a much more informed latent distribution, allowing for precise reconstruction.
  • Robustness at Low SNRs: The Guided VAE showed its greatest strength at -5 dB SNR. In these "whisper-in-a-storm" scenarios, the M2+IBM model reached an SI-SDR of 2.8 dB, whereas the purely supervised model failed, resulting in a negative score (-0.9 dB).
  • Overall Performance: The M2+IBM model achieved an average SI-SDR of 7.3 dB, significantly higher than the standard VAE (6.4 dB) and the supervised baseline (4.7 dB).

Conclusion and Future Work

The Guided VAE effectively combines the generalization capabilities of generative models with the robust detection of supervised learning. By decoupling the clean-speech prior from the noise-robust classifier, the system achieves state-of-the-art results in challenging acoustic conditions. Future work will focus on moving from offline MCEM processing to real-time applications and incorporating temporal context through Recurrent Neural Networks or Transformers.

Summary updated

Abstract

Recently, variational autoencoders have been successfully used to learn a probabilistic prior over speech signals, which is then used to perform speech enhancement. However, variational autoencoders are trained on clean speech only, which results in a limited ability of extracting the speech signal from noisy speech compared to supervised approaches. In this paper, we propose to guide the variational autoencoder with a supervised classifier separately trained on noisy speech. The estimated label is a high-level categorical variable describing the speech signal (e.g. speech activity) allowing for a more informed latent distribution compared to the standard variational autoencoder. We evaluate our method with different types of labels on real recordings of different noisy environments. Provided that the label better informs the latent distribution and that the classifier achieves good performance, the proposed approach outperforms the standard variational autoencoder and a conventional neural network-based supervised approach.
BibTeX
@inproceedings{carbajal2021guided,
  author = {Guillaume Carbajal and Julius Richter and Timo Gerkmann},
  title = {{Guided Variational Autoencoder for Speech Enhancement With a Supervised Classifier}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=4AEQ3M6-YN},
}
Figure