Abstract
Standard Variational Autoencoders (VAEs) for speech enhancement often struggle in noisy environments because they are typically trained only on clean data. This paper introduces a Noise-Aware VAE (NA-VAE) that utilizes a supervised latent mapping strategy. By training a noise-aware encoder to minimize the Kullback–Leibler (KL) divergence between noisy and clean latent distributions, the model achieves significant robustness in low signal-to-noise ratio (SNR) conditions and superior generalization to unseen noise types compared to traditional supervised models.
Introduction
In speech enhancement, the goal is to isolate target speech from background noise. VAEs have become popular for this task due to their ability to model the complex manifold of clean speech. However, a standard VAE encoder is often presented with noisy speech during inference that it did not see during training. At low SNRs, the encoder maps the noisy input to incorrect regions of the latent space, causing the decoder to generate distorted speech. While supervised Deep Neural Networks (DNNs) can be trained on noisy-clean pairs, they frequently overfit to specific training noises and fail to generalize to new environments.
Methodology
The authors propose a two-step learning strategy to create the NA-VAE:
- Unsupervised Pre-training: A standard VAE is trained on a large corpus of clean speech (WSJ0) to learn a robust latent representation ($z$) and a decoder ($p(s|z)$).
- Supervised Latent Mapping: A new "noise-aware" encoder is introduced and trained using pairs of noisy and clean speech. Rather than direct reconstruction, the objective is to minimize the KL divergence between the latent distribution produced by the noisy input and that produced by the clean input. This teaches the encoder to "see through" the noise and locate the correct latent coordinates on the clean speech manifold.
For enhancement, the system uses the VAE-NMF framework, modeling speech with the VAE and noise with Non-negative Matrix Factorization (NMF), combined via a Wiener filter.
Experimental Results
The model was evaluated using the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) across various SNR levels (-10 dB to 10 dB) using both seen (QUT-NOISE) and unseen (DEMAND) noise datasets.
- Superiority at Low SNR: In seen noise tests, the NA-VAE outperformed the standard VAE by nearly 1 dB at the difficult -10 dB SNR level.
- Robustness to Unseen Noise: The NA-VAE demonstrated exceptional generalization. When tested on the DEMAND dataset (unseen noise), it achieved an average SI-SDR of 11.88 dB, while a standard supervised DNN-Wiener Filter (DNN-WF) plummeted to 2.93 dB.
- Data Efficiency: The noise-aware encoder is highly efficient; using just 1% of the paired training data yielded noticeable improvements over the standard VAE.
Conclusion and Future Work
The NA-VAE effectively uses the latent space as a bridge to improve speech enhancement robustness without increasing parameter overhead. By focusing on latent alignment rather than direct spectral mapping, it avoids the overfitting common in supervised DNNs. Future research will focus on integrating temporal dynamics (e.g., via Transformers) and reducing the computational intensity of the inference algorithm for real-time applications.
Summary updated