Speech Enhancement with Variance Constrained Autoencoders

Daniel T. Braithwaite, W. Bastiaan KleijnInterspeech 2019 · DOI

Abstract

Speech Enhancement (SE) is critical for improving the quality and intelligibility of voice signals in noisy environments. While recent deep learning approaches like Generative Adversarial Networks (GANs) and WaveNets have shown promise in the time domain, they often suffer from high computational complexity and training instability. This paper introduces SE-VCAE, a model based on Variance Constrained Autoencoders. By relaxing the rigid distributional priors of standard Variational Autoencoders (VAEs) and employing a simplified 1D-Convolutional architecture, SE-VCAE achieves superior perceptual quality and robustness, particularly in low Signal-to-Noise Ratio (SNR) conditions.

Introduction

Traditional speech enhancement methods primarily operated in the frequency domain, often resulting in "robotic" artifacts due to the difficulty of accurately reconstructing phase information. Modern research has shifted toward time-domain processing of raw audio waves. However, state-of-the-art models like SE-GAN and SE-WaveNet come with significant drawbacks: GANs are notoriously difficult to stabilize during training, and WaveNet-style architectures require massive computational resources.

Furthermore, standard VAEs often struggle with "posterior collapse," where the model ignores the latent representation entirely, failing to capture the nuances of the input signal. SE-VCAE addresses these challenges by rethinking the latent space constraints and the objective function to prioritize both realism and content preservation.

Methodology: The Variance Constrained Autoencoder (VCAE)

The core innovation of this work is the Variance Constrained Autoencoder (VCAE). Unlike a standard VAE, which forces latent features into a specific Gaussian distribution, the VCAE merely constrains the variance of the latent space. This flexibility allows the model to map the complex manifold of clean speech more naturally without the restrictive bottleneck of a fixed prior.

The SE-VCAE architecture utilizes a stochastic encoder and a variance-constrained latent space to reconstruct clean speech.

Architecture and Loss Function

The model employs a straightforward 1D-Convolutional structure for both the encoder and decoder, avoiding the complexity of dilated convolutions or adversarial training. To ensure high-quality output, the authors utilize a tripartite loss function:

  1. Wasserstein Distance (WD): Serves as a distribution-matching term to ensure the enhanced speech sounds realistic.
  2. L1 Error: Minimizes the absolute difference between the enhanced signal and the ground-truth clean signal to preserve linguistic content.
  3. L1 Weight Regularization: Prevents overfitting to specific noise profiles, enhancing the model's ability to generalize to unseen environments.

Experimental Evaluation

The model was evaluated using the Voice Bank dataset, a standard benchmark in speech processing. The training set included 28 speakers with 10 types of noise (from the Demand dataset and artificial sources) at SNR levels ranging from 0 to 15 dB. To test generalization, the model was evaluated on two unseen speakers and five new noise types (e.g., public square, bus) at SNR levels of 2.5, 7.5, 12.5, and 17.5 dB.

Results and Key Insights

The performance was measured using a MUSHRA (Multiple Stimuli with Hidden Reference and Anchor) test, involving human listeners rating audio quality on a scale of 0 to 100.

  • Superior Perceptual Quality: SE-VCAE achieved an average MUSHRA score of 59.0, significantly outperforming SE-GAN (50.1) and SE-WaveNet (48.0).
  • Robustness in Noise: The model's advantage was most pronounced in high-noise environments (2.5 dB and 7.5 dB SNR), where it was consistently preferred over competing baselines.
  • Effective Latent Representation: By avoiding a Gaussian prior, the model successfully avoided posterior collapse, ensuring that the latent space remained meaningful and dependent on the input signal.
  • Generalization: The combination of a stochastic encoder and L1 regularization allowed the model to handle unseen noise types more effectively than more complex, over-parameterized architectures.

Conclusion and Future Work

SE-VCAE demonstrates that a well-posed generative framework can outperform heavier, more complex models. Its simpler 1D-CNN architecture and 37.5ms block size make it a strong candidate for real-time applications. Future research will focus on improving high-frequency reproduction through end-to-end architectural refinements and exploring disentangled latent representations to allow for independent control over speech characteristics like pitch and emotion.

Summary updated

BibTeX
@inproceedings{braithwaite2019speech,
  author = {Daniel T. Braithwaite and W. Bastiaan Kleijn},
  title = {{Speech Enhancement with Variance Constrained Autoencoders}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1809},
}
Figure