Predictive Auxiliary Variational Autoencoder for Representation Learning of Global Speech Characteristics

Sebastian Springenberg, Egor Lakomkin, Cornelius Weber, Stefan Wermter

Abstract

Speech signals are inherently hierarchical, containing rapidly changing local information (phonemes) and persistent global characteristics (speaker identity). This paper introduces the Predictive Auxiliary Variational Autoencoder (Predictive Aux-VAE), an unsupervised model designed to disentangle these features. By incorporating a predictive auxiliary variable and a structured latent space, the model achieves a speaker identification accuracy of 98.1% on the LibriSpeech dataset, rivaling supervised benchmarks. Furthermore, the architecture enables controllable speech generation, such as voice transformation, by manipulating the learned global representations.

Introduction

A fundamental challenge in unsupervised speech processing is the separation of "what is being said" from "who is saying it." While standard Variational Autoencoders (VAEs) excel at data compression, they often fail to disentangle multi-scale information, leading to mixed latent representations. Previous approaches like the Factorised Hierarchical VAE (FHVAE) suffer from high computational costs, while methods like Contrastive Predictive Coding (CPC) lack generative capabilities. The Predictive Aux-VAE addresses these limitations by providing a scalable, fully convolutional framework that isolates global speech characteristics for both discriminative and generative tasks.

Architecture and Methodology

The core innovation of the Predictive Aux-VAE lies in its structured latent space, which is partitioned into an auxiliary variable ($h$) for global features and a local variable ($z$) for frame-by-frame content.

Figure 1: The Predictive Aux-VAE architecture, illustrating the Encoder, Predictor, and Decoder modules.
  1. Global and Local Latents: The global variable $h$ is generated by a CNN that processes a wide temporal window. A global average pooling layer acts as a bottleneck, "smearing" information across time to ensure $h$ only captures slow-moving characteristics. Conversely, $z$ captures fine-grained acoustic details.
  2. The Predictor Network: To ensure the global context influences local representations, the model includes a Predictor Network that estimates $h$ from $z$. This forces the local variable to remain "aware" of the global speaker context without explicitly encoding it.
  3. Training Objective: The model optimizes a modified Evidence Lower Bound (ELBO) with hyperparameters $\alpha$ and $\beta$ to balance reconstruction fidelity against the degree of disentanglement.

Results and Discussion

The model was evaluated using the LibriSpeech dataset, with input features consisting of amplitude spectrograms converted to a decibel scale.

Speaker Identification Performance The learned representations proved highly discriminative. Using a simple linear classifier, the model achieved 98.1% accuracy in speaker identification. This significantly outperforms standard MFCC features (17.4%) and the popular CPC model (97.4%), falling only 0.4% behind fully supervised models. Notably, the local variable $z$ performed better at identification (98.1%) than the global variable $h$ (95.3%), suggesting that high-frequency details in individual frames are vital for precise recognition.

Generative Control and Voice Transformation Because the Predictive Aux-VAE retains a decoder, it allows for explicit manipulation of the latent space. By swapping the global vector $h$ of a source speaker with that of a target speaker, the model can perform "voice transformation."

Figure 2: Demonstration of speaker transformation from male to female by swapping the global latent variable $h$.

As shown in the spectrograms, the model successfully alters the vocal quality—such as pitch and harmonics—while preserving the underlying linguistic content. Robustness tests involving the addition of Gaussian noise to $h$ further confirmed that global texture can be modified without degrading the recognizability of the spoken words.

Conclusion

The Predictive Aux-VAE offers a powerful unsupervised approach to speech representation learning. By leveraging architectural bottlenecks and a predictive auxiliary objective, it achieves state-of-the-art disentanglement and identification performance. Future work will focus on improving generative audio quality through advanced decoders and exploring the model's performance in noisy, real-world environments.

Summary updated

Abstract

Unsupervised learning represents an important opportunity for obtaining useful speech representations. Recently, variational autoencoders (VAEs) have been shown to extract useful representations in an unsupervised manner. These models are usually not designed to explicitly disentangle specific sources of information. When processing data of sequential nature which involves multi-timescale information, disentanglement can however be beneficial. In this paper we address this issue by developing a predictive auxiliary variational autoencoder to obtain speech representations at different timescales. We will present an auxiliary lower bound which is used to develop a model that we call the Predictive Aux-VAE. The model is designed to disentangle global from local information into a dedicated auxiliary variable. Learned representations are analysed with respect to their ability to capture global speech characteristics. We observe that representations of individual speakers are separated well in the latent space and can successfully be used in a subsequent speaker identification task where they achieve high classification accuracy, comparable to a fully supervised model. Moreover, manipulating the global variable allows to change global characteristics while retaining the local content during generation which demonstrates the success of our model to disentangle global from local information.
BibTeX
@inproceedings{springenberg2019predictive,
  author = {Sebastian Springenberg and Egor Lakomkin and Cornelius Weber and Stefan Wermter},
  title = {{Predictive Auxiliary Variational Autoencoder for Representation Learning of Global Speech Characteristics}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2845},
}
Figure