Abstract
Speech signals are inherently hierarchical, containing rapidly changing local information (phonemes) and persistent global characteristics (speaker identity). This paper introduces the Predictive Auxiliary Variational Autoencoder (Predictive Aux-VAE), an unsupervised model designed to disentangle these features. By incorporating a predictive auxiliary variable and a structured latent space, the model achieves a speaker identification accuracy of 98.1% on the LibriSpeech dataset, rivaling supervised benchmarks. Furthermore, the architecture enables controllable speech generation, such as voice transformation, by manipulating the learned global representations.
Introduction
A fundamental challenge in unsupervised speech processing is the separation of "what is being said" from "who is saying it." While standard Variational Autoencoders (VAEs) excel at data compression, they often fail to disentangle multi-scale information, leading to mixed latent representations. Previous approaches like the Factorised Hierarchical VAE (FHVAE) suffer from high computational costs, while methods like Contrastive Predictive Coding (CPC) lack generative capabilities. The Predictive Aux-VAE addresses these limitations by providing a scalable, fully convolutional framework that isolates global speech characteristics for both discriminative and generative tasks.
Architecture and Methodology
The core innovation of the Predictive Aux-VAE lies in its structured latent space, which is partitioned into an auxiliary variable ($h$) for global features and a local variable ($z$) for frame-by-frame content.
- Global and Local Latents: The global variable $h$ is generated by a CNN that processes a wide temporal window. A global average pooling layer acts as a bottleneck, "smearing" information across time to ensure $h$ only captures slow-moving characteristics. Conversely, $z$ captures fine-grained acoustic details.
- The Predictor Network: To ensure the global context influences local representations, the model includes a Predictor Network that estimates $h$ from $z$. This forces the local variable to remain "aware" of the global speaker context without explicitly encoding it.
- Training Objective: The model optimizes a modified Evidence Lower Bound (ELBO) with hyperparameters $\alpha$ and $\beta$ to balance reconstruction fidelity against the degree of disentanglement.
Results and Discussion
The model was evaluated using the LibriSpeech dataset, with input features consisting of amplitude spectrograms converted to a decibel scale.
Speaker Identification Performance The learned representations proved highly discriminative. Using a simple linear classifier, the model achieved 98.1% accuracy in speaker identification. This significantly outperforms standard MFCC features (17.4%) and the popular CPC model (97.4%), falling only 0.4% behind fully supervised models. Notably, the local variable $z$ performed better at identification (98.1%) than the global variable $h$ (95.3%), suggesting that high-frequency details in individual frames are vital for precise recognition.
Generative Control and Voice Transformation Because the Predictive Aux-VAE retains a decoder, it allows for explicit manipulation of the latent space. By swapping the global vector $h$ of a source speaker with that of a target speaker, the model can perform "voice transformation."
As shown in the spectrograms, the model successfully alters the vocal quality—such as pitch and harmonics—while preserving the underlying linguistic content. Robustness tests involving the addition of Gaussian noise to $h$ further confirmed that global texture can be modified without degrading the recognizability of the spoken words.
Conclusion
The Predictive Aux-VAE offers a powerful unsupervised approach to speech representation learning. By leveraging architectural bottlenecks and a predictive auxiliary objective, it achieves state-of-the-art disentanglement and identification performance. Future work will focus on improving generative audio quality through advanced decoders and exploring the model's performance in noisy, real-world environments.
Summary updated