Abstract
Neural latent variable models enable the discovery of interesting structure in speech audio data. This paper presents a comparison of two different approaches which are broadly based on predicting future time-steps or auto-encoding the input signal. Our study compares the representations learned by vq-vae and vq-wav2vec in terms of sub-word unit discovery and phoneme recognition performance. Results show that future time-step prediction with vq-wav2vec achieves better performance. The best system achieves an error rate of 13.22 on the ZeroSpeech 2019 ABX phoneme discrimination challenge.
Introduction
The advancement of speech recognition technology has historically been tethered to the availability of massive, high-quality labeled datasets. Supervised learning in this domain is notoriously "data hungry," often requiring thousands of hours of audio paired with precise transcriptions. To mitigate this dependency, researchers have turned to unsupervised representation learning. The goal is to extract meaningful, high-level features—such as phonemes or sub-word units—directly from raw audio without human intervention.
This paper investigates the efficacy of discrete latent variable models for this task, specifically comparing two dominant paradigms: signal reconstruction and context prediction.
Architectural Philosophies: Reconstruction vs. Prediction
The core of this comparison lies in the "pre-text" task used to train the model. Both models evaluated utilize Vector Quantization (VQ) to map continuous audio features into a discrete set of codewords, effectively creating a learned "alphabet" of speech sounds.
The Reconstructor: VQ-VAE
The Vector Quantized Variational Autoencoder (VQ-VAE) operates on the principle of autoencoding. It compresses input audio $x$ into a sequence of discrete latent codes $z_q$ via an encoder and a quantization layer. A decoder then attempts to reconstruct the original waveform $\hat{y}$ from these codes. The underlying assumption is that for the model to reconstruct the audio perfectly, the discrete codes must capture the essential structural information of the speech.
The Predictor: VQ-Wav2Vec
In contrast, VQ-Wav2Vec shifts the focus from reconstruction to prediction. While it also encodes audio into discrete latents, it does not attempt to rebuild the waveform. Instead, a context network processes the sequence of latents to predict future representations. This model uses a contrastive loss, challenging the network to distinguish the actual future latent from a set of "distractor" samples.
Experimental Methodology
The models were pre-trained on the LibriSpeech corpus (960 hours) and the ZeroSpeech 2019 training set (20 hours). Two primary quantization methods were tested:
- K-means clustering: Selecting the nearest neighbor in the codebook.
- Gumbel-Softmax: A differentiable method for sampling from a discrete distribution.
Evaluation was conducted using the TIMIT dataset for Phoneme Error Rate (PER) and the ZeroSpeech 2019 ABX task, which tests if a model can tell the difference between similar phonetic units (e.g., "bit" vs "bet") without any supervised training.
Results and Discussion
The experimental results demonstrate a clear advantage for the predictive approach. VQ-Wav2Vec consistently outperformed VQ-VAE across all benchmarks.
- Phoneme Recognition: On TIMIT, VQ-Wav2Vec (using Gumbel-Softmax) achieved a PER of 16.54%, significantly better than the 19.34% achieved by VQ-VAE.
- ABX Challenge: VQ-Wav2Vec (using K-means) reached a state-of-the-art error rate of 13.22 on the ZeroSpeech benchmark, surpassing previous models like VQ-CPC.
Why Prediction Wins
The superiority of VQ-Wav2Vec suggests that signal reconstruction may be an inefficient objective for speech representation. Reconstructing raw audio forces the model to capture low-level acoustic details—such as phase, background noise, and speaker-specific characteristics—that are often irrelevant to phonetic structure. By focusing on predicting future context, VQ-Wav2Vec is forced to model the higher-level dependencies and "rules" of the language, leading to more robust and phonetically specialized features.
Conclusion
This study confirms that context prediction is a more effective pre-text task than signal reconstruction for learning discrete speech representations. By ignoring the "noise" of raw audio reconstruction and focusing on the temporal structure of speech, VQ-Wav2Vec discovers units that more closely align with human phonetic categories. Future work will explore the application of these models to low-resource and tonal languages to further validate these findings.
Summary updated