Benchmarking Generative Latent Variable Models for Speech

Jakob D. Havtorn, Lasse Borgholt, Søren Hauberg, Jes Frellsen, Lars Maaløe

While stochastic latent variable models (LVMs) have revolutionized natural image generation, they have historically struggled to keep pace with deterministic models like WaveNet in the speech domain. This paper establishes a rigorous benchmarking framework to close this gap, introducing standardized likelihood metrics and demonstrating that hierarchical LVMs can actually learn speech representations superior to traditional engineered features.

TL;DR

This paper provides a much-needed standardized benchmark for temporal Latent Variable Models (LVMs) in speech, comparing them against state-of-the-art deterministic baselines using likelihood and phoneme recognition. The authors find that hierarchical LVMs—including a newly adapted Clockwork VAE—outperform non-hierarchical versions and learn latent representations that are more phonetically rich than standard Mel spectrograms.

What Problem Does This Paper Address?

In the image domain, LVMs like Variational Autoencoders (VAEs) are the gold standard. In speech, however, deterministic autoregressive models like WaveNet remain dominant. Comparing these two worlds has been notoriously difficult because speech researchers often omit likelihood scores or report them using incomparable assumptions about data distributions (e.g., varying bit depths or different output distributions like Gaussian vs. Categorical).

Without a "common language" for evaluation, it’s hard to tell if progress in LVMs is genuine. This paper addresses this by:

  1. Standardizing the likelihood metric to bits per frame (bpf).
  2. Using a Discretized Mixture of Logistics (DMoL) output distribution to handle the discrete nature of digital audio fairly across all models.
  3. Evaluating not just how well models generate audio, but how useful their internal "thoughts" (latent variables) are for downstream tasks like phoneme recognition.

Methods

The authors benchmark several popular temporal LVMs and compare them to deterministic baselines (LSTM and WaveNet). The generative architectures compared are visualized below:

Figure 1. Generative models for a single time step of a deterministic autoregressive LSTM, the VRNN and SRNN as well as the STCN and CW-VAE both with a single layer of latent variables. Red arrows indicate purely deterministic paths from the output xt to previous input x<t without passing a stochastic node. In an LSTM, information flows deterministically from previous observed variables to the next. The VRNN and SRNN use stochastic latent variables but also include deterministic skip connections from previous observed variables. The STCN is also autoregressive in x but does not use deterministic skip connections from xt−1 to xt. The CW-VAE is not autoregressive on the observed variable which forces information to flow through the latent variables. We provide more elaborate graphical illustrations including inference models of the CW-VAE in figure 2 and of the VRNN, SRNN and STCN in appendix J.

Key models include:

  • VRNN & SRNN: Early recurrent LVMs that act as VAEs per timestep.
  • STCN (Stochastic Temporal Convolutional Network): A hierarchical model using WaveNet-style dilated convolutions.
  • CW-VAE (Clockwork VAE): Originally for video, the authors adapted this hierarchical model to speech. Its defining feature is that different latent layers update at different "strides" (temporal resolutions), mimicking how speech has structure at both the phoneme level (fast) and the word/speaker level (slow).

Datasets

The models were trained and evaluated on three major speech corpora:

  • TIMIT: A classic dataset with high-quality phoneme-level annotations, used for both likelihood and phoneme recognition benchmarks.
  • LibriSpeech: A large-scale corpus of read English speech (100h subset used).
  • LibriLight: A subset of LibriSpeech designed for limited-supervision ASR (10h subset used).

Key Insights and Ideas

  • Hierarchy is Essential: Just as in image modeling, deeper hierarchies of stochastic variables lead to better speech models. Hierarchical models (STCN, CW-VAE) consistently beat their flatter counterparts (VRNN, SRNN).
  • The Resolution Trade-off: Deterministic models like WaveNet are superior at high temporal resolutions (modeling every single sample). However, as the resolution decreases (modeling "stacks" of samples), LVMs take the lead, suggesting they are better at capturing broader intra-step correlations.
  • Beyond Engineered Features: Perhaps the most striking finding is that LVMs learn latent spaces that are naturally organized by phonetics. As shown in the LDA visualization below, the model clusters similar sounds (like "s" and "sh") together without ever being told what a phoneme is.
Clustering of phonemes in the CW-VAE latent space.

Key Results

  • Phoneme Recognition: On TIMIT, representations learned by the STCN achieved a 21.9% Phoneme Error Rate (PER), significantly outperforming the 24.1% PER of the standard log Mel spectrogram. This advantage held even when labeled data was reduced to just 10 minutes.
  • Likelihood: While WaveNet still holds the crown at the highest resolution (10.88 bpf on TIMIT), hierarchical LVMs like the CW-VAE and STCN close the gap significantly at lower resolutions, outperforming both WaveNet and LSTMs.
  • Representation Quality: The latent variables of the CW-VAE ($z^1$ and $z^2$) provided much higher classification accuracy for phonemes than Mel features across varying numbers of neighbors in a K-Nearest Neighbor test.

Limitations and Future Work

The primary limitation is scalability. While LVMs show great promise, the best-performing models (STCN and CW-VAE) currently face numerical instability or computational bottlenecks when trying to model audio at the single-sample level ($s=1$).

Future research will likely focus on stabilizing these hierarchical models at high resolutions. If successful, LVMs could eventually replace deterministic models as the backbone for high-fidelity speech synthesis and more robust speech recognition systems.

Summary updated

Abstract

Stochastic latent variable models (LVMs) achieve state-of-the-art performance on natural image generation but are still inferior to deterministic models on speech. In this paper, we develop a speech benchmark of popular temporal LVMs and compare them against state-of-the-art deterministic models. We report the likelihood, which is a much used metric in the image domain, but rarely, or incomparably, reported for speech models. To assess the quality of the learned representations, we also compare their usefulness for phoneme recognition. Finally, we adapt the Clockwork VAE, a state-of-the-art temporal LVM for video generation, to the speech domain. Despite being autoregressive only in latent space, we find that the Clockwork VAE can outperform previous LVMs and reduce the gap to deterministic models by using a hierarchy of latent variables.
BibTeX
@misc{havtorn2022benchmarking,
  author = {Jakob D. Havtorn and Lasse Borgholt and Søren Hauberg and Jes Frellsen and Lars Maaløe},
  title = {{Benchmarking Generative Latent Variable Models for Speech}},
  year = {2022},
  eprint = {2202.12707},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.12707},
}
Figure