Unbiased learning of deep generative models with structured discrete representations

Harry Bendekgey, Gabriel Hope, Erik B. Sudderth

Introduction

In the world of generative modeling, researchers often face a classic trade-off between expressivity and interpretability. Deep models like standard Variational Autoencoders (VAEs) are incredibly expressive but act as "black boxes," making it difficult to incorporate domain knowledge or understand the underlying factors of variation. Conversely, Probabilistic Graphical Models (PGMs) are highly structured and interpretable but often struggle to capture high-dimensional, complex real-world data.

The Structured VAE (SVAE) was proposed to bridge this gap by combining the structural priors of PGMs with the flexible likelihoods of deep learning. However, it quickly hit an "optimization wall." Because SVAE inference requires an inner loop of iterative optimization for every data point, backpropagating through that loop is either a memory nightmare (if unrolled) or mathematically biased (if shortcuts are taken). Consequently, SVAEs have been largely overshadowed by RNNs and Transformers for time-series tasks.

This paper introduces three critical optimization innovations—implicit differentiation, unbiased natural gradients, and parallelized inference—that finally make SVAEs competitive with state-of-the-art deep learning models.

Methods: Overcoming the Optimization Wall

The authors propose three key fixes to make SVAEs tractable and robust:

1. Implicit Differentiation: Instead of storing every intermediate step of the inference optimization (which consumes massive GPU memory), the authors leverage the Implicit Function Theorem. This allows them to compute gradients at the "equilibrium" of the inference process. Crucially, they propose a "capped" version of this method that remains stable even if the inference hasn't perfectly converged, acting as a form of regularization and significantly speeding up training.

2. Unbiased Natural Gradients: Natural gradients are essential for fast convergence in probability space. However, previous SVAE implementations dropped a "correction term" to simplify the math, leading to biased updates. The authors derive a novel way to compute the full, unbiased natural gradient using automatic differentiation, requiring no manual derivations.

Training loss comparison showing the faster convergence of unbiased natural gradients.

3. Parallel Kalman Smoothing: To handle temporal data efficiently, the researchers generalized recent work on parallelizing Kalman smoothers. This allows the sequential logic of time-series inference to run efficiently on modern, highly parallel GPUs.

Datasets and Evaluation

The researchers evaluated their optimized SVAE against heavyweights like SRNNs and Transformers on two challenging benchmarks:

  • MOCAP (H3.6M): Human motion capture data consisting of 3D coordinates for 32 joints, testing the model's ability to capture complex physical dynamics.
  • WSJ0: Audio spectrograms from the Wall Street Journal dataset, testing high-dimensional (513-dim) data with rapid transitions.

Key Insights and Results

Once thought too difficult to train, the optimized SVAE now performs competitively with state-of-the-art SRNNs and Transformers in terms of log-likelihood and generation quality (measured by FID scores).

Interpretability and Multimodal Uncertainty: Because the SVAE uses a Switching Linear Dynamical System (SLDS) as its core, it can represent different "modes" of behavior. The SVAE successfully learned discrete states that correspond to human-understandable actions without explicit labels.

Visualization of learned discrete states on MOCAP data, showing how the SVAE captures semantic actions like walking, sitting, and posing.

Principled Imputation: Unlike RNNs, which are often "one-directional" and struggle to fill in missing data in the middle of a sequence, the SVAE uses its graphical model structure to perform smooth, bidirectional interpolation. In imputation tasks (filling in 80% of a missing sequence), the SVAE-SLDS significantly outperformed RNN-based models. On MOCAP data, the SVAE achieved an Interpolation FID of 2.8, while the SRNN lagged at 14.2.

Memory Efficiency: The implicit differentiation scheme allowed the model to handle large batch sizes that caused standard "unrolled" gradient methods to crash due to out-of-memory errors.

Limitations and Future Work

Despite these significant advances, the authors note that initialization remains tricky. If started with a completely random model, the discrete states can "collapse," where the model only uses one state for everything. To mitigate this, they currently employ a three-stage pre-training routine.

Future work could involve extending these unbiased learning techniques to even more complex graphical structures beyond time series, such as hierarchical models or graph-structured data. This could potentially bring the interpretability and structured reasoning of PGMs to a much wider range of deep learning applications.

Summary updated

Abstract

By composing graphical models with deep learning architectures, we learn generative models with the strengths of both frameworks. The structured variational autoencoder (SVAE) inherits structure and interpretability from graphical models, and flexible likelihoods for high-dimensional data from deep learning, but poses substantial optimization challenges. We propose novel algorithms for learning SVAEs, and are the first to demonstrate the SVAE's ability to handle multimodal uncertainty when data is missing by incorporating discrete latent variables. Our memory-efficient implicit differentiation scheme makes the SVAE tractable to learn via gradient descent, while demonstrating robustness to incomplete optimization. To more rapidly learn accurate graphical model parameters, we derive a method for computing natural gradients without manual derivations, which avoids biases found in prior work. These optimization innovations enable the first comparisons of the SVAE to state-of-the-art time series models, where the SVAE performs competitively while learning interpretable and structured discrete data representations.
BibTeX
@inproceedings{bendekgey2023unbiased,
  author = {Harry Bendekgey and Gabriel Hope and Erik B. Sudderth},
  title = {{Unbiased learning of deep generative models with structured discrete representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=aRDKqIuVH1},
}
Figure