Where to Diffuse, How to Diffuse, and How to Get Back: Automated Learning for Multivariate Diffusions

Raghav Singhal, Mark Goldstein, Rajesh Ranganath

TL;DR

This paper introduces Automatic Multivariate Diffusion Training (AMDT), a framework that automates the design and optimization of complex, multi-variable noising processes for generative models. By moving beyond simple Gaussian noise to learned multivariate diffusions, the authors achieve state-of-the-art results with significantly smaller models—matching the performance of standard models while using only one-third of the parameters.

The Challenge of Multivariate Diffusions

Diffusion-based generative models (DBGMs) operate by gradually turning data into noise (the inference process) and then learning to reverse that process to generate new samples. Traditionally, researchers rely on simple, fixed noising processes like the Variance Preserving SDE (VPSDE).

Recent advancements, such as Critically-Damped Langevin Diffusion (CLD), have demonstrated that adding "auxiliary variables"—akin to adding "velocity" to a pixel's "position"—can dramatically improve sample quality. However, designing these multivariate diffusions is mathematically rigorous. Every new process requires a manual, case-by-case derivation of the Evidence Lower Bound (ELBO) and the transition kernels needed for training. This "math tax" has historically limited the field to a handful of well-known diffusion types, hindering rapid prototyping and experimentation.

Automated Learning for Multivariate Diffusions

To break this bottleneck, the authors propose Multivariate Diffusion Models (MDMs) and a training algorithm called AMDT. Their approach rests on three core pillars:

  1. Automatic Transition Kernels: Instead of solving complex differential equations by hand for every new diffusion, the framework uses a matrix factorization technique from filtering theory to compute the mean and covariance of the noising process analytically and generically.
  2. Stationary Parameterization: The authors use a recipe from gradient-based MCMC to parameterize the diffusion using two matrices: $Q$ (handling "mixing" or rotation between variables) and $D$ (handling "dissipation" or noise injection). This ensures the process always converges to the desired target distribution (the model prior).
  3. Learning the Diffusion: Because the underlying math is automated, the diffusion process itself becomes a learnable component. The model can optimize the $Q$ and $D$ matrices alongside standard score-model parameters to find the optimal way to noise data for a specific dataset.

Key Insights: Parameter Efficiency

One of the most striking findings is that a "smarter" diffusion process can compensate for a smaller neural network. We shouldn't assume standard Gaussian blurring is the best way to destroy information; by making the noising process learnable, the model finds more efficient paths through the latent space.

Parameter Efficiency Comparison

As shown above, a learned MDM with only 35.7 million parameters matched the performance (3.07 BPD) of a standard VPSDE-large model boasting 108 million parameters. Furthermore, the authors demonstrate that standard models like VPSDE and CLD are actually just specific, restricted cases of their broader $Q/D$ parameterization.

Key Results and New Diffusions

The researchers evaluated their framework on standard image generation benchmarks: MNIST, CIFAR-10, and ImageNet32.

Performance Across Datasets
  • Performance Gains: On CIFAR-10, the learned MDMs outperformed both the standard VPSDE and the manually-derived CLD in terms of Bits-Per-Dimension (BPD), a standard metric for likelihood.
  • Tighter Bounds: The MDM ELBO provides a tighter bound for existing models like CLD compared to previous evaluation methods (achieving $\leq 3.11$ BPD vs. the previous $3.31$ on CIFAR-10).
  • Rapid Prototyping: Using their automated framework, the authors easily implemented two new types of diffusions—ALDA (Accelerated Langevin Diffusion) and MALDA—which would have previously required extensive manual derivation.

Limitations and Future Work

The primary limitation of the current implementation is that it treats data dimensions (pixels) as independent and identically distributed (i.i.d.) regarding the diffusion parameters. Consequently, the $Q$ and $D$ matrices are shared across all pixels.

Future work could explore non-i.i.d. diffusions, where different parts of an image or different data coordinates interact during the noising process. Additionally, applying these automated techniques to non-linear diffusions and more complex stationary distributions beyond the standard Gaussian prior presents an exciting frontier for generative modeling.

Summary updated

Abstract

Diffusion-based generative models (DBGMs) perturb data to a target noise distribution and reverse this process to generate samples. The choice of noising process, or inference diffusion process, affects both likelihoods and sample quality. For example, extending the inference process with auxiliary variables leads to improved sample quality. While there are many such multivariate diffusions to explore, each new one requires significant model-specific analysis, hindering rapid prototyping and evaluation. In this work, we study Multivariate Diffusion Models (MDMs). For any number of auxiliary variables, we provide a recipe for maximizing a lower-bound on the MDMs likelihood without requiring any model-specific analysis. We then demonstrate how to parameterize the diffusion for a specified target noise distribution; these two points together enable optimizing the inference diffusion process. Optimizing the diffusion expands easy experimentation from just a few well-known processes to an automatic search over all linear diffusions. To demonstrate these ideas, we introduce two new specific diffusions as well as learn a diffusion process on the MNIST, CIFAR10, and ImageNet32 datasets. We show learned MDMs match or surpass bits-per-dims (BPDs) relative to fixed choices of diffusions for a given dataset and model architecture.
BibTeX
@inproceedings{singhal2023where,
  author = {Raghav Singhal and Mark Goldstein and Rajesh Ranganath},
  title = {{Where to Diffuse, How to Diffuse, and How to Get Back: Automated Learning for Multivariate Diffusions}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=osei3IzUia},
}
Figure