TL;DR
This paper introduces Automatic Multivariate Diffusion Training (AMDT), a framework that automates the design and optimization of complex, multi-variable noising processes for generative models. By moving beyond simple Gaussian noise to learned multivariate diffusions, the authors achieve state-of-the-art results with significantly smaller models—matching the performance of standard models while using only one-third of the parameters.
The Challenge of Multivariate Diffusions
Diffusion-based generative models (DBGMs) operate by gradually turning data into noise (the inference process) and then learning to reverse that process to generate new samples. Traditionally, researchers rely on simple, fixed noising processes like the Variance Preserving SDE (VPSDE).
Recent advancements, such as Critically-Damped Langevin Diffusion (CLD), have demonstrated that adding "auxiliary variables"—akin to adding "velocity" to a pixel's "position"—can dramatically improve sample quality. However, designing these multivariate diffusions is mathematically rigorous. Every new process requires a manual, case-by-case derivation of the Evidence Lower Bound (ELBO) and the transition kernels needed for training. This "math tax" has historically limited the field to a handful of well-known diffusion types, hindering rapid prototyping and experimentation.
Automated Learning for Multivariate Diffusions
To break this bottleneck, the authors propose Multivariate Diffusion Models (MDMs) and a training algorithm called AMDT. Their approach rests on three core pillars:
- Automatic Transition Kernels: Instead of solving complex differential equations by hand for every new diffusion, the framework uses a matrix factorization technique from filtering theory to compute the mean and covariance of the noising process analytically and generically.
- Stationary Parameterization: The authors use a recipe from gradient-based MCMC to parameterize the diffusion using two matrices: $Q$ (handling "mixing" or rotation between variables) and $D$ (handling "dissipation" or noise injection). This ensures the process always converges to the desired target distribution (the model prior).
- Learning the Diffusion: Because the underlying math is automated, the diffusion process itself becomes a learnable component. The model can optimize the $Q$ and $D$ matrices alongside standard score-model parameters to find the optimal way to noise data for a specific dataset.
Key Insights: Parameter Efficiency
One of the most striking findings is that a "smarter" diffusion process can compensate for a smaller neural network. We shouldn't assume standard Gaussian blurring is the best way to destroy information; by making the noising process learnable, the model finds more efficient paths through the latent space.
As shown above, a learned MDM with only 35.7 million parameters matched the performance (3.07 BPD) of a standard VPSDE-large model boasting 108 million parameters. Furthermore, the authors demonstrate that standard models like VPSDE and CLD are actually just specific, restricted cases of their broader $Q/D$ parameterization.
Key Results and New Diffusions
The researchers evaluated their framework on standard image generation benchmarks: MNIST, CIFAR-10, and ImageNet32.
- Performance Gains: On CIFAR-10, the learned MDMs outperformed both the standard VPSDE and the manually-derived CLD in terms of Bits-Per-Dimension (BPD), a standard metric for likelihood.
- Tighter Bounds: The MDM ELBO provides a tighter bound for existing models like CLD compared to previous evaluation methods (achieving $\leq 3.11$ BPD vs. the previous $3.31$ on CIFAR-10).
- Rapid Prototyping: Using their automated framework, the authors easily implemented two new types of diffusions—ALDA (Accelerated Langevin Diffusion) and MALDA—which would have previously required extensive manual derivation.
Limitations and Future Work
The primary limitation of the current implementation is that it treats data dimensions (pixels) as independent and identically distributed (i.i.d.) regarding the diffusion parameters. Consequently, the $Q$ and $D$ matrices are shared across all pixels.
Future work could explore non-i.i.d. diffusions, where different parts of an image or different data coordinates interact during the noising process. Additionally, applying these automated techniques to non-linear diffusions and more complex stationary distributions beyond the standard Gaussian prior presents an exciting frontier for generative modeling.
Summary updated