Abstract
Diffusion-based generative models (DBGMs) perturb data to a target noise distribution and reverse this process to generate samples. The choice of noising process, or inference diffusion process, affects both likelihoods and sample quality. For example, extending the inference process with auxiliary variables leads to improved sample quality. While there are many such multivariate diffusions to explore, each new one requires significant model-specific analysis, hindering rapid prototyping and evaluation. In this work, we study Multivariate Diffusion Models (MDMs). For any number of auxiliary variables, we provide a recipe for maximizing a lower-bound on the MDMs likelihood without requiring any model-specific analysis. We then demonstrate how to parameterize the diffusion for a specified target noise distribution; these two points together enable optimizing the inference diffusion process. Optimizing the diffusion expands easy experimentation from just a few well-known processes to an automatic search over all linear diffusions. To demonstrate these ideas, we introduce two new specific diffusions as well as learn a diffusion process on the MNIST, CIFAR10, and ImageNet32 datasets. We show learned MDMs match or surpass bits-per-dims (BPDs) relative to fixed choices of diffusions for a given dataset and model architecture.
Introduction
Diffusion models generate data by reversing a "noising" process that gradually transforms a data distribution into a simple target noise distribution. While standard models typically employ univariate noising—such as adding Gaussian noise to each pixel independently—recent advancements like Critically-Damped Langevin Diffusion (CLD) have demonstrated that incorporating auxiliary variables (e.g., "velocity") can significantly enhance sample quality and likelihoods.
Despite these benefits, a major bottleneck persists: every time a researcher wishes to explore a new multivariate diffusion process, they must manually derive complex mathematical components, including the Evidence Lower Bound (ELBO) and the transition kernels that describe how noise evolves over time. This tedious, model-specific analysis hinders rapid prototyping. This research introduces a framework to transform the choice of the diffusion process from a manual design task into an automated optimization problem.
Automated Multivariate Diffusion Training (AMDT)
To streamline the development of multivariate diffusions, the authors propose several key innovations under the umbrella of Multivariate Diffusion Models (MDMs):
1. Derivation-Free Prototyping
The core of the framework is Automatic Multivariate Diffusion Training (AMDT). This algorithm utilizes a matrix factorization technique to compute transition covariances analytically for any linear MDM. By using matrix exponentials of specific block matrices, the transition kernels for complex, high-dimensional auxiliary systems can be computed generically, eliminating the need for manual derivations.
2. Q/D Parameterization
To ensure the inference process always converges to the desired target noise distribution (the model prior), the researchers utilize results from gradient-based MCMC. They parameterize the diffusion using a skew-symmetric matrix ($Q$) and a positive semi-definite matrix ($D$). This parameterization guarantees that the stationary distribution of the noising process remains consistent with the intended prior.
3. Joint Learning of Diffusion and Score
Because the mathematical overhead is now automated, the framework allows for the joint optimization of the diffusion process parameters ($Q$ and $D$) alongside the score model. This enables the model to "learn" the most efficient noising path to maximize the ELBO for a specific dataset.
Experimental Evaluation
The researchers evaluated their approach on three standard image generation benchmarks: MNIST, CIFAR-10, and ImageNet32. The results highlight the impact of optimizing the inference process:
- State-of-the-Art Likelihoods: On CIFAR-10, the "Learned MDM" achieved a score of 3.07 bits-per-dimension (BPD). This outperformed both the standard VPSDE (3.20 BPD) and the specialized CLD (3.11 BPD) using the same underlying model architecture.
- Parameter Efficiency: The choice of diffusion process proved to be a powerful hyperparameter. A learned MDM with only 35.7 million parameters matched the performance of a significantly larger 108 million parameter model using standard VPSDE. This suggests that optimizing the noising path can be as effective as tripling the size of the neural network.
- Novel Diffusions: To demonstrate the flexibility of the framework, the authors introduced two new specific diffusions: Accelerated Langevin Diffusion (ALDA) and MALDA. These processes, which would have previously required significant manual analysis, were implemented and tested seamlessly.
Insights and Limitations
The Limits of Scalar Diffusion
A key theoretical insight of the paper is that the commonly used VPSDE is essentially the only scalar diffusion process with a stationary distribution. To discover more efficient or higher-quality noising paths, researchers must move into the multivariate space. MDMs provide the necessary tools to explore this space without the traditional mathematical burden.
Future Work
While highly effective, the current framework has limitations that provide avenues for future research:
- Linearity: The automated transition kernel computation is currently limited to linear time-varying processes. Extending this to non-linear diffusions would require more complex numerical methods.
- Spatial Interactions: The current experiments assume auxiliary variables for one pixel do not interact with those of another. Future work could explore "spatial" multivariate diffusions where auxiliary variables allow for interactions across different data coordinates.
- Modality Scope: The current study focuses on image data; the performance of automated multivariate diffusions on other modalities like audio or text remains an open question.
Conclusion
By automating the derivation and training of multivariate diffusion processes, this work lowers the barrier to entry for experimenting with complex diffusion physics. The results demonstrate that the "how" and "where" of the noising process are just as critical as the architecture of the generative model itself, offering a new path toward more efficient and high-fidelity generative modeling.
Summary updated