Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models

Daniela F De Albuquerque, John Pearson

TL;DR

This paper introduces Inflationary Flows, a new class of generative models that repurpose the deterministic "probability flow" of diffusion models for Bayesian inference. By using a novel noise schedule and coordinate rescaling, the authors create an invertible, neighborhood-preserving map that can compress high-dimensional data into a low-dimensional latent space while maintaining accurate, calibrated uncertainty estimates.

The Challenge of Bayesian Inference

In the world of statistical inference, researchers don't just want a "best guess" for a parameter; they want to quantify their uncertainty. Bayesian inference provides this via the posterior distribution, but calculating it in high dimensions is notoriously difficult.

Historically, researchers have been stuck between two extremes:

  1. Sampling-based methods (like MCMC): These are the gold standard for accuracy but are painfully slow and fail to scale to high-dimensional data like images.
  2. Variational methods (like VAEs): These are fast and scalable but often produce "overconfident" (poorly calibrated) results and suffer from non-identifiability—meaning many different latent configurations can produce the exact same output.

While Diffusion-Based Models (DBMs) have recently dominated generative AI, they aren't naturally built for inference. They typically map data to a noise distribution of the same dimension, offering no data compression. Furthermore, their stochastic nature makes it hard to track how uncertainty in the data propagates to the latent space.

Methods: Exploiting SDE-ODE Duality

The authors bridge this gap by exploiting the SDE-ODE Duality of diffusion models. While standard diffusion uses a Stochastic Differential Equation (SDE) to add noise, there exists a corresponding Probability Flow ODE (pfODE) that describes the exact same marginal distributions but is entirely deterministic.

SDE-ODE Duality and Probability Flow ODE

The paper introduces Inflationary Flows, which are pfODEs defined by three core components:

  • A Novel Noise Schedule: Instead of just adding isotropic noise, they "inflate" the data, smoothing it out on increasingly coarser scales.
  • Coordinate Rescaling: To prevent the variance from exploding to infinity, they apply a time-dependent rescaling that keeps the latent distribution as a stable, unit-variance Gaussian.
  • Dimensionality Control: They use a metric called the Participation Ratio (PR) to measure the effective dimensionality of the data. They derive two types of flows: dimension-preserving (keeps intrinsic dimensionality constant) and dimension-reducing (systematically shrinks less important dimensions for principled data compression).

Crucially, because the ODE is deterministic and invertible, local neighborhoods in the data space are preserved in the latent space. This allows the model to act as a "calibrated" bridge for Bayesian inference.

Key Insights and Results

The authors evaluated their model on a mix of synthetic toy datasets (2D and 3D distributions like Circles, Moons, and Swirls) and real-world image benchmarks (CIFAR-10 and AFHQv2).

Inflation and Generation on Toy Datasets
  • Deterministic Inference & Identifiability: By moving from SDEs to pfODEs, the authors transform a "noisy" generation process into a unique, bidirectional map. This makes the latent space identifiable, solving a major headache in variational inference.
  • Non-linear Compression: Unlike simple Principal Component Analysis (PCA), which is a linear projection, inflationary flows use the learned "score function" to repel particles from high-density areas. This results in a nonlinear mapping that can collapse complex manifolds into a low-dimensional Gaussian.
  • Superior Generative Quality: On CIFAR-10, Inflationary Flows achieved significantly better FID scores than existing injective flow models. For example, at 62 dimensions, IFs scored 23.2 compared to ~280 for the baselines.
  • Extreme Compression: The model successfully generated high-quality images even when compressed to as little as 0.03% of the original nominal dimensionality (e.g., reducing 3072 dimensions down to just 1 or 2).
  • Calibrated Posteriors: Using Hamiltonian Monte Carlo (HMC) on toy datasets, the authors showed that the posterior distributions inferred by the model correctly covered the ground-truth values, proving the model provides mathematically rigorous uncertainty estimates.

Limitations and Future Work

While powerful, Inflationary Flows have a few limitations. The current measure of dimensionality depends only on the covariance of the data (second-order statistics), meaning higher-order metrics might allow for even better compression of highly curved manifolds. Additionally, the "inflation" schedule requires training the model over a much larger range of noise than standard diffusion models, which can lead to longer training times. Finally, compression currently favors top principal components; future work could explore more "axis-agnostic" flows that don't rely on a pre-computed PCA.

Summary updated

Abstract

Beyond estimating parameters of interest from data, one of the key goals of statistical inference is to properly quantify uncertainty in these estimates. In Bayesian inference, this uncertainty is provided by the posterior distribution, the computation of which typically involves an intractable high-dimensional integral. Among available approximation methods, sampling-based approaches come with strong theoretical guarantees but scale poorly to large problems, while variational approaches scale well but offer few theoretical guarantees. In particular, variational methods are known to produce overconfident estimates of posterior uncertainty and are typically non-identifiable, with many latent variable configurations generating equivalent predictions. Here, we address these challenges by showing how diffusion-based models (DBMs), which have recently produced state-of-the-art performance in generative modeling tasks, can be repurposed for performing calibrated, identifiable Bayesian inference. By exploiting a previously established connection between the stochastic and probability flow ordinary differential equations (pfODEs) underlying DBMs, we derive a class of models, \emph{inflationary flows,} that uniquely and deterministically map high-dimensional data to a lower-dimensional Gaussian distribution via ODE integration. This map is both invertible and neighborhood-preserving, with controllable numerical error, with the result that uncertainties in the data are correctly propagated to the latent space. We demonstrate how such maps can be learned via standard DBM training using a novel noise schedule and are effective at both preserving and reducing intrinsic data dimensionality. The result is a class of highly expressive generative models, uniquely defined on a low-dimensional latent space, that afford principled Bayesian inference.
BibTeX
@inproceedings{albuquerque2024inflationary,
  author = {Daniela F De Albuquerque and John Pearson},
  title = {{Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=IM4LtYRWdE},
}
Figure