Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models

Daniela de Albuquerque, John M. Pearson

TL;DR

Researchers have developed "Inflationary Flows," a new class of models that repurpose the power of diffusion for calibrated Bayesian inference. By exploiting the deterministic "probability flow" underlying diffusion models, they’ve created a way to map high-dimensional data into a compressed, low-dimensional latent space that preserves local structure and provides mathematically rigorous uncertainty estimates.

What Problem Does This Paper Address?

In the world of machine learning, there is a long-standing tension between generation (making new data) and inference (understanding the data you have).

  • MCMC (Markov Chain Monte Carlo) is the gold standard for inference because it provides exact uncertainty estimates, but it’s painfully slow and doesn't scale to high-dimensional images.
  • VAEs (Variational Autoencoders) are fast and compress data, but they are often "overconfident"—they underestimate how much they don't know—and produce blurry samples.
  • Diffusion-Based Models (DBMs) are currently the kings of generation (think DALL-E or Midjourney), but they are typically "noisy" and "mixing." They don't naturally compress data, and because they rely on random noise injection, they don't provide a clear, deterministic path back to a latent space for inference.

The authors aim to bridge this gap: they want a model that has the generative quality of diffusion but the compression and calibrated uncertainty of a principled Bayesian model.

Methods

The core innovation is the Inflationary Flow. It builds on the "SDE-ODE Duality"—the fact that for every noisy Stochastic Differential Equation (SDE) used in diffusion, there is a corresponding deterministic Probability Flow Ordinary Differential Equation (pfODE) that describes the same change in density.

Key technical components include:

  1. Deterministic Mapping: Instead of adding noise and "mixing" the data, they use the pfODE to "inflate" the distribution. This creates a bidirectional, neighborhood-preserving map between data and a Gaussian latent space.
  2. Coordinate Rescaling: To prevent the variance from exploding during this "inflation," they introduce a time-dependent scaling factor. This keeps the math numerically stable.
  3. Participation Ratio (PR) Control: They use a metric called the Participation Ratio to measure the intrinsic dimensionality of the data. By applying "anisotropic inflation"—inflating some dimensions faster than others—they can effectively "shrivel" less important dimensions, achieving massive data compression.
  4. Score-Matching Training: They train the model using standard denoising objectives but with a novel noise schedule designed to support this inflationary process.

Datasets

The authors validated their approach across a range of complexities:

  • Toy Datasets: 2D and 3D manifolds including Circles, Moons, S-Curves, and Swirls.
  • Image Benchmarks: CIFAR-10 (32x32 images of objects) and AFHQv2 (animal faces, downsampled to 32x32).
  • Comparison Baselines: They compared their results against existing "injective flow" models like M-Flow, Rectangular Flows (RFs), and Canonical Manifold Flows (CMFs).

Key Insights and Ideas

  • Identifiability is Key: Because the pfODE is unique and deterministic, the mapping between a specific image and its location in the latent space is "identifiable." This is a major upgrade over VAEs, where many different latent configurations can produce the same output.
  • Uncertainty Propagation: Because the map is an ODE, you can take a "ball" of uncertainty in the data space and mathematically propagate it into the latent space. The authors proved this works by using Hamiltonian Monte Carlo (HMC) to recover ground-truth posterior distributions.
  • Non-Linear Projection: Even though the compression is based on second-order statistics (the eigenbasis), the flow itself is non-linear. The score function "repels" particles based on local density, meaning the final compressed position of a data point depends on its neighbors, not just a simple linear projection.

Key Results

  • Superior Image Quality: On CIFAR-10, Inflationary Flows achieved significantly better FID scores (a measure of image quality) than other injective flow models. For example, at 30 dimensions, the IF model scored 23.3, while the next best baseline (CMF) scored 532.6.
  • Extreme Compression: The model maintained high generative quality even when reducing the latent space to as little as 0.03% of the original data dimensionality.
  • Calibration Success: In toy experiments with Gaussian Mixture Models, the model's posterior estimates correctly covered the ground-truth values, suggesting that numerical errors in the score function do not accumulate enough to ruin the inference.

Limitations and Future Work

  • Linear Dimensionality Metric: The Participation Ratio (PR) is based on second-order statistics (covariance). While effective, it might struggle with highly curved or complex manifolds where linear measures of "importance" fail.
  • Training Difficulty: The authors noted that their schedule requires training DBMs over a much larger range of noise than is typical, which can lead to trade-offs in performance as the "inflation gap" increases.
  • Fixed-Axis Compression: Currently, the model favors the top principal components. Future work could explore more sophisticated schedules that don't just remove information along fixed axes but adapt more dynamically to the data's geometry.

Summary updated

Abstract

Beyond estimating parameters of interest from data, one of the key goals of statistical inference is to properly quantify uncertainty in these estimates. In Bayesian inference, this uncertainty is provided by the posterior distribution, the computation of which typically involves an intractable high-dimensional integral. Among available approximation methods, sampling-based approaches come with strong theoretical guarantees but scale poorly to large problems, while variational approaches scale well but offer few theoretical guarantees. In particular, variational methods are known to produce overconfident estimates of posterior uncertainty and are typically non-identifiable, with many latent variable configurations generating equivalent predictions. Here, we address these challenges by showing how diffusion-based models (DBMs), which have recently produced state-of-the-art performance in generative modeling tasks, can be repurposed for performing calibrated, identifiable Bayesian inference. By exploiting a previously established connection between the stochastic and probability flow ordinary differential equations (pfODEs) underlying DBMs, we derive a class of models, \emph{inflationary flows,} that uniquely and deterministically map high-dimensional data to a lower-dimensional Gaussian distribution via ODE integration. This map is both invertible and neighborhood-preserving, with controllable numerical error, with the result that uncertainties in the data are correctly propagated to the latent space. We demonstrate how such maps can be learned via standard DBM training using a novel noise schedule and are effective at both preserving and reducing intrinsic data dimensionality. The result is a class of highly expressive generative models, uniquely defined on a low-dimensional latent space, that afford principled Bayesian inference.
BibTeX
@inproceedings{albuquerque2024inflationary,
  author = {Daniela de Albuquerque and John M. Pearson},
  title = {{Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=IM4LtYRWdE},
}
Figure