Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models

Daniela de Albuquerque, John M. Pearson

Abstract

Beyond estimating parameters of interest from data, one of the key goals of statistical inference is to properly quantify uncertainty in these estimates. In Bayesian inference, this uncertainty is provided by the posterior distribution, the computation of which typically involves an intractable high-dimensional integral. Among available approximation methods, sampling-based approaches come with strong theoretical guarantees but scale poorly to large problems, while variational approaches scale well but offer few theoretical guarantees. In particular, variational methods are known to produce overconfident estimates of posterior uncertainty and are typically non-identifiable, with many latent variable configurations generating equivalent predictions. Here, we address these challenges by showing how diffusion-based models (DBMs), which have recently produced state-of-the-art performance in generative modeling tasks, can be repurposed for performing calibrated, identifiable Bayesian inference. By exploiting a previously established connection between the stochastic and probability flow ordinary differential equations (pfODEs) underlying DBMs, we derive a class of models, inflationary flows, that uniquely and deterministically map high-dimensional data to a lower-dimensional Gaussian distribution via ODE integration. This map is both invertible and neighborhood-preserving, with controllable numerical error, with the result that uncertainties in the data are correctly propagated to the latent space. We demonstrate how such maps can be learned via standard DBM training using a novel noise schedule and are effective at both preserving and reducing intrinsic data dimensionality. The result is a class of highly expressive generative models, uniquely defined on a low-dimensional latent space, that afford principled Bayesian inference.

Introduction

In modern statistical machine learning, a fundamental tension exists between generative modeling and principled Bayesian inference. While Markov Chain Monte Carlo (MCMC) methods provide exact uncertainty estimates, they are computationally prohibitive for high-dimensional data. Conversely, Variational Autoencoders (VAEs) offer scalability and data compression but are prone to overconfidence and non-identifiability. Diffusion-Based Models (DBMs) have emerged as the state-of-the-art for high-quality generation, yet their reliance on stochastic noise injection typically obscures the deterministic path back to a latent space required for rigorous inference.

This paper introduces Inflationary Flows, a framework that repurposes the generative power of diffusion for calibrated Bayesian inference. By leveraging the deterministic "probability flow" underlying DBMs, the authors create a unique, invertible mapping between high-dimensional data and a compressed latent space.

Methodology: The Inflationary Flow Framework

The core innovation lies in the "SDE-ODE Duality." For every stochastic differential equation (SDE) used in traditional diffusion models to add noise, there exists a corresponding deterministic Probability Flow Ordinary Differential Equation (pfODE) that describes the identical evolution of the probability density.

Conceptual overview comparing standard diffusion (SDE) with the deterministic Probability Flow ODE (pfODE) mapping.

Inflationary Flows utilize this pfODE to "inflate" a data distribution into a Gaussian latent space. Key technical components include:

  1. Deterministic Mapping: Unlike standard diffusion, which "mixes" data points with noise, the pfODE provides a bidirectional, neighborhood-preserving map. This ensures that the mapping between a specific data point and its latent representation is identifiable.
  2. Coordinate Rescaling: To maintain numerical stability during the inflation process, the authors introduce a time-dependent scaling factor that prevents variance explosion.
  3. Anisotropic Inflation: By controlling the "Participation Ratio" (PR)—a metric for intrinsic dimensionality—the model can inflate important dimensions while effectively "shriveling" others. This allows for massive data compression without sacrificing generative quality.

Experimental Validation and Key Results

The authors validated Inflationary Flows across various toy manifolds and complex image benchmarks, including CIFAR-10 and AFHQv2.

Visualization of the bidirectional mapping between data manifolds and Gaussian latent space for various toy datasets.

Superior Generative Quality and Compression

On the CIFAR-10 dataset, Inflationary Flows demonstrated significant improvements over existing injective flow models. For instance, at a latent dimensionality of 30, the proposed model achieved an FID score of 23.3, vastly outperforming the next best baseline, Canonical Manifold Flows (CMF), which scored 532.6. The model maintained high fidelity even when compressed to 0.03% of the original data dimensionality.

Calibrated Uncertainty Propagation

Because the mapping is an ODE, uncertainty in the data space can be mathematically propagated into the latent space. In experiments with Gaussian Mixture Models, the model's posterior estimates correctly covered ground-truth values. This suggests that the numerical errors inherent in learning the score function do not accumulate sufficiently to compromise Bayesian calibration.

Discussion and Future Directions

Inflationary Flows represent a significant step toward unifying high-quality generative modeling with identifiable Bayesian inference. By providing a deterministic, neighborhood-preserving map, they allow for principled uncertainty quantification in high-dimensional spaces.

However, some limitations remain. The Participation Ratio metric relies on second-order statistics, which may not fully capture the complexity of highly curved manifolds. Additionally, the required training schedule involves a wider range of noise than standard DBMs, presenting a trade-off in training difficulty. Future research may explore more dynamic, non-linear dimensionality metrics to further refine the compression process.

Summary updated

Abstract

Beyond estimating parameters of interest from data, one of the key goals of statistical inference is to properly quantify uncertainty in these estimates. In Bayesian inference, this uncertainty is provided by the posterior distribution, the computation of which typically involves an intractable high-dimensional integral. Among available approximation methods, sampling-based approaches come with strong theoretical guarantees but scale poorly to large problems, while variational approaches scale well but offer few theoretical guarantees. In particular, variational methods are known to produce overconfident estimates of posterior uncertainty and are typically non-identifiable, with many latent variable configurations generating equivalent predictions. Here, we address these challenges by showing how diffusion-based models (DBMs), which have recently produced state-of-the-art performance in generative modeling tasks, can be repurposed for performing calibrated, identifiable Bayesian inference. By exploiting a previously established connection between the stochastic and probability flow ordinary differential equations (pfODEs) underlying DBMs, we derive a class of models, inflationary flows, that uniquely and deterministically map high-dimensional data to a lower-dimensional Gaussian distribution via ODE integration. This map is both invertible and neighborhood-preserving, with controllable numerical error, with the result that uncertainties in the data are correctly propagated to the latent space. We demonstrate how such maps can be learned via standard DBM training using a novel noise schedule and are effective at both preserving and reducing intrinsic data dimensionality. The result is a class of highly expressive generative models, uniquely defined on a low-dimensional latent space, that afford principled Bayesian inference.
BibTeX
@inproceedings{albuquerque2024inflationary,
  author = {Daniela de Albuquerque and John M. Pearson},
  title = {{Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=IM4LtYRWdE},
}
Figure