TL;DR
This paper introduces Inflationary Flows, a new class of generative models that repurpose the deterministic "probability flow" of diffusion models for Bayesian inference. By using a novel noise schedule and coordinate rescaling, the authors create an invertible, neighborhood-preserving map that can compress high-dimensional data into a low-dimensional latent space while maintaining accurate, calibrated uncertainty estimates.
The Challenge of Bayesian Inference
In the world of statistical inference, researchers don't just want a "best guess" for a parameter; they want to quantify their uncertainty. Bayesian inference provides this via the posterior distribution, but calculating it in high dimensions is notoriously difficult.
Historically, researchers have been stuck between two extremes:
- Sampling-based methods (like MCMC): These are the gold standard for accuracy but are painfully slow and fail to scale to high-dimensional data like images.
- Variational methods (like VAEs): These are fast and scalable but often produce "overconfident" (poorly calibrated) results and suffer from non-identifiability—meaning many different latent configurations can produce the exact same output.
While Diffusion-Based Models (DBMs) have recently dominated generative AI, they aren't naturally built for inference. They typically map data to a noise distribution of the same dimension, offering no data compression. Furthermore, their stochastic nature makes it hard to track how uncertainty in the data propagates to the latent space.
Methods: Exploiting SDE-ODE Duality
The authors bridge this gap by exploiting the SDE-ODE Duality of diffusion models. While standard diffusion uses a Stochastic Differential Equation (SDE) to add noise, there exists a corresponding Probability Flow ODE (pfODE) that describes the exact same marginal distributions but is entirely deterministic.
The paper introduces Inflationary Flows, which are pfODEs defined by three core components:
- A Novel Noise Schedule: Instead of just adding isotropic noise, they "inflate" the data, smoothing it out on increasingly coarser scales.
- Coordinate Rescaling: To prevent the variance from exploding to infinity, they apply a time-dependent rescaling that keeps the latent distribution as a stable, unit-variance Gaussian.
- Dimensionality Control: They use a metric called the Participation Ratio (PR) to measure the effective dimensionality of the data. They derive two types of flows: dimension-preserving (keeps intrinsic dimensionality constant) and dimension-reducing (systematically shrinks less important dimensions for principled data compression).
Crucially, because the ODE is deterministic and invertible, local neighborhoods in the data space are preserved in the latent space. This allows the model to act as a "calibrated" bridge for Bayesian inference.
Key Insights and Results
The authors evaluated their model on a mix of synthetic toy datasets (2D and 3D distributions like Circles, Moons, and Swirls) and real-world image benchmarks (CIFAR-10 and AFHQv2).
- Deterministic Inference & Identifiability: By moving from SDEs to pfODEs, the authors transform a "noisy" generation process into a unique, bidirectional map. This makes the latent space identifiable, solving a major headache in variational inference.
- Non-linear Compression: Unlike simple Principal Component Analysis (PCA), which is a linear projection, inflationary flows use the learned "score function" to repel particles from high-density areas. This results in a nonlinear mapping that can collapse complex manifolds into a low-dimensional Gaussian.
- Superior Generative Quality: On CIFAR-10, Inflationary Flows achieved significantly better FID scores than existing injective flow models. For example, at 62 dimensions, IFs scored 23.2 compared to ~280 for the baselines.
- Extreme Compression: The model successfully generated high-quality images even when compressed to as little as 0.03% of the original nominal dimensionality (e.g., reducing 3072 dimensions down to just 1 or 2).
- Calibrated Posteriors: Using Hamiltonian Monte Carlo (HMC) on toy datasets, the authors showed that the posterior distributions inferred by the model correctly covered the ground-truth values, proving the model provides mathematically rigorous uncertainty estimates.
Limitations and Future Work
While powerful, Inflationary Flows have a few limitations. The current measure of dimensionality depends only on the covariance of the data (second-order statistics), meaning higher-order metrics might allow for even better compression of highly curved manifolds. Additionally, the "inflation" schedule requires training the model over a much larger range of noise than standard diffusion models, which can lead to longer training times. Finally, compression currently favors top principal components; future work could explore more "axis-agnostic" flows that don't rely on a pre-computed PCA.
Summary updated