Building Normalizing Flows with Stochastic Interpolants

Michael Samuel Albergo, Eric Vanden-Eijnden

Generative modeling has largely been dominated by two powerful paradigms: Normalizing Flows and Diffusion Models. Normalizing flows use Ordinary Differential Equations (ODEs) to define a clean, deterministic mapping between a simple base distribution (like Gaussian noise) and a complex target distribution (like natural images). However, training them typically requires Maximum Likelihood Estimation (MLE), which involves computationally expensive and unstable backpropagation through ODE solvers.

On the other hand, Diffusion Models use Stochastic Differential Equations (SDEs) and score matching to achieve simulation-free training and high-quality samples. Yet, they are mathematically complex, theoretically require infinite time to converge, and lack the straightforward deterministic mapping of flows.

This paper introduces InterFlow, a novel framework that bridges this gap. By leveraging "stochastic interpolants," InterFlow retains the deterministic, invertible nature of normalizing flows while adopting the highly efficient, simulation-free training style of diffusion models.

The Core Idea: Stochastic Interpolants

Instead of attempting to learn the complex mapping directly via MLE, the authors propose defining a time-dependent path—an interpolant $I_t(x_0, x_1)$—that explicitly connects a sample from the base distribution ($x_0$) to a sample from the target ($x_1$) over a finite time interval $t \in [0, 1]$.

Density evolution over time from base to target distribution

By decoupling the path from the training objective, the user can choose the interpolant, and the model simply learns the velocity field required to follow it. The authors prove mathematically that the velocity field of this flow is the unique minimizer of a simple quadratic loss function. Because this loss relies only on expectations over the interpolant, the model can be trained without ever solving an ODE during the training loop.

Why InterFlow Stands Out

The InterFlow framework introduces several key technical advantages that solve long-standing bottlenecks in generative modeling:

  • Simulation-Free Training: Because the quadratic loss avoids ODE solvers during training, the computational cost per iteration is constant. This leads to massive speedups compared to traditional continuous normalizing flows.
  • Finite Time Convergence: Unlike standard diffusion models that theoretically require infinite time to reach the target distribution, InterFlow connects the distributions exactly within the finite interval $t \in [0, 1]$.
  • Optimal Transport: The framework allows for the interpolant itself to be optimized. By maximizing the objective over the choice of path, the model can discover "optimal transport" maps—the most efficient, straight-line paths between distributions.
  • Duality with Diffusion: If the base distribution is Gaussian, the learned velocity field can be directly converted into a "score" function for a diffusion model, elegantly bridging the ODE and SDE perspectives.
Comparison of generative modeling frameworks

Key Results and Performance

The authors evaluated InterFlow across 2D toy datasets, high-dimensional tabular data, and image generation benchmarks, demonstrating significant improvements in both speed and scalability:

  • Massive Speedups: On the MiniBooNE tabular dataset, InterFlow achieved a staggering 400x speedup in training time compared to the state-of-the-art FFJORD model, while simultaneously reaching a better log-likelihood.
  • Superior Density Estimation: On standard tabular benchmarks, InterFlow matched or surpassed conventional continuous flows. For instance, on the GAS dataset, it achieved a negative log-likelihood (NLL) of -12.35, easily beating FFJORD's -8.59.
  • Scaling to High Resolutions: Historically, ODE-based flows have struggled to scale due to the memory demands of the adjoint method (backpropagating through solvers). By bypassing this entirely, InterFlow successfully generated $128\times128$ images of flowers, proving that ODE flows can reach previously unattainable resolutions.
  • Competitive Image Quality: On CIFAR-10, InterFlow achieved an NLL of 2.99 bits/dim, making it highly competitive with top-tier diffusion models (like ScoreSDE) and superior to many previous flow models.

Limitations and Future Directions

While InterFlow represents a major step forward, some open questions remain. Most experiments in the paper utilize a fixed linear or trigonometric path; finding the absolute "best" class of interpolants for specific data modalities is an area for future research. Additionally, while training is simulation-free and incredibly fast, sampling still requires solving an ODE at inference time, which can be slower than single-step models like GANs.

Finally, the authors note that while their ODE approach is simpler and highly effective, the exact role and necessity of stochasticity (noise) in high-dimensional image generation is still being explored. Future work will likely investigate hybrid models that use the stochastic interpolant framework to blend the best aspects of SDEs and ODEs even further.

Summary updated

Abstract

A generative model based on a continuous-time normalizing flow between any pair of base and target probability densities is proposed. The velocity field of this flow is inferred from the probability current of a time-dependent density that interpolates between the base and the target in finite time. Unlike conventional normalizing flow inference methods based the maximum likelihood principle, which require costly backpropagation through ODE solvers, our interpolant approach leads to a simple quadratic loss for the velocity itself which is expressed in terms of expectations that are readily amenable to empirical estimation. The flow can be used to generate samples from either the base or target, and to estimate the likelihood at any time along the interpolant. In addition, the flow can be optimized to minimize the path length of the interpolant density, thereby paving the way for building optimal transport maps. In situations where the base is a Gaussian density, we also show that the velocity of our normalizing flow can also be used to construct a diffusion model to sample the target as well as estimate its score. However, our approach shows that we can bypass this diffusion completely and work at the level of the probability flow with greater simplicity, opening an avenue for methods based solely on ordinary differential equations as an alternative to those based on stochastic differential equations. Benchmarking on density estimation tasks illustrates that the learned flow can match and surpass conventional continuous flows at a fraction of the cost, and compares well with diffusions on image generation on CIFAR-10 and ImageNet $32 \times 32$. The method scales ab-initio ODE flows to previously unreachable image resolutions, demonstrated up to $128\times128$.
BibTeX
@inproceedings{albergo2023building,
  author = {Michael Samuel Albergo and Eric Vanden-Eijnden},
  title = {{Building Normalizing Flows with Stochastic Interpolants}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=li7qeBbCR1t},
}
Figure