Abstract
Generative modeling has increasingly shifted toward continuous-time methods, such as diffusion models and continuous normalizing flows (CNFs). However, CNFs traditionally suffer from significant computational overhead due to the need for backpropagation through ODE solvers during training. This paper introduces a framework using stochastic interpolants to transform CNF training into a simulation-free regression problem. This approach decouples path selection from optimization, enabling scaling to high-resolution images and providing a theoretical bridge between ODE-based flows and SDE-based diffusion models.
Introduction: The Training Bottleneck in CNFs
The fundamental goal of generative modeling is to map a simple base distribution $\rho_0$ (e.g., Gaussian) to a complex target distribution $\rho_1$ (e.g., natural images). Continuous Normalizing Flows (CNFs) achieve this by learning a velocity field that transports samples along a path over time.
Historically, CNFs have been trained using the Maximum Likelihood Principle (MLE). This requires solving an Ordinary Differential Equation (ODE) and backpropagating through the solver to compute likelihood gradients. This process is computationally expensive, memory-intensive, and often unstable in high-dimensional spaces, limiting CNFs to low-resolution data. While diffusion models address some efficiency issues using Stochastic Differential Equations (SDEs), they often require infinite time to converge. The proposed framework aims to combine the deterministic, invertible nature of flows with the training efficiency of score-based methods.
Methods: InterFlow and Stochastic Interpolants
The authors introduce InterFlow, a framework built on stochastic interpolants. This method bypasses the ODE-backprop bottleneck by redefining the training objective.
1. The Interpolant
Instead of the model discovering a path through MLE, the user defines a stochastic process $x_t = I_t(x_0, x_1)$ that explicitly connects samples from $\rho_0$ and $\rho_1$. A common choice is the trigonometric interpolant: $x_t = \cos(\frac{1}{2}\pi t)x_0 + \sin(\frac{1}{2}\pi t)x_1$.
2. Simulation-Free Training
The velocity field $v_t$ generating this process is shown to be the unique minimizer of a quadratic loss function: $$G(\hat{v}) = \mathbb{E} [|\hat{v}_t(I_t)|^2 - 2\partial_t I_t \cdot \hat{v}_t(I_t)]$$ This objective is "simulation-free," requiring only simple regression over sampled pairs $(x_0, x_1)$ and time $t$, without solving any ODEs during training.
3. Comparison with Existing Methods
InterFlow occupies a unique position in the landscape of generative models, offering simulation-free training while maintaining computable likelihoods and finite-time integration.
Results and Benchmarks
The framework was evaluated across various scales, from toy tasks to high-resolution image synthesis.
2D Toy Tasks and Tabular Data
On 2D datasets like anisotropic swirls and checkerboards, InterFlow demonstrated precise mode capture and density estimation. In tabular benchmarks (e.g., POWER, MINIBOONE), it matched or exceeded state-of-the-art CNF performance while achieving up to a 400x speedup in training time per iteration compared to MLE-based methods like FFJORD.
Image Synthesis
InterFlow successfully scaled to image datasets including CIFAR-10 and ImageNet ($32 \times 32$), producing competitive FID and NLL scores. Notably, it enabled the generation of $128 \times 128$ resolution images (Oxford Flowers), a task previously considered computationally prohibitive for standard ODE-based flows.
Discussion and Future Directions
By turning flow training into a regression task, this work decouples path selection from optimization. This allows for the injection of inductive biases—such as preferring straight-line paths—which can simplify the learned dynamics.
However, challenges remain. While training is efficient, sampling still requires an ODE solver, the speed of which depends on the complexity of the learned velocity field. Additionally, the choice of interpolant $I_t$ introduces a new hyperparameter. Future research may focus on "rectifying" paths to ensure they are as straight as possible, further accelerating sampling by reducing the number of required ODE solver steps.
Summary updated