Abstract
Normalizing flow is a class of deep generative models for efficient sampling and density estimation. In practice, the flow often appears as a chain of invertible neural network blocks; to facilitate training, existing works have regularized flow trajectories and designed special network architectures. The current paper develops a neural ODE flow network inspired by the Jordan-Kinderleherer-Otto (JKO) scheme, which allows efficient block-wise training of the residual blocks and avoids inner loops of score matching or variational learning. As the JKO scheme unfolds the dynamic of gradient flow, the proposed model naturally stacks residual network blocks one-by-one, reducing the memory load and difficulty of performing end-to-end training of deep flow networks. We also develop adaptive time reparameterization of the flow network with a progressive refinement of the trajectory in probability space, which improves the model training efficiency and accuracy in practice. Using numerical experiments with synthetic and real data, we show that the proposed JKO-iFlow model achieves similar or better performance in generating new samples compared with existing flow and diffusion models at a significantly reduced computational and memory cost.
Introduction
Normalizing flows are a powerful class of generative models, prized for their ability to perform both efficient sampling and exact likelihood estimation. However, training them—especially Continuous Normalizing Flows (CNFs)—often involves balancing complex ODE solvers, massive memory requirements for end-to-end backpropagation, and the "non-uniqueness" of the flow, which can lead to erratic, hard-to-learn trajectories.
JKO-iFlow addresses these hurdles by borrowing a classic idea from mathematical physics: the Jordan-Kinderleherer-Otto (JKO) scheme. This framework enables the training of continuous normalizing flows through a block-wise procedure, where each residual block is trained independently to minimize a local objective. By following the natural "Wasserstein gradient flow" of the data distribution toward a Gaussian, the model achieves state-of-the-art performance with significantly lower memory and computational costs than traditional end-to-end methods.
Methodology
The core innovation is the application of the JKO scheme, which describes how a probability density evolves toward equilibrium by taking discrete "proximal steps" that minimize energy while staying close to the previous state in terms of optimal transport distance (Wasserstein-2 metric).
Key Technical Components:
- Block-wise Training: Instead of training a 100-layer network all at once, JKO-iFlow trains blocks sequentially. Each block learns a small part of the total transformation, drastically reducing the memory footprint.
- Fokker-Planck Guidance: The training objective for each block is derived from the Fokker-Planck Equation (FPE). This ensures the flow follows a unique, deterministic path—the "Wasserstein gradient flow"—which is naturally smooth and regular compared to standard CNFs.
- Adaptive Time Reparameterization: The authors developed a "progressive refinement" technique that adaptively adjusts the "time step" (the amount of transformation) for each block. This ensures the movement in probability space is even and efficient, as shown in the refinement from 3 to 6 blocks.
- Finite-Difference Divergence Estimator: To calculate the change in log-density, JKO-iFlow uses a finite-difference trick that is 1.5–2x faster than standard automatic differentiation while maintaining accuracy.
Key Insights and Results
By forcing the flow to follow the Wasserstein gradient flow, the model avoids the "non-uniqueness" problem. The path from data to Gaussian becomes a straight-ish line in probability space, making it much easier for the neural network to approximate. This greedy, layer-by-layer training is not only viable but superior in terms of resource efficiency without sacrificing generative quality.
- Efficiency: On tabular datasets, JKO-iFlow achieved competitive or better Negative Log-Likelihood (NLL) and Maximum Mean Discrepancy (MMD) scores while using 10x fewer mini-batches than baselines like FFJORD.
- Image Quality: On CIFAR-10, the model achieved a Fréchet Inception Distance (FID) of 29.10, outperforming most existing CNF baselines with much lower training time (e.g., 24 hours on a single A100 GPU).
- Invertibility: The model maintained extremely low numerical inversion errors (around $10^{-5}$ to $10^{-6}$), proving that block-wise training successfully preserves the flow's mathematical properties.
Conclusion
JKO-iFlow demonstrates that decoupling network depth from memory usage through block-wise training is a highly effective strategy for normalizing flows. By leveraging the JKO scheme to follow a natural probability trajectory, the model simplifies the training process and improves efficiency, paving the way for more scalable and stable generative modeling.
Summary updated