Stochastic continuous normalizing flows: training SDEs as ODEs

Liam Hodgkinson, Christopher van der Heide, Fred Roosta, Michael W. MahoneyUAI 2021 · OpenReview

Abstract

This paper introduces Stochastic Continuous Normalizing Flows (SCNF), a framework that bridges the gap between deterministic Neural Ordinary Differential Equations (Neural ODEs) and Stochastic Differential Equations (SDEs). By leveraging Rough Path Theory, the authors demonstrate how to treat SDEs as "random ODEs," allowing them to be trained using the efficient tools developed for Continuous Normalizing Flows. This approach enables density estimation and memory-efficient training for SDE-based models using standard, high-order ODE solvers.

Introduction

Normalizing flows are powerful tools for generative modeling and density estimation, but they are typically deterministic. Continuous Normalizing Flows (CNFs)—often called Neural ODEs—offer memory efficiency and continuous-time dynamics but lack the inherent robustness and exploratory properties of stochastic systems.

Training models based on Stochastic Differential Equations (SDEs) is historically difficult because standard stochastic calculus (Itô calculus) is "rough" and incompatible with the "adjoint method"—a technique used to calculate gradients without storing the entire forward pass in memory. Previous attempts to create "Neural SDEs" often relied on restrictive assumptions or were incompatible with high-order numerical solvers. SCNF seeks a general, mathematically rigorous framework that allows SDEs to be trained just like ODEs.

Methods: SDEs as Random ODEs

The core innovation of SCNF lies in the use of Rough Path Theory and Wong-Zakai approximations to transform stochastic dynamics into a form compatible with ODE machinery.

  1. Noise as a Latent Variable: Instead of treating Brownian motion as unpredictable "white noise," the authors treat it as a latent path. This rough path is approximated with a sequence of smooth, differentiable paths, such as piecewise linear interpolation or Karhunen-Loève expansions.
  2. The Stratonovich Correction: To ensure mathematical consistency when moving from "rough" to "smooth" paths, standard Itô SDEs are converted into Stratonovich SDEs by adding a specific correction term to the drift.
  3. Density Evolution: Once the noise path is smoothed, the SDE effectively becomes a "random ODE." This allows the application of the Liouville equation to track the evolution of probability density over time.
  4. The Adjoint Method: Because the system is treated as an ODE, the standard adjoint method can be used for backpropagation. This enables memory-efficient training even with complex, adaptive numerical solvers like Dormand-Prince.
Approximation of Brownian motion using smooth (left) and piecewise linear (right) paths.

Key Results and Insights

The authors demonstrate the theoretical and practical validity of SCNFs using synthetic 2D densities, such as banana-shaped and star-shaped distributions.

Theoretical Convergence

The paper provides a formal proof showing that as the noise approximation becomes more refined, the log-densities and gradients of the "random ODE" converge to the true values of the underlying SDE. This justifies the use of smooth approximations for training.

Trainable Diffusion

In experiments with a banana-shaped density, the authors compared models with fixed versus trainable diffusion coefficients ($\sigma$). A trainable $\sigma$ allows the model to adapt its noise levels to the underlying curvature of the data, resulting in a more effective sampler.

Comparison of fixed vs. variable diffusion coefficients ($\sigma$) on a banana-shaped density.

Stochasticity as Regularization

The diffusion term acts as a natural smoother. By increasing noise intensity, the model gains the capacity to "smooth out" minor, potentially unwanted details in the data distribution, which can improve overall robustness.

Discussion and Limitations

SCNF provides a unified framework where existing Neural ODE machinery carries over to stochastic systems. Unlike previous stochastic adjoint methods, this approach is "solver-agnostic," working with sophisticated adaptive solvers that are more stable than simple Euler-Maruyama schemes.

However, several challenges remain. The density estimator relies on Monte Carlo sampling over multiple noise paths, which can lead to high gradient variance in high-dimensional spaces. Additionally, there is a tendency for the model to shrink the diffusion coefficient to zero during training, causing the SCNF to collapse into a deterministic CNF. While theoretically robust, scaling these stochastic flows to large-scale datasets remains a key area for future work.

Summary updated

BibTeX
@inproceedings{hodgkinson2021stochastic,
  author = {Liam Hodgkinson and Christopher van der Heide and Fred Roosta and Michael W. Mahoney},
  title = {{Stochastic continuous normalizing flows: training SDEs as ODEs}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=HalpPKCbwf},
}
Figure