Neural ODE Control for Trajectory Approximation of Continuity Equation

Karthik Elamvazhuthi, Bahman Gharesifard, Andrea L. Bertozzi, Stanley J. Osher

Abstract

We consider the controllability problem for the continuity equation, corresponding to neural ordinary differential equations (ODEs), which describes how a probability measure is pushedforward by the flow. We show that the controlled continuity equation has very strong controllability properties. Particularly, a given solution of the continuity equation corresponding to a bounded Lipschitz vector field defines a trajectory on the set of probability measures. For this trajectory, we show that there exist piecewise constant training weights for a neural ODE such that the solution of the continuity equation corresponding to the neural ODE is arbitrarily close to it. As a corollary to this result, we establish that the continuity equation of the neural ODE is approximately controllable on the set of compactly supported probability measures that are absolutely continuous with respect to the Lebesgue measure.

Introduction

Neural Ordinary Differential Equations (Neural ODEs) have emerged as a powerful framework for modeling continuous-time dynamics in machine learning. By parameterizing the derivative of the state with a neural network, these models allow for the transformation of data through a continuous flow. In applications such as Normalizing Flows and density estimation, Neural ODEs are used to evolve an initial probability distribution into a target distribution.

While the expressive power of standard feed-forward networks is well-understood via the Universal Approximation Theorem, the trajectory-level expressive power of Neural ODEs—specifically their ability to steer probability measures along arbitrary paths—remains a critical theoretical question. This paper addresses this by framing the evolution of distributions as a control problem governed by the continuity equation.

Problem Formulation

The evolution of a probability measure $\mu_t$ under a flow generated by a vector field $v_t$ is described by the continuity equation: $$\partial_t \mu_t + \text{div}(v_t \mu_t) = 0$$ In a Neural ODE, the vector field $v_t$ is restricted to the class of functions representable by a neural network architecture, where the weights and biases act as time-varying control inputs. The fundamental question is whether these limited "control knobs" are sufficient to approximate any trajectory of measures generated by a general bounded Lipschitz vector field. This property is known as approximate controllability.

Methodology

The authors utilize a control-theoretic approach to prove that Neural ODEs are universal trajectory approximators. The proof structure relies on three primary pillars:

1. Time-Slicing and Piecewise Approximation

The authors demonstrate that any smooth, time-varying vector field can be discretized into a sequence of piecewise constant vector fields. By dividing the time interval $[0, T]$ into sufficiently small segments, the complex evolution of the distribution can be treated as a series of simpler transformations.

2. Spatial Approximation via Universal Approximation

At any fixed time step, the vector field required to move the distribution is a spatial function. The authors leverage the Universal Approximation Theorem for single-hidden-layer networks to show that a neural network with appropriate weights $A$, internal parameters $W$, and biases $\theta$ can approximate these required spatial vector fields to any desired precision.

3. Weak Convergence and Oscillatory Control

The most technical contribution involves the use of weak convergence. In scenarios where a neural network of limited width cannot perfectly match a complex target vector field, the authors prove that perfect instantaneous matching is unnecessary. By employing rapidly oscillating control inputs that converge "weakly" to the target vector field over time, the resulting trajectory of the probability distribution in the Wasserstein-2 space still converges uniformly to the desired path. This suggests that the temporal depth of the ODE can compensate for constraints in network width.

Key Results

Theorem IV.1: Universal Trajectory Approximation

The main result establishes that for any trajectory of probability measures $\mu_t$ generated by a bounded Lipschitz vector field, and for any $\epsilon > 0$, there exists a Neural ODE with piecewise constant weights such that its solution $\mu^\epsilon_t$ satisfies: $$\sup_{t \in [0,T]} W_2(\mu^\epsilon_t, \mu_t) \leq \epsilon$$ This confirms that Neural ODEs can stay arbitrarily close to any "well-behaved" path of a probability distribution.

Corollary IV.2: Approximate Controllability

As a consequence of the main theorem, the authors establish that the continuity equation of the Neural ODE is approximately controllable on the set of absolutely continuous probability measures with compact support. This means one can steer an initial distribution to virtually any target distribution by following a specific trajectory.

Discussion and Implications

This work provides a rigorous mathematical foundation for the empirical success of Neural ODEs in generative modeling. By connecting deep learning to Optimal Transport and the geometry of the Wasserstein space, the results justify the use of continuous-depth models for complex density transformation tasks.

A key insight is the trade-off between width and depth: the ability to change weights over time (depth) allows even relatively narrow networks to achieve high-fidelity control over the flow of probability. This has significant implications for the design of memory-efficient generative models.

Limitations and Future Work

While the paper provides a robust proof for Lipschitz vector fields, several avenues for future research remain:

  • Relaxing Lipschitz Constraints: Ideal vector fields in certain optimal transport problems may not be uniformly Lipschitz; extending the proof to broader classes of vector fields is a logical next step.
  • Stochastic Dynamics: The current framework is deterministic. Extending these controllability results to Neural SDEs (Stochastic Differential Equations) would better account for noise in real-world data.
  • Higher-Order Systems: Investigating second-order dynamics (modeling acceleration) could offer more flexibility for approximating trajectories with physical constraints.

Summary updated

Abstract

We consider the controllability problem for the continuity equation, corresponding to neural ordinary differential equations (ODEs), which describes how a probability measure is pushedforward by the flow. We show that the controlled continuity equation has very strong controllability properties. Particularly, a given solution of the continuity equation corresponding to a bounded Lipschitz vector field defines a trajectory on the set of probability measures. For this trajectory, we show that there exist piecewise constant training weights for a neural ODE such that the solution of the continuity equation corresponding to the neural ODE is arbitrarily close to it. As a corollary to this result, we establish that the continuity equation of the neural ODE is approximately controllable on the set of compactly supported probability measures that are absolutely continuous with respect to the Lebesgue measure.
BibTeX
@inproceedings{elamvazhuthi2022neural,
  author = {Karthik Elamvazhuthi and Bahman Gharesifard and Andrea L. Bertozzi and Stanley J. Osher},
  title = {{Neural ODE Control for Trajectory Approximation of Continuity Equation}},
  booktitle = {IEEE Control. Syst. Lett},
  year = {2022},
  url = {https://openreview.net/forum?id=dl9XTxqFgl},
}
Figure