Quantitative Flow Approximation Properties of Narrow Neural ODEs

Karthik Elamvazhuthi

Abstract

This note revisits the flow approximation properties of narrow Neural Ordinary Differential Equations (NODEs), where the hidden layer width is constrained to the input dimension. While previous literature establishes these properties using complex geometric control theory and Lie bracket constructions, we provide a simplified, quantitative proof using fundamental ODE techniques and Grönwall's lemma. We derive an explicit bound on the number of time-dependent weight switches required for a narrow NODE to mimic the behavior of a wide, single-layer NODE. This approach clarifies how depth (time) can effectively substitute for width in neural dynamical systems, providing a more accessible theoretical framework for understanding NODE expressivity.

Introduction

Neural Ordinary Differential Equations (NODEs) represent a paradigm shift in deep learning, modeling the transformation of data as a continuous flow governed by a differential equation. A central question in the study of NODEs is their expressivity: what class of functions or flows can a given architecture represent?

Standard universal approximation theorems for NODEs typically rely on "wide" architectures, where the hidden layer width significantly exceeds the input dimension. However, recent research has shown that "narrow" NODEs—those where the width matches the input dimension—are also surprisingly powerful. The existing proofs for this capability are often "black boxes" for the general machine learning community, frequently relying on deep results from control theory, such as controllability on groups of diffeomorphisms. This paper provides a more intuitive, quantitative derivation of these properties using standard tools from mathematical analysis.

The Approximation Mechanism: Time-Multiplexing Width

The core insight behind the expressivity of narrow NODEs is the trade-off between width and time (depth). A wide NODE typically defines its velocity field as a sum of multiple neural network components:

$$\dot{x} = \sum_{i=1}^m f_i(x)$$

A narrow NODE, by contrast, is restricted to using only one of these components at any given moment. However, if the narrow NODE can switch its parameters rapidly, it can "time-multiplex" these components. By cycling through $f_1, f_2, \dots, f_m$ over very small time intervals, the system's trajectory effectively averages these individual fields, allowing the narrow system to mimic the aggregate behavior of the wide system.

Quantitative Analysis via Grönwall’s Lemma

To provide a rigorous bound on this approximation, we avoid complex Lie-theoretic arguments and instead utilize Grönwall’s lemma, a fundamental tool in ODE analysis used to bound the difference between two solutions. We consider a target trajectory $y(t)$ from a wide NODE and an approximating trajectory $z(t)$ from a narrow NODE that switches between components $N$ times over a total time $T$.

The primary result of this analysis is the derivation of an explicit error bound (Lemma 4):

$$|z(t) - y(t)| \leq \left( \frac{2T}{N}X + \tilde{K}X \frac{T^2}{2N} \right) e^{\tilde{K}T}$$

In this expression, $X$ and $\tilde{K}$ are constants determined by the Lipschitz continuity of the activation functions and the magnitude of the weights. This bound demonstrates that the approximation error is $O(1/N)$. Consequently, as the switching frequency $N$ increases ($N \to \infty$), the narrow NODE's trajectory converges perfectly to the flow of the wide NODE.

Key Insights and Implications

This simplified proof offers several key contributions to the understanding of neural dynamics:

  1. Demystifying Expressivity: By using standard tools like Grönwall’s lemma, the proof makes the universal approximation properties of narrow NODEs accessible to a broader audience of researchers who may not be familiar with control theory.
  2. Zeroth-Order Sufficiency: The author demonstrates that the approximation does not strictly require the exploitation of non-commutative vector fields or complex Lie brackets. Simple averaging—a zeroth-order effect—is sufficient to achieve the desired expressivity.
  3. Theoretical Bridge: This work links narrow NODEs directly to the well-understood approximation properties of shallow, wide neural networks, providing a clearer path for future theoretical exploration.

Limitations and Future Work

While theoretically robust, several practical challenges remain. The current proof relies on the weights being piecewise constant (abruptly switching), whereas practical machine learning models typically utilize continuous weight functions. Furthermore, while the paper proves that such narrow NODEs exist, it does not address the difficulty of training them. The rapid parameter switching required for close approximation likely results in a highly non-convex loss landscape, which may be difficult for standard gradient-based optimizers to navigate effectively.

Conclusion

We have provided a simplified, quantitative proof that narrow NODEs can approximate the flows of wide NODEs through rapid time-switching of weights. By establishing an $O(1/N)$ error bound, we clarify the relationship between switching frequency and approximation accuracy, offering a more intuitive understanding of how depth and time can compensate for width in constrained neural dynamical systems.

Summary updated

Abstract

In this note, we revisit the problem of flow approximation properties of neural ordinary differential equations (NODEs). The approximation properties have been considered as a flow controllability problem in recent literature. The neural ODE is considered {\it narrow} when the parameters have dimension equal to the input of the neural network, and hence have limited width. We derive the relation of narrow NODEs in approximating flows of shallow but wide NODEs. Due to existing results on approximation properties of shallow neural networks, this facilitates understanding which kind of flows of dynamical systems can be approximated using narrow neural ODEs. While approximation properties of narrow NODEs have been established in literature, the proofs often involve extensive constructions or require invoking deep controllability theorems from control theory. In this paper, we provide a simpler proof technique that involves only ideas from ODEs and Gr{ö}nwall's lemma. Moreover, we provide an estimate on the number of switches needed for the time dependent weights of the narrow NODE to mimic the behavior of a NODE with a single layer wide neural network as the velocity field.
BibTeX
@misc{elamvazhuthi2025quantitative,
  author = {Karthik Elamvazhuthi},
  title = {{Quantitative Flow Approximation Properties of Narrow Neural ODEs}},
  year = {2025},
  eprint = {2503.04068},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.04068},
}
Figure