Abstract
This note revisits the flow approximation properties of narrow Neural Ordinary Differential Equations (NODEs), where the hidden layer width is constrained to the input dimension. While previous literature establishes these properties using complex geometric control theory and Lie bracket constructions, we provide a simplified, quantitative proof using fundamental ODE techniques and Grönwall's lemma. We derive an explicit bound on the number of time-dependent weight switches required for a narrow NODE to mimic the behavior of a wide, single-layer NODE. This approach clarifies how depth (time) can effectively substitute for width in neural dynamical systems, providing a more accessible theoretical framework for understanding NODE expressivity.
Introduction
Neural Ordinary Differential Equations (NODEs) represent a paradigm shift in deep learning, modeling the transformation of data as a continuous flow governed by a differential equation. A central question in the study of NODEs is their expressivity: what class of functions or flows can a given architecture represent?
Standard universal approximation theorems for NODEs typically rely on "wide" architectures, where the hidden layer width significantly exceeds the input dimension. However, recent research has shown that "narrow" NODEs—those where the width matches the input dimension—are also surprisingly powerful. The existing proofs for this capability are often "black boxes" for the general machine learning community, frequently relying on deep results from control theory, such as controllability on groups of diffeomorphisms. This paper provides a more intuitive, quantitative derivation of these properties using standard tools from mathematical analysis.
The Approximation Mechanism: Time-Multiplexing Width
The core insight behind the expressivity of narrow NODEs is the trade-off between width and time (depth). A wide NODE typically defines its velocity field as a sum of multiple neural network components:
$$\dot{x} = \sum_{i=1}^m f_i(x)$$
A narrow NODE, by contrast, is restricted to using only one of these components at any given moment. However, if the narrow NODE can switch its parameters rapidly, it can "time-multiplex" these components. By cycling through $f_1, f_2, \dots, f_m$ over very small time intervals, the system's trajectory effectively averages these individual fields, allowing the narrow system to mimic the aggregate behavior of the wide system.
Quantitative Analysis via Grönwall’s Lemma
To provide a rigorous bound on this approximation, we avoid complex Lie-theoretic arguments and instead utilize Grönwall’s lemma, a fundamental tool in ODE analysis used to bound the difference between two solutions. We consider a target trajectory $y(t)$ from a wide NODE and an approximating trajectory $z(t)$ from a narrow NODE that switches between components $N$ times over a total time $T$.
The primary result of this analysis is the derivation of an explicit error bound (Lemma 4):
$$|z(t) - y(t)| \leq \left( \frac{2T}{N}X + \tilde{K}X \frac{T^2}{2N} \right) e^{\tilde{K}T}$$
In this expression, $X$ and $\tilde{K}$ are constants determined by the Lipschitz continuity of the activation functions and the magnitude of the weights. This bound demonstrates that the approximation error is $O(1/N)$. Consequently, as the switching frequency $N$ increases ($N \to \infty$), the narrow NODE's trajectory converges perfectly to the flow of the wide NODE.
Key Insights and Implications
This simplified proof offers several key contributions to the understanding of neural dynamics:
- Demystifying Expressivity: By using standard tools like Grönwall’s lemma, the proof makes the universal approximation properties of narrow NODEs accessible to a broader audience of researchers who may not be familiar with control theory.
- Zeroth-Order Sufficiency: The author demonstrates that the approximation does not strictly require the exploitation of non-commutative vector fields or complex Lie brackets. Simple averaging—a zeroth-order effect—is sufficient to achieve the desired expressivity.
- Theoretical Bridge: This work links narrow NODEs directly to the well-understood approximation properties of shallow, wide neural networks, providing a clearer path for future theoretical exploration.
Limitations and Future Work
While theoretically robust, several practical challenges remain. The current proof relies on the weights being piecewise constant (abruptly switching), whereas practical machine learning models typically utilize continuous weight functions. Furthermore, while the paper proves that such narrow NODEs exist, it does not address the difficulty of training them. The rapid parameter switching required for close approximation likely results in a highly non-convex loss landscape, which may be difficult for standard gradient-based optimizers to navigate effectively.
Conclusion
We have provided a simplified, quantitative proof that narrow NODEs can approximate the flows of wide NODEs through rapid time-switching of weights. By establishing an $O(1/N)$ error bound, we clarify the relationship between switching frequency and approximation accuracy, offering a more intuitive understanding of how depth and time can compensate for width in constrained neural dynamical systems.
Summary updated