Abstract
Neural Ordinary Differential Equations (Neural ODEs) model continuous dynamical systems but are inherently limited by the discrete numerical solvers used during training. This paper demonstrates that Neural ODEs do not learn the "true" underlying physics but instead converge toward an Inverse Modified Differential Equation (IMDE). This perturbed vector field accounts for systematic discretization errors, explaining why solver choice and step size are critical for model generalization and the preservation of physical laws like energy conservation.
Introduction
Neural ODEs represent a powerful paradigm in machine learning, parameterizing the derivative of a state with a neural network. However, the transition from continuous theory to discrete implementation introduces a gap. While researchers have noted that solver settings impact performance, the theoretical nature of what the network actually learns has remained elusive. This work bridges that gap by introducing the framework of Inverse Modified Differential Equations (IMDE).
The IMDE Framework
In classical numerical analysis, a Modified Differential Equation (MDE) is a continuous system whose exact solution matches the numerical solution of a given ODE. The authors invert this concept for the learning problem. They posit that a neural network $f_\theta$ trained via a discrete solver does not approximate the true vector field $f$, but rather a perturbed field $f_h$ (the IMDE). When $f_h$ is solved exactly, it yields the same result as the numerical integration of $f$.
Using Lie derivatives and Taylor expansions, the authors prove that the training process minimizes the distance between $f_\theta$ and $f_h$. The discrepancy between the learned model and the true physics is thus bounded by the solver's discretization error, typically $O(h^p)$ where $p$ is the solver's order.
Empirical Validation
The theory was tested on three benchmark systems: a pendulum, a damped harmonic oscillator, and a nonlinear Lorenz system.
As illustrated in Figure 1, the trajectories generated by the learned Neural ODE (blue dashed) align almost perfectly with the IMDE (red line), while deviating significantly from the ground-truth ODE (grey line) when coarse solvers are used. This confirms that the network actively compensates for the solver's systematic errors.
Quantitative Error Analysis
The authors further quantified the relationship between step size $h$ and learning error.
Figure 2 demonstrates that the error between the learned field $f_\theta$ and the true field $f$ scales linearly for the Euler method and quadratically for the Explicit Midpoint method. Crucially, the error between $f_\theta$ and the theoretical IMDE $f_h$ remains near zero, validating that $f_h$ is indeed the true target of the optimization process.
Implications for Physical Systems
A significant consequence of this finding is the failure of non-symplectic solvers to preserve conservation laws. For Hamiltonian systems like the pendulum, the IMDE of a non-symplectic solver is itself non-Hamiltonian. Consequently, a Neural ODE trained with such a solver will learn "artificial" dissipation or energy gain, causing phase-space trajectories to spiral rather than close, even if the training data is perfectly conservative.
Conclusion and Limitations
This work provides a rigorous foundation for understanding the interaction between numerical integration and neural network training. While it clarifies the "target shift" caused by discretization, limitations remain. The current framework assumes fixed step sizes and analytic vector fields. Future work is needed to extend these insights to adaptive-step solvers and to further explore the generalization properties of models that have "baked-in" solver errors.
Summary updated