TL;DR
Neural Ordinary Differential Equations (NODEs) are powerful but can be slow to train due to the computational cost of backpropagation via adjoint ODEs. This paper introduces a simple, 12-line code modification that uses a "seminorm" instead of a standard norm in the ODE solver's step acceptance criterion. This change significantly reduces the number of rejected steps, leading to faster training with no loss in accuracy across various applications.
What Problem Does This Paper Address?
Training Neural Ordinary Differential Equations (NODEs) involves solving a system of ODEs in the forward pass and then solving another set of ODEs, called adjoint equations, in the backward pass to compute gradients. These adjoint equations are typically solved using adaptive-step-size numerical ODE solvers. A key part of these solvers is an "accept/reject" mechanism: a proposed integration step is accepted if the estimated error is below a certain threshold, otherwise, it's rejected, the step size is reduced, and the process repeats. The authors observe that for adjoint ODEs, the standard norms (like L2) used in this mechanism are often too stringent. This leads to many unnecessary step rejections, slowing down the backpropagation process and thus the overall training time of NODEs. The problem is that some components of the adjoint equations are not true ODEs but rather integrals, meaning their "errors" don't propagate in the same way as true ODEs.
Methods
The core contribution is the introduction of a seminorm for the accept/reject step in adaptive-step-size ODE solvers when applied to adjoint equations. Instead of using a norm that equally weights all components of the adjoint state (which includes gradients with respect to the state, parameters, and time), the proposed seminorm effectively ignores or down-weights the components corresponding to parameters and time. This is justified because these components are essentially integrals, not ODEs, and their accuracy is less critical for the overall gradient computation. By using this seminorm, the solver prioritizes the accuracy of the state and its gradient with respect to the state, which are the true ODE components. This leads to fewer spurious rejections and allows for larger step sizes, speeding up the computation. The authors emphasize that this requires only a minor modification to the ODE solver's norm calculation, easily implemented in libraries like torchdiffeq.
Datasets
The paper evaluates its method on several standard datasets across different applications:
- Speech Commands dataset: Used for Neural Controlled Differential Equations (Neural CDEs).
- MNIST and CIFAR-10: Used for Continuous Normalizing Flows (CNFs).
- Acrobot (fully-actuated double pendulum) problem: Used for Hamiltonian dynamics.
Key Insights and Ideas
- Adjoint equations are not always ODEs: The crucial insight is that certain components of the adjoint equations (those related to parameter and time gradients) are actually integrals, not differential equations. This structural difference means that standard ODE solver norms are unnecessarily strict.
- Spurious rejections are costly: The accept/reject mechanism, while essential for accurate ODE solving, can become a bottleneck when applied to components that don't behave like true ODEs, leading to wasted computation.
- Seminorms offer a simple fix: By using a seminorm that focuses on the true ODE components of the adjoint state, the solver can be more efficient without sacrificing accuracy.
- Minimal code change, maximal impact: The proposed solution is remarkably simple, requiring only a few lines of code modification, yet yields significant speedups.
Key Results
The proposed seminorm approach consistently leads to substantial reductions in the number of function evaluations during the backward pass (adjoint computation):
- Median improvement: 40% fewer function evaluations.
- Maximum improvement: Up to 62% fewer function evaluations on some problems.
- Overall training time: Roughly halved in some cases.
- Accuracy: No observed degradation in model accuracy across various tasks.
- Specific examples:
- Neural CDEs: 40%-62% fewer backward steps.
- Continuous Normalizing Flows: Reduced backward NFEs, allowing larger models to be trained with similar computational cost.
- Hamiltonian Dynamics: 43% fewer backward function evaluations.
- Rejected steps: The seminorm approach drastically reduces or eliminates rejected steps, particularly for Hamiltonian dynamics where rejected steps went from ~38% to 0%.
Limitations and Future Work
- Limitations: The paper notes that the degree of improvement varies across problems, influenced by factors like tolerances, vector field complexity, and the ratio of parameters to state variables. The "rule-of-thumb" nature of the improvement is acknowledged.
- Future Work: The authors suggest exploring refinements by combining ODE solvers for the state and its gradient with quadrature rules for the parameter and time gradients. They also point out that libraries like
DifferentialEquations.jloffer similar mechanisms for implementing seminorms.
Summary updated