TL;DR
Researchers have discovered that the "adjoint equations" used to train Neural Ordinary Differential Equations (ODEs) aren't entirely ODEs—some parts are just simple integrals. By using a seminorm that ignores these integral components during error estimation, they reduced the number of function evaluations by 40–62%, effectively doubling training speed without sacrificing model accuracy.
What Problem Does This Paper Address?
Neural ODEs are a powerful class of models that define a hidden state as a continuous flow. To train them, we need to calculate gradients, typically using the adjoint method. This involves solving a second differential equation backward in time.
Most modern software uses adaptive-step-size solvers. These solvers propose a step, estimate the numerical error, and compare it to a tolerance. If the error is too high (relative to a chosen norm, like $L^2$), the step is rejected and retried with a smaller increment.
The authors identified a major inefficiency: the standard approach treats every component of the adjoint system as a full-blown ODE. In reality, many of these components (specifically those related to network parameters) are just integrals. Because the solver tries to be unnecessarily precise with these "integral-only" channels, it triggers spurious rejections—wasting time on tiny steps that don't actually improve the overall accuracy of the gradient.
Methods: The Adjoint Seminorm
The core innovation is the Adjoint Seminorm.
Standard solvers use a norm that weights all channels equally when deciding whether to accept a step. The authors propose replacing this with a seminorm that assigns zero weight to the channels corresponding to parameters ($\theta$) and time ($t$).
By doing this, the adaptive solver chooses its step sizes based solely on the "true" ODE components (the state $z$ and the adjoint state $a_z$). The integral components ($a_\theta$ and $a_t$) are still computed, but they no longer have the "veto power" to reject a step. This is a minor code modification—now officially supported in the torchdiffeq library—that requires no hyperparameter tuning.
Key Insights
- The "Not an ODE" Realization: In the adjoint system, the gradients for the parameters ($a_\theta$) do not appear in the vector fields of the other variables. This means $a_\theta$ is a "passive" observer; errors in $a_\theta$ don't propagate back into the rest of the system.
- Spurious Rejections: The paper demonstrates that solvers often reject steps near the end of a trajectory because the parameter gradients are small, making the relative error ratio explode. These rejections are "spurious" because they don't actually help the solver find a better solution for the state variables.
- The Parameter-State Ratio: The authors found that the speedup is most dramatic when the number of parameters ($p$) is much larger than the state dimension ($d$). In models where $p \gg d$, the seminorm prevents thousands of parameter channels from bottlenecking the solver.
Key Results
The authors evaluated their method across three distinct applications: Time Series Modeling (Speech Commands), Generative Modeling (MNIST/CIFAR-10), and Physical Modeling (Hamiltonian dynamics).
- Significant Speedups: Across all tasks, the method achieved a median improvement of 40% fewer function evaluations (NFE).
- Peak Performance: On some problems, the NFE reduction reached 62%. Since the forward pass remains unchanged, this effectively cuts total training time in half.
- No Accuracy Trade-off: In every experiment, the models trained with seminorms reached the same level of accuracy (or bits-per-dimension for generative models) as those trained with standard norms.
- Efficiency Gains: For Continuous Normalizing Flows, the authors showed they could train a much larger model (128 hidden channels) using the same computational budget previously required for a smaller model (32 hidden channels), leading to better performance for the same cost.
Limitations and Future Work
While the seminorm approach is highly effective, the authors note that the integral components ($a_\theta, a_t$) are still being estimated using the points chosen to solve the $z$ and $a_z$ ODEs.
A promising direction for future work is to decouple these entirely: using a standard ODE solver for the "true" ODE components and specialized quadrature rules (which are more efficient for simple integrals) for the parameter gradients. This could potentially lead to even greater efficiency gains.
Summary updated