Abstract
We propose a simple interpolation-based method for the efficient approximation of gradients in neural ODE models. We compare it with the reverse dynamic method (known in the literature as "adjoint method") to train neural ODEs on classification, density estimation, and inference approximation tasks. We also propose a theoretical justification of our approach using logarithmic norm formalism. As a result, our method allows faster model training than the reverse dynamic method that was confirmed and validated by extensive numerical experiments for several standard benchmarks.
Introduction
Neural Ordinary Differential Equations (Neural ODEs) represent a paradigm shift in deep learning, treating network layers as a continuous system where the output is the result of solving an Initial Value Problem (IVP). Training these models requires computing gradients through an ODE solver, which traditionally involves a trade-off between memory efficiency and numerical stability.
Direct backpropagation is accurate but requires storing every intermediate step of the ODE solver, leading to high memory consumption. Conversely, the Adjoint Method (or Reverse Dynamic Method, RDM) is memory-efficient but can be numerically unstable. If the forward dynamics are complex or divergent, solving them backward to reconstruct the state can lead to significant errors in the gradients. This paper introduces the Interpolated Reverse Dynamic Method (IRDM) as a robust and efficient alternative that bridges this gap.
Methodology: Interpolated Reverse Dynamic Method
The IRDM replaces the unstable backward reconstruction of states with a highly efficient interpolation technique. Instead of solving an ODE to find previous states $z(t)$ during the backward pass, the method utilizes Barycentric Lagrange Interpolation (BLI).
Key Components
- Chebyshev Grid Sampling: During the forward pass, the model saves activations at specific time points defined by a Chebyshev grid. This non-uniform grid is denser at the boundaries, which significantly minimizes interpolation errors (Runge's phenomenon).
- Decoupling the State: By using interpolation, the backward pass no longer needs to solve for the state $z(t)$ and the adjoint variable $a(t)$ as a coupled system. This decoupling reduces the dimensionality of the "augmented" ODE system, directly decreasing the number of function evaluations (NFE) required.
- Theoretical Foundation: The paper provides a rigorous mathematical proof using logarithmic norm formalism to show that gradient errors remain bounded, ensuring stability even with the small perturbations introduced by interpolation.
Experimental Evaluation
The researchers evaluated IRDM across several standard machine learning benchmarks, including density estimation (MINIBOONE, toy 2D datasets), image classification (MNIST, CIFAR10), and Variational Autoencoders (Caltech Silhouettes, Freyfaces).
Key Results and Insights
- Efficiency and Speed: On the MINIBOONE density estimation task, IRDM converged significantly faster than the standard RDM baseline. In toy problems, IRDM required up to 10x fewer right-hand side (RHS) evaluations per iteration compared to RDM while maintaining similar accuracy.
- Classification Performance: On CIFAR10, IRDM reached higher test accuracy in less training time compared to the standard adjoint method.
- Stability: IRDM is inherently more stable than RDM because it does not rely on the reversible nature of the ODE, which can be ill-conditioned when run backward. The authors demonstrated that BLI is superior to simpler methods like piecewise-linear interpolation, which can become unstable over long training periods.
Discussion and Conclusion
IRDM exploits the inherent smoothness of Neural ODE solutions to provide a "Goldilocks" solution for gradient computation: it is more stable than the adjoint method and more memory-efficient than direct backpropagation. While the number of Chebyshev nodes ($N$) is a hyperparameter, the authors found that a modest $N$ (e.g., 8 to 16) is often sufficient to outperform existing baselines. Future work may address the memory cost for extremely high-dimensional ODE blocks and the performance on non-smooth activation functions where the analyticity assumption is challenged.
Summary updated