Abstract: Neural ordinary differential equations (NODE) have been proposed as a continuous depth generalization to popular deep learning models such as Residual networks (ResNets). They provide parameter efficiency and automate the model selection process in deep learning models to some extent. However, they lack the much-required uncertainty modelling and robustness capabilities which are crucial for their use in several real-world applications such as autonomous driving and healthcare. We propose a novel and unique approach to model uncertainty in NODE by considering a distribution over the end-time $T$ of the ODE solver. The proposed approach, latent time NODE (LT-NODE), treats $T$ as a latent variable and apply Bayesian learning to obtain a posterior distribution over $T$ from the data. In particular, we use variational inference to learn an approximate posterior and the model parameters. Prediction is done by considering the NODE representations from different samples of the posterior and can be done efficiently using a single forward pass. As $T$ implicitly defines the depth of a NODE, posterior distribution over $T$ would also help in model selection in NODE. We also propose, adaptive latent time NODE (ALT-NODE), which allow each data point to have a distinct posterior distribution over end-times. ALT-NODE uses amortized variational inference to learn an approximate posterior using inference networks. We demonstrate the effectiveness of the proposed approaches in modelling uncertainty and robustness through experiments on synthetic and several real-world image classification data.
Introduction
Neural Ordinary Differential Equations (NODEs) offer a highly parameter-efficient, continuous-depth alternative to standard Residual Networks (ResNets). By treating the hidden layers of a neural network as a continuous differential equation, NODEs effectively create a network with "infinite" layers determined dynamically by an ODE solver. However, standard NODEs struggle with adversarial robustness and uncertainty estimation, limiting their use in high-stakes environments like medical imaging or autonomous driving. This research introduces architectural and numerical improvements to bridge this gap.
Methodology
The proposed framework addresses robustness through two distinct angles: architectural uncertainty modelling and numerical solver stability.
NODE-GP and Latent Time Modelling
To provide reliable uncertainty estimates, the authors introduce NODE-GP, which replaces the standard fully connected classification layer with a Gaussian Process (GP). Because GPs are non-parametric and Bayesian, they output a distribution of predictions rather than a single point estimate. Furthermore, the researchers propose modelling the end-time $T$ of the ODE solver as a latent variable (LT-NODE and ALT-NODE). Since $T$ implicitly defines the depth of the network, learning a posterior distribution over $T$ allows the model to adaptively determine its own depth and quantify its confidence.
The Role of Numerical Solvers
A key insight of this work is that the choice of numerical solver dictates adversarial robustness. Simple first-order solvers, like the Euler method, suffer from approximation drift that increases over time. This drift creates messy feature spaces that are easily exploited by adversarial attacks.
In contrast, higher-order solvers like the 4th-order Runge-Kutta (RK4) create tight, distinct class clusters. To balance computational cost with robustness, the authors propose a mixed-order approach, utilizing adaptive solvers for initial blocks and RK4 for subsequent transformations.
Key Results
Adversarial Defense
The combination of stable solvers and Bayesian layers yields a compounding effect on robustness. Under strong Projective Gradient Descent (PGD) attacks, the proposed NODE_Adap-NODE_NT2-GP model demonstrated exceptional resilience.
At noise intensities of $\epsilon \ge 0.05$, the proposed model was 52% more accurate than the baseline standard NODE and 30% more accurate than a NODE using only an adaptive solver without the GP layer.
Uncertainty and Efficiency
When tested on out-of-distribution (OOD) data (CIFAR-100 for a model trained on CIFAR-10), the NODE-GP model successfully recognized its own ignorance. It achieved an Entropy score of 1.51, compared to a mere 0.137 for the overconfident baseline. Remarkably, these gains in robustness and uncertainty awareness were achieved with significantly fewer parameters: 13.2 million for NODE-GP versus 23.4 million for an equivalent ResNet34-GP.
Conclusion
By integrating Gaussian Processes and higher-order numerical solvers, this research transforms Neural ODEs into robust, uncertainty-aware models. The resulting architecture provides a parameter-efficient solution for complex classification tasks where knowing "when the model doesn't know" is just as important as the prediction itself.
Summary updated