Abstract
Neural ordinary differential equations (NODE) have been proposed as a continuous depth generalization to popular deep learning models such as Residual networks (ResNets). They provide parameter efficiency and automate the model selection process in deep learning models to some extent. However, they lack the much-required uncertainty modelling and robustness capabilities which are crucial for their use in several real-world applications such as autonomous driving and healthcare. We propose a novel and unique approach to model uncertainty in NODE by considering a distribution over the end-time $T$ of the ODE solver. The proposed approach, latent time NODE (LT-NODE), treats $T$ as a latent variable and apply Bayesian learning to obtain a posterior distribution over $T$ from the data. In particular, we use variational inference to learn an approximate posterior and the model parameters. Prediction is done by considering the NODE representations from different samples of the posterior and can be done efficiently using a single forward pass. As $T$ implicitly defines the depth of a NODE, posterior distribution over $T$ would also help in model selection in NODE. We also propose, adaptive latent time NODE (ALT-NODE), which allow each data point to have a distinct posterior distribution over end-times. ALT-NODE uses amortized variational inference to learn an approximate posterior using inference networks. We demonstrate the effectiveness of the proposed approaches in modelling uncertainty and robustness through experiments on synthetic and several real-world image classification data.
Introduction
Neural Ordinary Differential Equations (NODEs) represent a paradigm shift in deep learning, modeling neural networks as continuous-depth systems. By defining the derivative of the hidden state with respect to "time" (depth), NODEs offer parameter efficiency and a natural generalization of Residual Networks (ResNets). However, standard NODEs face significant challenges in high-stakes environments like autonomous driving or healthcare. Specifically, they lack robust uncertainty quantification, often providing overconfident point estimates. Furthermore, the "depth" of a NODE is determined by a fixed integration end-time $T$, which typically requires manual, computationally expensive tuning.
Methodology: Latent Time NODEs
The core innovation of this work is treating the integration end-time $T$ as a latent variable rather than a fixed hyperparameter. This approach, termed Latent Time NODE (LT-NODE), models uncertainty by examining the stability of representations across different integration depths.
LT-NODE and ALT-NODE
In the LT-NODE framework, a global prior distribution (typically a Gamma distribution) is defined over $T$. The model employs Variational Inference to learn an approximate posterior distribution $q(T)$ from the training data. For prediction, multiple end-times are sampled from this posterior. Disagreement between representations at different sampled times serves as a signal for high uncertainty.
To handle cases where different inputs require varying depths, the authors propose Adaptive LT-NODE (ALT-NODE). This variant uses amortized variational inference, where an inference network $r(x)$ predicts a unique posterior distribution over $T$ for each specific input $x$.
Computational Efficiency
A key advantage of this method is its efficiency. To avoid solving the ODE multiple times for different samples of $T$, the model sorts the sampled times and performs a single forward pass. By leveraging the intermediate states and interpolation capabilities of adaptive ODE solvers (e.g., Dopri5), representations at all sampled times are computed simultaneously.
Experimental Results
The authors evaluated LT-NODE and ALT-NODE across several benchmarks, including 1D synthetic regression, image classification (MNIST, CIFAR-10, SVHN), and out-of-distribution (OOD) detection.
Uncertainty and Robustness
The proposed models consistently outperformed standard NODEs and baselines like SDE-Net in terms of Log-Likelihood (LL) and Expected Calibration Error (ECE). As shown in the SVHN experiments, LT-NODE maintains better calibration and higher log-likelihood even under data shifts like image rotation.
In OOD detection tasks (e.g., training on CIFAR-10 and testing on SVHN), ALT-NODE demonstrated superior performance, achieving an average entropy of 1.444 compared to 0.572 for standard NODEs. This indicates a significantly better ability to identify unfamiliar data. Additionally, the models showed enhanced resilience to adversarial attacks, such as FGSM, maintaining lower error rates across various perturbation strengths.
Conclusion
LT-NODE introduces an elegant, depth-centric approach to uncertainty in continuous-depth models. By adding only a few parameters (the shape and rate of the Gamma distribution), it enables automated model selection and robust uncertainty quantification. Future work may explore multi-modal variational posteriors for more complex datasets and further optimization of the computational overhead associated with ODE solvers.
Summary updated