Abstract: Neural ordinary differential equations (NODE) have been proposed as a continuous depth generalization to popular deep learning models such as Residual networks (ResNets). They provide parameter efficiency and automate the model selection process in deep learning models to some extent. However, they lack the much-required uncertainty modelling and robustness capabilities which are crucial for their use in several real-world applications such as autonomous driving and healthcare. We propose a novel and unique approach to model uncertainty in NODE by considering a distribution over the end-time T of the ODE solver. The proposed approach, latent time NODE (LT-NODE), treats T as a latent variable and apply Bayesian learning to obtain a posterior distribution over T from the data. In particular, we use variational inference to learn an approximate posterior and the model parameters. Prediction is done by considering the NODE representations from different samples of the posterior and can be done efficiently using a single forward pass. As T implicitly defines the depth of a NODE, posterior distribution over T would also help in model selection in NODE. We also propose, adaptive latent time NODE (ALT-NODE), which allow each data point to have a distinct posterior distribution over end-times. ALT-NODE uses amortized variational inference to learn an approximate posterior using inference networks. We demonstrate the effectiveness of the proposed approaches in modelling uncertainty and robustness through experiments on synthetic and several real-world image classification data.
Introduction
Neural Ordinary Differential Equations (NODEs) represent a continuous-depth paradigm in deep learning, generalizing Residual Networks (ResNets) by modeling hidden state transitions as a continuous flow. While parameter-efficient, standard NODEs provide only point estimates, lacking the uncertainty quantification necessary for safety-critical applications like autonomous driving or healthcare. Furthermore, the "depth" of a NODE—determined by the integration end-time $T$—is typically a manually tuned hyperparameter.
Methodology: Latent Time NODEs
To address these limitations, we introduce Latent Time Neural Ordinary Differential Equations (LT-NODE). The core innovation is treating the integration end-time $T$ as a latent variable rather than a fixed constant.
LT-NODE and ALT-NODE
In LT-NODE, we assume $T$ follows a Gamma distribution. We employ Variational Inference to learn the parameters of this distribution alongside the model weights. During inference, we sample multiple end-times from the learned posterior. Because the ODE solver computes states sequentially, these samples can be obtained in a single forward pass, providing efficient uncertainty estimates.
Adaptive Latent Time NODE (ALT-NODE) extends this by making the end-time distribution input-dependent. Using an inference network, ALT-NODE learns a unique posterior for each data point, allowing the model to adapt its computational depth based on input complexity. This effectively automates model selection while guarding against overconfident errors.
Experimental Evaluation
We evaluated our models on synthetic 1D regression and standard image classification benchmarks including CIFAR10, SVHN, MNIST, and Fashion-MNIST.
Uncertainty and Calibration
The proposed models demonstrate superior calibration and out-of-distribution (OOD) detection. In OOD scenarios, ALT-NODE exhibits significantly higher entropy compared to standard NODEs, indicating a better ability to signal "unknown" data. For instance, on SVHN OOD data, ALT-NODE achieved an entropy of 1.444 vs. 0.572 for a standard NODE.
As shown in the figure above, LT-NODE and ALT-NODE maintain better Log-Likelihood (LL), lower Brier scores, and lower Expected Calibration Error (ECE) as data undergoes distribution shifts, such as image rotation. This suggests that the models are less likely to be fooled by data shifts and provide more reliable probability estimates.
Robustness
Our approach also enhances robustness against adversarial attacks. In tests using the Fast Gradient Sign Method (FGSM), LT-NODE variants consistently maintained lower error rates compared to baselines as perturbation strength ($\epsilon$) increased, proving their utility in security-sensitive environments.
Conclusion
LT-NODE provides a principled Bayesian framework for modeling uncertainty in continuous-depth networks. By treating depth as a latent distribution, it automates model selection and provides reliable confidence estimates with minimal computational overhead. Future work will explore more complex variational distributions and scaling these methods to high-resolution datasets like ImageNet.
Summary updated