Improving Robustness and Uncertainty Modelling in Neural Ordinary Differential Equations

Srinivas Anumasa, P. K. Srijith

Abstract

Deep learning models such as Resnets have resulted in state-of-the-art accuracy in many computer vision problems. Neural ordinary differential equations (NODE) provides a continuous depth generalization of Resnets and overcome drawbacks of Resnet such as model selection and parameter complexity. Though NODE is more robust than Resnet, we find that NODE based architectures are still far away from providing robustness and uncertainty handling required for many computer vision problems. We propose novel NODE models which address these drawbacks. In particular, we propose Gaussian processes (GPs) to model the fully connected neural networks in NODE (NODE-GP) to improve robustness and uncertainty handling capabilities of NODE. The proposed model is flexible to accommodate different NODE architectures, and further improves the model selection capabilities in NODEs. We also find that numerical techniques play an important role in modelling NODE robustness, and propose to use different numerical techniques to improve NODE robustness. We demonstrate the superior robustness and uncertainty handling capabilities of proposed models on adversarial attacks and out-of-distribution experiments for the image classification tasks.

Introduction

Deep learning models, particularly ResNets, have achieved remarkable success in computer vision. However, they often suffer from overconfidence, failing to provide reliable uncertainty estimates when encountering adversarial noise or out-of-distribution (OOD) data. Neural Ordinary Differential Equations (NODEs) offer a continuous-depth alternative that is more parameter-efficient, yet they remain vulnerable to adversarial attacks and lack principled uncertainty quantification. This research addresses these limitations by introducing the NODE-GP framework and investigating the impact of numerical solvers on model robustness.

Methodology: The NODE-GP Framework

The core contribution of this work is the integration of Gaussian Processes (GPs) into the NODE architecture. Instead of utilizing standard fully connected layers for classification, the proposed model uses NODE or Augmented NODE (ANODE) blocks as feature extractors, passing the resulting representations into a Sparse Gaussian Process.

The proposed NODE-GP architecture, which replaces final layers with a Gaussian Process to enhance uncertainty modelling.

By employing variational inference and sparse approximations, the model maintains computational efficiency while providing a full probability distribution over outputs. This allows the model to signal high uncertainty when test samples deviate significantly from the training distribution, bridging the gap between continuous-depth modelling and Bayesian uncertainty.

Numerical Solvers as a Regularization Mechanism

A key finding of this work is that the choice of numerical solver is not merely a technical implementation detail but a critical factor in adversarial robustness. The authors compare first-order methods like the Euler method with higher-order methods like Runge-Kutta 4 (RK4). Higher-order solvers produce more stable trajectories and better-defined class boundaries in the latent space. This geometric clarity effectively acts as a form of regularization that makes it significantly harder for an adversary to "nudge" an image across a decision boundary.

Experimental Results

The models were evaluated on CIFAR-10 for robustness and CIFAR-100 for OOD detection.

Adversarial Robustness

Under Projective Gradient Descent (PGD) attacks, the NODE-GP variants demonstrated superior resilience. As shown in the performance analysis, the proposed models maintain higher accuracy across increasing noise intensities compared to standard ResNet and NODE baselines. For instance, at a noise level of $\epsilon \geq 0.05$, the proposed NODE_Adap-NODE_NT3-GP model was 52% more accurate than the baseline NODE.

Test accuracy vs. noise intensity ($\epsilon$), demonstrating that NODE-GP variants significantly outperform standard architectures under adversarial pressure.

Uncertainty and OOD Detection

In OOD experiments using CIFAR-100, the NODE-GP model achieved an entropy score of 1.51, a massive improvement over the 0.137 score of standard NODEs. This indicates a much stronger ability to identify "unknown" data rather than making overconfident, incorrect predictions. Furthermore, NODE-GP achieves this performance with significantly fewer parameters than ResNet-GP counterparts (13.2M vs 23.4M).

Conclusion

This research demonstrates that combining the continuous-depth advantages of NODEs with the Bayesian uncertainty of Gaussian Processes creates a more robust and reliable deep learning model. Furthermore, it highlights that the numerical solver itself is a vital component of a model's defense strategy. While higher-order solvers and GPs introduce some computational overhead, the gains in safety and parameter efficiency make NODE-GP a compelling choice for high-stakes applications like autonomous driving or medical diagnosis.

Summary updated

Abstract

Deep learning models such as Resnets have resulted in state-of-the-art accuracy in many computer vision problems. Neural ordinary differential equations (NODE) provides a continuous depth generalization of Resnets and overcome drawbacks of Resnet such as model selection and parameter complexity. Though NODE is more robust than Resnet, we find that NODE based architectures are still far away from providing robustness and uncertainty handling required for many computer vision problems. We propose novel NODE models which address these drawbacks. In particular, we propose Gaussian processes (GPs) to model the fully connected neural networks in NODE (NODE-GP) to improve robustness and uncertainty handling capabilities of NODE. The proposed model is flexible to accommodate different NODE architectures, and further improves the model selection capabilities in NODEs. We also find that numerical techniques play an important role in modelling NODE robustness, and propose to use different numerical techniques to improve NODE robustness. We demonstrate the superior robustness and uncertainty handling capabilities of proposed models on adversarial attacks and out-of-distribution experiments for the image classification tasks.
BibTeX
@inproceedings{anumasa2021improving,
  author = {Srinivas Anumasa and P. K. Srijith},
  title = {{Improving Robustness and Uncertainty Modelling in Neural Ordinary Differential Equations}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=T3yVcnMPeTT},
}
Figure