On Robustness of Neural Ordinary Differential Equations

Hanshu Yan, Jiawei Du, Vincent Y. F. Tan, Jiashi Feng

Deep neural networks, particularly Convolutional Neural Networks (CNNs), are notoriously fragile. Small amounts of random noise or carefully crafted "adversarial perturbations"—changes to an image so minute they are invisible to the human eye—can cause these models to fail spectacularly.

While Neural Ordinary Differential Equations (Neural ODEs or NODEs) have gained popularity for their parameter efficiency and ability to model continuous-time data, their security properties have remained largely unexplored. This paper investigates the inherent security benefits of Neural ODEs, demonstrating that they are naturally more robust to noise and adversarial attacks than standard CNNs. Furthermore, the authors introduce TisODE, a variant that uses time-invariance and steady-state constraints to create an even more resilient, "drop-in" module for deep learning architectures.

The Vulnerability of Standard Networks

Standard CNNs perform discrete jumps from layer to layer. This discrete nature can create jagged decision boundaries, making it easier for an adversarial attack to find a "gap" or a sudden spike to exploit. The authors set out to answer two fundamental questions: Are NODEs intrinsically more robust than CNNs? And if so, can we mathematically exploit their structure to make them even tougher?

To test this, the researchers built "ODENets"—networks where the middle layers are replaced by a continuous Neural ODE block—and compared them against standard CNNs with a similar number of parameters across datasets like MNIST, SVHN, and a custom ImgNet10 subset.

Why Neural ODEs? The Theoretical Edge

The researchers approached the problem through both empirical testing and theoretical modeling, leveraging a fundamental mathematical property: the uniqueness of ODE solutions.

In a well-posed ODE, two different starting points (inputs) will generate two different "integral curves" (paths through the network) that can never intersect. Unlike CNNs, NODEs follow a smooth, continuous trajectory.

Integral curves of an ODE cannot intersect, naturally bounding perturbed inputs.

This leads to what can be described as a "Sandwich" Effect. Because ODE paths cannot cross, a perturbed input is effectively "trapped" between the paths of its unperturbed neighbors. If the network correctly classifies the surrounding neighborhood, this non-intersecting property acts as a natural guardrail, preventing the perturbed input from wandering into the "territory" of a different class.

Introducing TisODE: Time-Invariant Steady Neural ODEs

To improve upon vanilla NODEs, the authors proposed two key modifications, culminating in the Time-Invariant Steady Neural ODE (TisODE):

  1. Time-Invariance: They removed the explicit time dependence from the dynamics function. This makes the "flow" of data more stable and less dependent on the specific integration time.
  2. Steady-State Regularization: They added a loss term ($L_{ss}$) that penalizes the model if the internal state is still changing rapidly at the end of the ODE's "time" limit. By enforcing a "steady state," TisODE ensures that the model doesn't just pass through a correct classification state, but actually settles there. This makes the final output much less sensitive to the exact depth at which the ODE solver stops, effectively "absorbing" small input perturbations.

Empirical Results and Robustness Gains

The empirical results demonstrate that ODENets consistently outperform CNNs in noisy and adversarial environments. For instance, on MNIST with heavy Gaussian noise ($\sigma=100$), the vanilla ODENet maintained 73.2% accuracy, while the equivalent CNN plummeted to 56.4%. Against FGSM (Fast Gradient Sign Method) attacks on the SVHN dataset, the ODENet achieved 43.0% accuracy, nearly tripling the CNN's 13.7%.

Robustness evaluation showing TisODE outperforming CNNs and combining effectively with other defenses.

The proposed TisODE model pushed these boundaries even further. On MNIST under a strict PGD (Projected Gradient Descent) attack, TisODE achieved 78.9% accuracy, significantly higher than the vanilla ODENet's 12.3% in the same setting. Crucially, the authors showed that TisODE can be seamlessly combined with other state-of-the-art defenses (like feature denoising), acting as a highly effective "drop-in" component to boost the overall robustness of larger systems.

Limitations and Future Work

While Neural ODEs offer a promising path toward robust deep learning, there are hurdles to overcome:

  • Computational Overhead: Solving ODEs requires numerical solvers (like Euler or Runge-Kutta methods). While parameter-efficient, these solvers can be slower to train and execute than standard discrete layers.
  • Representational Limits: The non-intersecting property, while excellent for robustness, can sometimes limit the mathematical "expressiveness" of the network—a challenge often addressed by Augmented Neural ODEs.
  • Scaling: While results on ImgNet10 are highly promising, testing these properties on the full-scale ImageNet dataset remains a computational challenge for future research.

Summary updated

Abstract

Neural ordinary differential equations (ODEs) have been attracting increasing attention in various research domains recently. There have been some works studying optimization issues and approximation capabilities of neural ODEs, but their robustness is still yet unclear. In this work, we fill this important gap by exploring robustness properties of neural ODEs both empirically and theoretically. We first present an empirical study on the robustness of the neural ODE-based networks (ODENets) by exposing them to inputs with various types of perturbations and subsequently investigating the changes of the corresponding outputs. In contrast to conventional convolutional neural networks (CNNs), we find that the ODENets are more robust against both random Gaussian perturbations and adversarial attack examples. We then provide an insightful understanding of this phenomenon by exploiting a certain desirable property of the flow of a continuous-time ODE, namely that integral curves are non-intersecting. Our work suggests that, due to their intrinsic robustness, it is promising to use neural ODEs as a basic block for building robust deep network models. To further enhance the robustness of vanilla neural ODEs, we propose the time-invariant steady neural ODE (TisODE), which regularizes the flow on perturbed data via the time-invariant property and the imposition of a steady-state constraint. We show that the TisODE method outperforms vanilla neural ODEs and also can work in conjunction with other state-of-the-art architectural methods to build more robust deep networks.
BibTeX
@inproceedings{yan2020robustness,
  author = {Hanshu Yan and Jiawei Du and Vincent Y. F. Tan and Jiashi Feng},
  title = {{On Robustness of Neural Ordinary Differential Equations}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=iLi8EGT9Sy_},
}
Figure