Abstract
Neural ordinary differential equations (ODEs) have been attracting increasing attention in various research domains recently. There have been some works studying optimization issues and approximation capabilities of neural ODEs, but their robustness is still yet unclear. In this work, we fill this important gap by exploring robustness properties of neural ODEs both empirically and theoretically. We first present an empirical study on the robustness of the neural ODE-based networks (ODENets) by exposing them to inputs with various types of perturbations and subsequently investigating the changes of the corresponding outputs. In contrast to conventional convolutional neural networks (CNNs), we find that the ODENets are more robust against both random Gaussian perturbations and adversarial attack examples. We then provide an insightful understanding of this phenomenon by exploiting a certain desirable property of the flow of a continuous-time ODE, namely that integral curves are non-intersecting. Our work suggests that, due to their intrinsic robustness, it is promising to use neural ODEs as a basic block for building robust deep network models. To further enhance the robustness of vanilla neural ODEs, we propose the time-invariant steady neural ODE (TisODE), which regularizes the flow on perturbed data via the time-invariant property and the imposition of a steady-state constraint. We show that the TisODE method outperforms vanilla neural ODEs and also can work in conjunction with other state-of-the-art architectural methods to build more robust deep networks.
Introduction
Deep learning models, particularly Convolutional Neural Networks (CNNs), are famously "brittle." A tiny amount of random noise or a carefully crafted adversarial perturbation can cause a state-of-the-art model to fail. While Neural ODEs—models that represent network layers as a continuous flow rather than discrete steps—have gained popularity for their memory efficiency and invertibility, their security properties remained largely unexplored. This paper investigates whether Neural ODEs are naturally more robust than CNNs and introduces methods to further harden these models.
Theoretical Analysis: The Non-Intersecting Property
A key mathematical property of ODEs is that their integral curves do not intersect. In the context of a neural network, this means that if two inputs start close together in the representation space, their trajectories through the network are constrained by the trajectories of their neighbors.
As illustrated in Figure 2, a perturbed input is effectively "sandwiched" between the trajectories of nearby clean inputs. This provides a natural, intrinsic regularization that prevents the wild output swings common in discrete CNNs, which can "shatter" the input space. Neural ODEs move data through a continuous flow, which preserves the topology of the input better than discrete transformations.
Empirical Robustness of ODENets
To validate these theoretical insights, the researchers compared "ODENets" (networks where the middle representation is an ODE block) against CNNs with a similar number of parameters across standard benchmarks including MNIST, SVHN, and ImgNet10.
As shown in Figure 4, ODENets consistently outperform CNNs in hostile environments. For instance, on the SVHN dataset under FGSM-5/255 attacks, the ODENet achieved 43.0% accuracy, nearly triple the CNN’s 13.7%. Similarly, on MNIST with heavy Gaussian noise (σ=100), the ODENet maintained 73.2% accuracy, while the equivalent CNN plummeted to 56.4%.
Enhancing Robustness with TisODE
To improve on the "vanilla" Neural ODE, the authors proposed the Time-Invariant Steady Neural ODE (TisODE), which introduces two key modifications:
- Time-Invariance: Removing explicit time dependence from the dynamics function ensures the system's behavior is consistent regardless of the "time" at which a state is evaluated.
- Steady-State Constraint: A new loss term ($L_{ss}$) penalizes the network if the representation continues to change rapidly after the terminal time $T$, forcing the network to reach a stable, steady state.
TisODE significantly pushes the boundaries of robustness. On MNIST under a high-strength PGD-0.3 attack, TisODE achieved 78.9% accuracy, significantly outperforming both the vanilla ODENet (12.3%) and the CNN (5.3%). Furthermore, TisODE can be used as a modular defense; when paired with "Feature Denoising," it provides an additive boost to robustness.
Conclusion
Neural ODEs offer a promising path toward building more robust deep learning models. Their continuous nature and non-intersecting flow properties provide a natural defense against perturbations. While numerical solvers introduce some computational overhead, the resulting stability and synergy with other defense mechanisms make them a compelling choice for security-critical applications.
Summary updated