Meta-Solver for Neural Ordinary Differential Equations

Julia Gusak, Alexandr Katrutsa, Talgat Daulbaev, Andrzej Cichocki, Ivan Oseledets

TL;DR

Instead of using a fixed numerical integrator to train Neural Ordinary Differential Equations (ODEs), this paper introduces "meta-solvers" that vary the solver's parameters during training. This simple change prevents the model from overfitting to a specific discretization scheme and significantly enhances its robustness against adversarial attacks without increasing computational cost.

What Problem Does This Paper Address?

Neural ODEs have gained popularity by treating neural network layers as a continuous flow, modeled by an ODE and solved using numerical integrators like Runge-Kutta methods. Typically, researchers pick a specific solver (e.g., the Midpoint rule) and keep it fixed throughout training.

The authors argue that this leads to discretization overfitting: the neural network learns to compensate for the specific numerical errors and properties of that one solver. Consequently, the model might perform poorly if a different solver is used at inference time, and more importantly, it remains vulnerable to adversarial attacks. The paper explores whether introducing variability into the "solver space" can produce more generalized and robust models.

Methods

The core innovation is treating the ODE solver not as a constant, but as a parameterizable component of the training loop.

  1. Solver Parametrization: The authors focus on explicit Runge-Kutta (RK) methods. They show that families of RK methods (like 2nd-order or 4th-order) can be uniquely defined by one or two scalar parameters ($u$) using a mathematical framework called a Butcher tableau. Examples of Butcher tableaux for RK2 methods, showing how a single parameter $u$ defines the solver.
Figure 2: Examples of Butcher tableaux corresponding to 2-stage RK methods of the 2-nd order.
  1. Solver Switching: At each training epoch, the algorithm randomly selects a solver from a pre-defined discrete set of parameters.
  2. Solver Smoothing: This is a continuous version of switching. In every epoch, the solver parameter $u$ is sampled from a continuous distribution (like a Normal or Cauchy distribution) centered around a base solver. This forces the model to perform well across a whole neighborhood of numerical integrators.
  3. Meta-Learning Perspective: This approach acts like a meta-model affecting a learner; by changing the "environment" (the solver) in which the weights are learned, the model develops more resilient internal dynamics.

Datasets

The authors evaluated their approach on two standard image classification benchmarks:

  • MNIST: Used for initial proof-of-concept and visualizing how robust accuracy changes with solver parameters.
  • CIFAR-10: Used for more rigorous testing of solver smoothing and its combination with adversarial training techniques.

Key Insights and Ideas

  • The Solver Choice is a Hidden Hyperparameter: The specific choice of $u$ in an RK2 solver significantly impacts the resulting model's robust accuracy, even if the "order" of the solver remains the same. Figure 3: Robust accuracy on MNIST varies significantly depending on the solver parameter $u$.

  • Overcoming Discretization Artifacts: By training with a variety of solvers, the network is discouraged from "relying" on the specific numerical quirks of any single integration scheme.

  • Robustness for Free: Unlike standard adversarial training, which requires generating expensive adversarial examples during every training step, solver smoothing adds virtually zero computational overhead. You still only perform one forward/backward pass; you just use a slightly different solver to do it.
  • Synergy with Adversarial Training: Solver smoothing can be layered on top of traditional adversarial training (like FGSM or PGD training) to push robustness even further.
Figure 3: The value u for the 2-nd order two stages Runge-Kutta method affects the robust accuracy of neural ODE model for MNIST classification task.

Key Results

  • Improved Robustness: On CIFAR-10, using solver smoothing increased robust accuracy against PGD attacks from 31.81% (standard training) to 32.78%.
  • Enhanced Adversarial Training: When combined with adversarial training on CIFAR-10 ($\epsilon = 8/255$), the robust accuracy against PGD attacks improved from 36.13% to 36.37%, and against FGSM attacks from 40.86% to 41.39%.
  • Training Stability: The authors observed that solver smoothing actually allowed them to train models using higher learning rates than would otherwise be stable.

Limitations and Future Work

  • Ensembling Costs: The authors tested "ensembling" (averaging the outputs of multiple solvers), but found it provided insignificant gains while being much more computationally expensive.
  • Optimal Distributions: While they used Normal and Cauchy distributions for smoothing, the "optimal" distribution for sampling solver parameters remains an open question.
  • Extension to Other Solvers: The current work focuses on explicit Runge-Kutta methods. Future research could extend this "meta-solver" philosophy to adaptive-step solvers or multi-step methods.
  • Automatic Tuning: Instead of random sampling, could the solver parameters themselves be learned via gradient descent? The authors noted initial attempts were unstable, suggesting a need for better constrained optimization techniques in the solver space.

Summary updated

Abstract

A conventional approach to train neural ordinary differential equations (ODEs) is to fix an ODE solver and then learn the neural network's weights to optimize a target loss function. However, such an approach is tailored for a specific discretization method and its properties, which may not be optimal for the selected application and yield the overfitting to the given solver. In our paper, we investigate how the variability in solvers' space can improve neural ODEs performance. We consider a family of Runge-Kutta methods that are parameterized by no more than two scalar variables. Based on the solvers' properties, we propose an approach to decrease neural ODEs overfitting to the pre-defined solver, along with a criterion to evaluate such behaviour. Moreover, we show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks. Recently it was shown that neural ODEs demonstrate superiority over conventional CNNs in terms of robustness. Our work demonstrates that the model robustness can be further improved by optimizing solver choice for a given task. The source code to reproduce our experiments is available at https://github.com/juliagusak/neural-ode-metasolver.
BibTeX
@misc{gusak2021meta,
  author = {Julia Gusak and Alexandr Katrutsa and Talgat Daulbaev and Andrzej Cichocki and Ivan Oseledets},
  title = {{Meta-Solver for Neural Ordinary Differential Equations}},
  year = {2021},
  eprint = {2103.08561},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.08561},
}
Figure