TL;DR
Instead of using a fixed numerical integrator to train Neural Ordinary Differential Equations (ODEs), this paper introduces "meta-solvers" that vary the solver's parameters during training. This simple change prevents the model from overfitting to a specific discretization scheme and significantly enhances its robustness against adversarial attacks without increasing computational cost.
What Problem Does This Paper Address?
Neural ODEs have gained popularity by treating neural network layers as a continuous flow, modeled by an ODE and solved using numerical integrators like Runge-Kutta methods. Typically, researchers pick a specific solver (e.g., the Midpoint rule) and keep it fixed throughout training.
The authors argue that this leads to discretization overfitting: the neural network learns to compensate for the specific numerical errors and properties of that one solver. Consequently, the model might perform poorly if a different solver is used at inference time, and more importantly, it remains vulnerable to adversarial attacks. The paper explores whether introducing variability into the "solver space" can produce more generalized and robust models.
Methods
The core innovation is treating the ODE solver not as a constant, but as a parameterizable component of the training loop.
- Solver Parametrization: The authors focus on explicit Runge-Kutta (RK) methods. They show that families of RK methods (like 2nd-order or 4th-order) can be uniquely defined by one or two scalar parameters ($u$) using a mathematical framework called a Butcher tableau. Examples of Butcher tableaux for RK2 methods, showing how a single parameter $u$ defines the solver.
- Solver Switching: At each training epoch, the algorithm randomly selects a solver from a pre-defined discrete set of parameters.
- Solver Smoothing: This is a continuous version of switching. In every epoch, the solver parameter $u$ is sampled from a continuous distribution (like a Normal or Cauchy distribution) centered around a base solver. This forces the model to perform well across a whole neighborhood of numerical integrators.
- Meta-Learning Perspective: This approach acts like a meta-model affecting a learner; by changing the "environment" (the solver) in which the weights are learned, the model develops more resilient internal dynamics.
Datasets
The authors evaluated their approach on two standard image classification benchmarks:
- MNIST: Used for initial proof-of-concept and visualizing how robust accuracy changes with solver parameters.
- CIFAR-10: Used for more rigorous testing of solver smoothing and its combination with adversarial training techniques.
Key Insights and Ideas
-
The Solver Choice is a Hidden Hyperparameter: The specific choice of $u$ in an RK2 solver significantly impacts the resulting model's robust accuracy, even if the "order" of the solver remains the same. Figure 3: Robust accuracy on MNIST varies significantly depending on the solver parameter $u$.
-
Overcoming Discretization Artifacts: By training with a variety of solvers, the network is discouraged from "relying" on the specific numerical quirks of any single integration scheme.
- Robustness for Free: Unlike standard adversarial training, which requires generating expensive adversarial examples during every training step, solver smoothing adds virtually zero computational overhead. You still only perform one forward/backward pass; you just use a slightly different solver to do it.
- Synergy with Adversarial Training: Solver smoothing can be layered on top of traditional adversarial training (like FGSM or PGD training) to push robustness even further.
Key Results
- Improved Robustness: On CIFAR-10, using solver smoothing increased robust accuracy against PGD attacks from 31.81% (standard training) to 32.78%.
- Enhanced Adversarial Training: When combined with adversarial training on CIFAR-10 ($\epsilon = 8/255$), the robust accuracy against PGD attacks improved from 36.13% to 36.37%, and against FGSM attacks from 40.86% to 41.39%.
- Training Stability: The authors observed that solver smoothing actually allowed them to train models using higher learning rates than would otherwise be stable.
Limitations and Future Work
- Ensembling Costs: The authors tested "ensembling" (averaging the outputs of multiple solvers), but found it provided insignificant gains while being much more computationally expensive.
- Optimal Distributions: While they used Normal and Cauchy distributions for smoothing, the "optimal" distribution for sampling solver parameters remains an open question.
- Extension to Other Solvers: The current work focuses on explicit Runge-Kutta methods. Future research could extend this "meta-solver" philosophy to adaptive-step solvers or multi-step methods.
- Automatic Tuning: Instead of random sampling, could the solver parameters themselves be learned via gradient descent? The authors noted initial attempts were unstable, suggesting a need for better constrained optimization techniques in the solver space.
Summary updated