Ode to an ODE

Krzysztof Marcin Choromanski, Jared Quincy Davis, Valerii Likhosherstov, Xingyou Song, Jean-Jacques E. Slotine, Jacob Varley, Honglak Lee, Adrian Weller, Vikas SindhwaniNeurIPS 2020 · OpenReview

Neural Ordinary Differential Equations (ODEs) represent a fascinating shift in deep learning, modeling the continuous-depth limit of residual networks (ResNets). Instead of passing data through discrete layers, the data "flows" according to a differential equation. However, this elegant continuous formulation inherits a classic deep learning headache: as the "depth" (or integration time) increases, gradients tend to either explode to infinity or vanish to zero.

To solve this, researchers have introduced ODEtoODE, a nested framework for Neural ODEs where the network's parameters aren't static. Instead, they evolve according to their own secondary matrix flow. By constraining this parameter flow to a compact manifold, the authors provably eliminate the vanishing and exploding gradient problems, leading to more stable training and superior performance.

The Challenge of Continuous Depth

Previous attempts to fix vanishing and exploding gradients in discrete networks often involved enforcing "orthogonality"—ensuring weight matrices preserve the norm of the data as it passes from layer to layer.

However, applying these discrete constraints to a continuous-time system is non-trivial. The challenge is to create a parameterization that changes over time but remains "well-behaved" throughout the entire continuous flow. If the weights stretch or squash the data too much at any point during integration, the gradients will destabilize.

The ODEtoODE Architecture

The authors propose a hierarchical, nested system. In a standard Neural ODE, the hidden state $x_t$ evolves as $\dot{x}_t = f(x_t, t, \theta)$. In ODEtoODE, the static parameters $\theta$ are replaced by a time-dependent matrix $W_t$ that follows its own ODE.

Architecture and Manifold Flow of ODEtoODE

This creates two distinct but coupled flows:

  1. The Main Flow: $\dot{x}_t = f(W_t x_t)$, where $x_t$ is the hidden state.
  2. The Parameter Flow: $\dot{W}_t = W_t b_\psi(t, W_t)$, where $b_\psi$ is a function that outputs skew-symmetric matrices.

The "secret sauce" of this architecture is the skew-symmetric constraint on $b_\psi$. Mathematically, this guarantees that the matrix $W_t$ stays within the orthogonal group $O(d)$ (visualized as the manifold $\Sigma$ in the figure above). Because the weights are constrained to this compact manifold, they are always rotating the data rather than stretching or squashing it, which keeps the dynamics perfectly stable.

The paper explores two specific flavors of this flow:

  • ISO-ODEtoODEs: These use "isospectral flows" (specifically double-bracket flows), historically used in mathematics to solve combinatorial problems like sorting.
  • Gated-ODEtoODEs: A more flexible version inspired by the gating mechanisms in LSTMs/GRUs, adapted for continuous manifolds.

Key Insights and Results

The researchers evaluated their approach across Reinforcement Learning (OpenAI Gym, DeepMind Control Suite) and Supervised Learning (Corrupted MNIST).

Stability through Geometry

The most striking insight from ODEtoODE is that the "vanishing gradient" problem isn't just a training hurdle—it's a geometric one. By forcing parameters to live on the orthogonal group, the authors provide a theoretical lemma proving that the norm of the loss gradient is bounded by a constant. The model is mathematically immune to the exponential gradient decay or growth seen in standard architectures.

Reinforcement Learning Superiority

Because ODEtoODEs are highly expressive even with a small number of parameters (due to their complex internal dynamics), they are uniquely well-suited for Evolution Strategies (ES), which typically struggle with high-dimensional parameter spaces.

Reinforcement Learning Performance Across 8 Environments

As shown in the training curves above, ODEtoODE (green) was the top performer on 4 out of 8 RL tasks and was among the top two on the remaining four. Crucially, it was the only method that successfully learned a high-performing policy for the highly complex "Humanoid" environment, where other models flatlined.

Robust Classification

On the Corrupted MNIST dataset, which includes 11 different variants of noise and distortions, ODEtoODE outperformed standard Neural ODEs, ResNets, and HyperNets on 11 out of 15 tasks. It showed particular strength in maintaining accuracy under severe "Stripe" and "Glass Blur" corruptions.

Limitations and Future Work

While ODEtoODE offers a mathematically elegant solution to gradient instability, it comes with trade-offs:

  • Fully-Connected Focus: The current work focuses primarily on fully-connected (linear) layers. Extending this manifold-constrained flow to convolutional layers is complex because the definition of "orthogonality" for tensors is less straightforward.
  • Computational Overhead: The matrix exponential computations required for the parameter flow are manageable for smaller hidden sizes (e.g., $h=64$), but scaling this to massive models would require highly efficient Riemannian optimization techniques.
  • Hyperparameter Tuning: Like all ODE-based models, performance is sensitive to the choice of numerical integrator and the integration step size, adding complexity to the training pipeline.

Despite these limitations, ODEtoODE successfully demonstrates that nesting continuous flows and constraining them geometrically is a powerful path forward for stable, deep continuous-time neural networks.

Summary updated

Abstract

We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the orthogonal group O(d). This nested system of two flows, where the parameter-flow is constrained to lie on the compact manifold, provides stability and effectiveness of training and solves the gradient vanishing-explosion problem which is intrinsically related to training deep neural network architectures such as Neural ODEs. Consequently, it leads to better downstream models, as we show on the example of training reinforcement learning policies with evolution strategies, and in the supervised learning setting, by comparing with previous SOTA baselines. We provide strong convergence results for our proposed mechanism that are independent of the width of the network, supporting our empirical studies. Our results show an intriguing connection between the theory of deep neural networks and the field of matrix flows on compact manifolds.
BibTeX
@inproceedings{choromanski2020ode,
  author = {Krzysztof Marcin Choromanski and Jared Quincy Davis and Valerii Likhosherstov and Xingyou Song and Jean-Jacques E. Slotine and Jacob Varley and Honglak Lee and Adrian Weller and Vikas Sindhwani},
  title = {{Ode to an ODE}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=TZmRbTh85FO},
}

Loading related content...

Figure