An Ode to an ODE

Krzysztof Choromanski, Jared Quincy Davis, Valerii Likhosherstov, Xingyou Song, Jean-Jacques E. Slotine, Jacob Varley, Honglak Lee, Adrian Weller, Vikas SindhwaniarXiv 2020 · OpenReview

TL;DR

This paper introduces "ODEtoODE," a novel Neural Ordinary Differential Equation (Neural ODE) architecture where the ODE's parameters evolve on the compact orthogonal group O(d). This approach stabilizes training and provably solves the vanishing/exploding gradient problem, leading to improved performance in reinforcement learning and supervised learning tasks.

What Problem Does This Paper Address?

Deep neural networks, particularly Neural ODEs, can suffer from training instabilities like vanishing or exploding gradients, especially as they become deeper. While discrete architectures have explored orthogonal constraints to mitigate this, continuous Neural ODEs have lacked a robust, principled solution. The paper aims to bridge this gap by introducing a method that leverages the geometric properties of compact manifolds to stabilize the training of Neural ODEs.

Methods

The core innovation is the ODEtoODE architecture, which features a nested system of two ODEs. The primary ODE governs the network's activations ($x_t$), while a secondary ODE governs the parameters of the primary ODE ($W_t$). Crucially, the parameter ODE is constrained to evolve on the orthogonal group O(d), a compact manifold. This constraint is enforced by ensuring that the derivative of the parameter ODE outputs skew-symmetric matrices, which, when exponentiated, map to the orthogonal group.

Two specific ODEtoODE architectures are proposed:

  • ISO-ODEtoODEs: Leverage isospectral flows (double-bracket flows) which preserve matrix spectra.
  • Gated-ODEtoODEs: Utilize gated mechanisms, inspired by previous work, to control parameter updates.

The paper also discusses learning initial orthogonal matrices ($W_0$) and applying Riemannian gradients for optimization on the manifold. Theoretical analysis proves that ODEtoODEs inherently solve the gradient vanishing/exploding problem.

Datasets

  • Reinforcement Learning: OpenAI Gym and DeepMind Control Suite tasks.
  • Supervised Learning: Corrupted MNIST dataset (11 variants).

Key Insights and Ideas

  • Geometric Constraints for Stability: Constraining parameter evolution to a compact manifold (O(d)) is a powerful mechanism for stabilizing deep learning models, particularly Neural ODEs.
  • Bridging Deep Learning and Differential Geometry: The work establishes a strong connection between deep neural network theory and the mathematical field of matrix flows on compact manifolds, opening new avenues for research.
  • Provable Gradient Stability: The ODEtoODE architecture offers theoretical guarantees against vanishing and exploding gradients, a long-standing challenge in deep learning.
  • Compact Models for RL: The paper demonstrates that structured, compact parameterizations enabled by ODEtoODE can be highly beneficial for reinforcement learning, especially when combined with evolution strategies.

Key Results

  • Reinforcement Learning: ODEtoODE significantly outperforms other methods on four out of eight RL tasks and is among the top performers on the remaining four, showing consistent learning of good policies.
  • Supervised Learning: ODEtoODE variants achieve state-of-the-art results on the Corrupted MNIST dataset, outperforming DenseNets, NODE, NANODE, and HyperNets on 11 out of 15 tasks.
  • Theoretical Guarantees: The paper provides theoretical proofs demonstrating that ODEtoODEs solve the gradient vanishing/exploding problem and exhibit strong convergence properties independent of network depth.

Limitations and Future Work

  • Computational Cost: While exponential mapping for orthogonal flows is efficient, the paper notes that HyperNets (a baseline) are characterized by a large number of parameters, impacting training speed.
  • Convolutional Architectures: The current work focuses on fully-connected architectures. The authors suggest that exploring orthogonality in convolutional settings is a direction for future work.
  • Broader Applications: The authors envision applying ODEtoODEs to model-based reinforcement learning and robotics, and exploring learnable isospectral flows for combinatorial problems.

Summary updated

Abstract

We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the orthogonal group O(d). This nested system of two flows, where the parameter-flow is constrained to lie on the compact manifold, provides stability and effectiveness of training and provably solves the gradient vanishing-explosion problem which is intrinsically related to training deep neural network architectures such as Neural ODEs. Consequently, it leads to better downstream models, as we show on the example of training reinforcement learning policies with evolution strategies, and in the supervised learning setting, by comparing with previous SOTA baselines. We provide strong convergence results for our proposed mechanism that are independent of the depth of the network, supporting our empirical studies. Our results show an intriguing connection between the theory of deep neural networks and the field of matrix flows on compact manifolds.
BibTeX
@misc{choromanski2020ode,
  author = {Krzysztof Choromanski and Jared Quincy Davis and Valerii Likhosherstov and Xingyou Song and Jean-Jacques E. Slotine and Jacob Varley and Honglak Lee and Adrian Weller and Vikas Sindhwani},
  title = {{An Ode to an ODE}},
  year = {2020},
  eprint = {2006.11421},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.11421},
}
Figure