TorchDyn: A Neural Differential Equations Library

Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, Jinkyoo ParkarXiv 2020 · arXiv

Abstract

TorchDyn is an open-source PyTorch library designed to simplify the implementation and training of Neural Differential Equations (NDEs). By providing a modular, object-oriented framework, it bridges the gap between discrete neural networks and continuous-depth models. The library supports a wide array of state-of-the-art dynamical systems, sensitivity methods like the adjoint method, and specialized layers for physics-informed learning.

Introduction

"Continuous-depth" learning, where neural networks parameterize the derivative of a system's state, has shown significant promise in density estimation, time-series forecasting, and physics-informed modeling. However, implementing these models is traditionally difficult. Unlike standard discrete layers, NDEs require numerical solvers (e.g., Runge-Kutta) and specialized sensitivity analysis (the adjoint method) for memory-efficient backpropagation. TorchDyn addresses these challenges by providing a "plug-and-play" framework that eliminates boilerplate code and lowers the barrier to entry for researchers.

Methods and Architecture

TorchDyn introduces a modular philosophy that decouples model logic from solver logic. The architecture is centered around two core abstractions:

  • NeuralDE: A high-level class that wraps the differential equation and solver, behaving like a standard PyTorch nn.Module.
  • DEFunc: An internal class managing the vector field machinery, including higher-order dynamics and integral cost functions.
Modular architecture of TorchDyn, showing the hierarchy of support modules and core abstractions.

The library includes several support modules to enhance model expressivity:

  • Sensitivity Tools: Support for standard backpropagation and the $O(1)$ memory Adjoint Method.
  • Depth-Variance Layers: Layers like DepthCat and GalLinear allow parameters to evolve smoothly across continuous depth.
  • Physics-Inspired Wrappers: Dedicated modules for Hamiltonian and Lagrangian Neural Networks to enforce physical constraints like energy conservation.

Comprehensive Model Support

TorchDyn is designed to be a comprehensive suite for continuous-depth modeling. It supports a vast range of state-of-the-art (SOTA) models, including Augmented Neural ODEs (ANODEs), Continuous Normalizing Flows (CNFs), and Graph Neural ODEs.

Feature matrix of state-of-the-art continuous-depth models supported by TorchDyn.

This breadth allows researchers to benchmark different architectures within a single unified framework. For instance, swapping a standard neural network for a Hamiltonian one requires minimal code changes, a significant improvement over previous monolithic implementations.

Key Insights

  • Modularity: Decoupling solvers from models allows for greater flexibility and easier experimentation.
  • Integral Losses: TorchDyn uniquely supports gradients for losses defined over the entire integration path, which is vital for optimal control and physics-informed machine learning.
  • Ecosystem Integration: While Julia's SciML offers extensive solvers, TorchDyn provides a premier Pythonic alternative that leverages the existing PyTorch community and GPU-accelerated ecosystem.

Conclusion

TorchDyn successfully democratizes Neural Differential Equations by providing a robust, modular, and memory-efficient library. It reduces code complexity and supports a wide variety of SOTA models. Future work aims to expand the library's solver variety—bridging the gap with non-Python ecosystems—and further integrate stochastic differential equations (SDEs) and "hypersolvers" for faster numerical integration.

Summary updated

Abstract

Continuous-depth learning has recently emerged as a novel perspective on deep learning, improving performance in tasks related to dynamical systems and density estimation. Core to these approaches is the neural differential equation, whose forward passes are the solutions of an initial value problem parametrized by a neural network. Unlocking the full potential of continuous-depth models requires a different set of software tools, due to peculiar differences compared to standard discrete neural networks, e.g inference must be carried out via numerical solvers. We introduce TorchDyn, a PyTorch library dedicated to continuous-depth learning, designed to elevate neural differential equations to be as accessible as regular plug-and-play deep learning primitives. This objective is achieved by identifying and subdividing different variants into common essential components, which can be combined and freely repurposed to obtain complex compositional architectures. TorchDyn further offers step-by-step tutorials and benchmarks designed to guide researchers and contributors.
BibTeX
@misc{poli2020torchdyn,
  author = {Michael Poli and Stefano Massaroli and Atsushi Yamashita and Hajime Asama and Jinkyoo Park},
  title = {{TorchDyn: A Neural Differential Equations Library}},
  year = {2020},
  eprint = {2009.09346},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.09346},
}
Figure