Semi-Implicit Neural Ordinary Differential Equations

Hong Zhang, Ying Liu, Romit MaulikAAAI 2025 · DOI

TL;DR

This paper introduces SINODE, a novel semi-implicit neural ODE framework that significantly improves training efficiency and stability for stiff differential equations. By partitioning the ODE's dynamics into linear and nonlinear components and using implicit-explicit (IMEX) integration, SINODE achieves faster training and handles challenging problems that are intractable for existing methods.

What Problem Does This Paper Address?

Neural Ordinary Differential Equations (ODEs) have emerged as a powerful tool for modeling continuous-depth networks and dynamical systems. However, training these models often faces a significant bottleneck: numerical stability. Explicit ODE solvers, commonly used for training, are only conditionally stable and require extremely small time steps for "stiff" problems – systems with vastly different time scales. This leads to computationally prohibitive training times due to the sheer number of function evaluations needed. While fully implicit methods offer better stability, they can suffer from convergence issues in their nonlinear solvers. This paper addresses the need for a method that combines the stability of implicit methods with the computational efficiency of explicit methods, particularly for stiff ODEs encountered in areas like graph learning and scientific machine learning.

Methods

The core innovation of this paper is the Semi-Implicit Neural ODE (SINODE) framework. SINODE tackles the ODE problem by:

  • Partitioning the Dynamics: The right-hand side of the ODE is split into a nonlinear part ($G(u)$) and a linear part ($H(u)$). This allows for differential treatment of these components.
  • Implicit-Explicit (IMEX) Integration: The nonlinear part ($G(u)$) is integrated explicitly, while the linear part ($H(u)$) is integrated implicitly. This is achieved using IMEX Runge-Kutta (IMEX-RK) methods, which are known for their favorable stability properties for stiff systems.
  • Differentiable IMEX Solver: A custom IMEX ODE solver is developed. Crucially, it employs a discrete adjoint approach for gradient computation. This method provides reverse-accurate gradients without the need for backpropagating through the entire ODE solver, significantly reducing memory consumption and computational cost compared to traditional backpropagation.
  • Efficient Linear Solvers: For the implicit part, which involves solving linear systems, SINODE leverages off-the-shelf linear solvers. The paper highlights the advantage of reusing LU factorizations when the linear part's Jacobian is constant across time steps or mini-batches, further boosting efficiency.

Datasets

The paper evaluates SINODE on the following datasets:

  • Graph Classification: Cora, CoauthorCS, and Photo datasets.
  • Time-Series Prediction:
    • Kuramoto-Sivashinsky (KS) Equation
    • Viscous Burgers Equation (VBE)

These datasets are chosen to represent problems that are inherently nonlinear, stiff, or chaotic.

Key Insights and Ideas

  • Partitioning is Key: Splitting the ODE dynamics into linear and nonlinear components is a surprisingly effective strategy for enabling semi-implicit integration. This allows the best of both worlds: explicit treatment for the potentially complex nonlinearities and implicit treatment for the stable linear dynamics.
  • IMEX Methods Shine for Stiffness: The paper strongly advocates for IMEX-RK methods as a superior choice for stiff ODEs in machine learning. Their inherent stability allows for much larger time steps than explicit methods, leading to dramatic speedups.
  • Discrete Adjoint for Efficiency: Moving from continuous adjoints to discrete adjoints for gradient calculation is crucial for practical implementation and efficiency, especially when dealing with discrete time-stepping schemes.
  • Leveraging Existing Solvers: The ability to use standard, highly optimized linear solvers (like those in PETSc) for the implicit part is a significant practical advantage, allowing SINODE to benefit from decades of numerical analysis research.

Key Results

  • Significant Speedups: SINODE with IMEX-RK methods achieved substantial speedups compared to classical neural ODEs with explicit methods. For example, in the Viscous Burgers Equation, SINODE with IMEX-RK2 was reported to be ~47 times faster than Dopri5 and ~33 times faster than Crank-Nicolson for reaching a loss of $10^{-3}$.
  • Enhanced Stability: SINODE demonstrated superior stability, allowing for time step sizes orders of magnitude larger than explicit methods (e.g., step size of 1 vs. 0.005). This enabled training on stiff problems that were intractable for explicit solvers.
  • Handling Extreme Stiffness: For very stiff cases, such as the KS equation with a grid size of 512, SINODE was able to complete training, while explicit methods required extremely small step sizes (on the order of $10^{-7}$) and fully implicit methods diverged.
  • Reduced Function Evaluations (NFEs): Table 1 and Table 2 show that SINODE consistently requires significantly fewer Number of Function Evaluations (NFEs) per epoch compared to explicit and fully implicit methods, directly translating to faster training.
  • Competitive Performance on Graph Tasks: SINODE performed competitively on graph classification tasks, demonstrating its versatility.

Limitations and Future Work

  • Not for Non-Stiff Problems: The paper acknowledges that SINODE is not intended for non-stiff problems, where classic neural ODEs with explicit methods might be more efficient.
  • Direct Solvers and Varying Step Sizes: When direct linear solvers are used, varying step sizes during training can degrade SINODE's performance because the Jacobian cannot be reused across time steps. However, this is mitigated when using matrix-free iterative linear solvers.
  • Future Work: The authors suggest that SINODE's ability to leverage off-the-shelf linear solvers opens avenues for adapting well-established techniques from other fields. Further exploration of different partitioning strategies and IMEX schemes could also yield benefits.

Summary updated

BibTeX
@inproceedings{zhang2025semi,
  author = {Hong Zhang and Ying Liu and Romit Maulik},
  title = {{Semi-Implicit Neural Ordinary Differential Equations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34398},
}
Figure