Explicit and Effectively Symmetric Schemes for Neural SDEs

Daniil Shmelev, Cristopher SalviSubmission to ICLR 2026 · Rejected

TL;DR

Training Neural Stochastic Differential Equations (Neural SDEs) using reversible solvers offers $O(1)$ memory backpropagation without storing forward activations, but existing methods like Reversible Heun suffer from severe numerical instability. This paper introduces Explicit and Effectively Symmetric (EES) Runge–Kutta schemes for rough and stochastic differential equations, trading strict algebraic reversibility for high-order "near-reversibility." The result is an explicit solver with massive stability domains that trains reliably on stiff, high-volatility dynamics where existing reversible baselines diverge.

What Problem Does This Paper Address?

Backpropagating gradients through Neural SDEs usually faces a hard trade-off:

  1. Discretise-then-optimise: Standard automatic differentiation stores every intermediate step in memory, leading to prohibitive memory usage for deep trajectories.
  2. Optimise-then-discretise (Adjoint method): Achieves constant $O(1)$ memory by solving a backward adjoint SDE, but introduces numerical gradient mismatch errors and requires slower forward recomputations.
  3. Reversible solvers: Solvers that algebraically reverse their steps in time can reconstruct past states on the fly without storing intermediate computation graphs.

Unfortunately, explicit Runge–Kutta (RK) methods cannot be algebraically reversible. Previous workarounds—such as the Reversible Heun scheme—introduce auxiliary state variables to force reversibility. However, this severely destabilizes the solver; on linear test problems, Reversible Heun is stable only along a tiny segment of the imaginary axis ($[-i, i]$). In practice, training collapses whenever trajectories encounter stiff dynamics or high volatility.

Methods

The authors resolve this dilemma by relaxing the requirement of exact reversibility to effective reversibility within the framework of Rough Differential Equations (RDEs):

  • Effectively Symmetric Schemes ($\text{EES}_R$): Rather than strictly requiring the reverse step to perfectly invert the forward step ($\Phi_{-h} \circ \Phi_h = \text{id}$), an $\text{EES}(n, m)$ method matches the forward solution to order $n$ while recovering the reversed initial state up to a higher order $m \ge n$. Specifically, the authors focus on a 3-stage $\text{EES}_R(2, 5)$ scheme.
  • Rough Path Formulation and B-Series: By viewing SDEs as RDEs driven by rough paths, the authors use rooted-tree B-series expansions to analyze local and global convergence rates. They substitute intractable tree-iterated stochastic integrals with products of driving increments, enabling an explicit, simplified RK formulation.
  • Reverse Backpropagation (Algorithm 1): The authors design a stage-by-stage reverse autodiff algorithm that steps backward through internal RK stages, propagating gradients with respect to both state and network parameters without caching full trajectory graphs.

Datasets

The authors evaluate their methods on synthetic stochastic benchmark systems designed to stress-test stability:

  • High-Volatility Ornstein–Uhlenbeck (OU) Process: A 1D OU process driven by strong noise ($\sigma = 2$, $\nu = 0.2$).
  • High-Dimensional Stiff Geometric Brownian Motion (GBM): A 25-dimensional GBM featuring a stiff drift matrix with negative eigenvalues down to $-40$.
  • Fractional Brownian Motion (fBm): Used in the appendix to empirically confirm theoretical convergence rates for Hurst parameters $H \in \{0.4, 0.5, 0.6\}$.

(No real-world empirical datasets were used in the evaluation.)

Key Insights and Ideas

  • Near-reversibility unlocks classical RK stability: By permitting an $O(h^{m+1})$ error upon reversal instead of exact inversion, the method eliminates parasitic auxiliary states. Mean-square stability analysis shows that $\text{EES}_R(2, 5)$ matches or exceeds the broad stability regions of standard RK3 and RK4.
  • Stability beats exactness for gradient estimation: Even though $\text{EES}_R$ schemes reconstruct history with a tiny theoretical error, they achieve significantly lower gradient mean-squared error than "exact" reversible schemes because they prevent trajectory explosions.
  • Unified pathwise perspective: Treating SDEs through RDE theory abstracts away stochastic Itô/Stratonovich subtleties and directly connects the numerical scheme to deterministic B-series analysis.

Key Results

Experiments matched the total number of drift/diffusion function evaluations across solvers to ensure fair runtime comparisons:

  • High-Volatility OU Process:
  • $\text{EES}_R(2, 5)$ achieved a terminal Mean Squared Error (MSE) of $0.0582$ in $261.3\text{s}$.
  • Reversible Heun stalled at $1.0190$ ($368.2\text{s}$), while McCallum–Foster (MCF) Euler and MCF Midpoint achieved $1.3048$ and $1.1651$, respectively.
  • Stiff High-Dimensional GBM:
  • Reversible Heun, MCF Euler, and MCF Midpoint all diverged completely (MSE exploded).
  • $\text{EES}_R(2, 5)$ remained numerically stable across all 1,000 training epochs, reaching a final MSE of $1.18 \times 10^{-4}$ and maintaining low gradient MSE.

Limitations and Future Work

  • Higher Evaluation Cost per Step: $\text{EES}_R(2, 5)$ requires 3 drift and diffusion evaluations per step, compared to only 1 for Reversible Heun. When models are simple and step sizes are very small, Reversible Heun is computationally cheaper.
  • Fixed Step Sizes Only: The current formulation relies on fixed step sizes; adaptive step-size controllers or partitioned integrators for stiff systems remain open problems.
  • Scope of Applications: The current experiments are limited to synthetic latent SDE tasks; future work needs to validate $\text{EES}_R$ on large-scale architectures such as Neural CDEs, Jump SDEs, or continuous-time diffusion models.

Summary updated

Abstract

Backpropagation through (neural) SDE solvers is traditionally approached in two ways: discretise-then-optimise, which offers accurate gradients but incurs prohibitive memory costs due to storing the full computational graph (even when mitigated by checkpointing); and optimise-then-discretise, which achieves constant memory cost by solving an auxiliary backward SDE, but suffers from slower evaluation and gradient approximation errors. Algebraically reversible solvers promise both memory efficiency and gradient accuracy, yet existing methods such as the Reversible Heun scheme are often unstable under complex models and large step sizes. We address these limitations by introducing a novel class of stable, near-reversible Runge–Kutta schemes for neural SDEs. These Explicit and Effectively Symmetric (EES) schemes retain the benefits of reversible solvers while overcoming their instability, enabling memory-efficient training without severe restrictions on step size or model complexity. Through numerical experiments, we demonstrate the superior stability and reliability of our schemes, establishing them as a practical foundation for scalable and accurate training of neural SDEs.
BibTeX
@misc{shmelev2025explicit,
  author = {Daniil Shmelev and Cristopher Salvi},
  title = {{Explicit and Effectively Symmetric Schemes for Neural SDEs}},
  year = {2025},
  eprint = {2509.20599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.20599},
}
Figure