Many foundational concepts in machine learning are naturally conceived in continuous time. We model momentum optimization as a heavy ball rolling down a loss landscape, view neural ordinary differential equations (Neural ODEs) as continuous trajectories, and analyze zero-sum games as continuous dynamical systems. Continuous-time formulations offer elegant mathematical tools, such as conservation laws and Lyapunov stability, making them highly attractive for theoretical design.
However, digital computers cannot execute continuous time; we must discretize. The standard approach is to apply naive numerical integration schemes like the Forward Euler method. Unfortunately, this translation often breaks the very properties that made the continuous model attractive. Naive discretization can introduce artificial energy drift, destroy acceleration guarantees, and cause trajectories to spiral out of control.
Symplectic discretization solves this problem by preserving the underlying geometric structure of these dynamical systems when moving from continuous equations to discrete algorithms.
Why Naive Discretization Fails
To build intuition for why standard discretization fails, consider a simple physical system: an undamped harmonic oscillator (a pendulum swinging without friction). In continuous time, the pendulum's total energy is perfectly conserved. If you plot its position ($q$) and momentum ($p$) over time, the trajectory forms a perfectly closed loop in phase space.
If we discretize this system using the standard Forward Euler method, each discrete step computes the next state using the current tangent vector. Because the true trajectory is curved, this linear step pushes the state slightly outward from the true orbit. Over time, these tiny errors accumulate, causing the system to gain artificial energy and spiral outward to infinity. Conversely, using Backward Euler causes the trajectory to spiral inward, artificially draining energy.
Forward Euler (Spirals Outward) Symplectic Euler (Preserves Orbit)
▲ ▲
│ ┌─►─┐ │ ┌─►─┐
│ ▲ ▼ │ ▲ ▼
│ ┌┘ │ │ └───◄─┘
│ │ ▼ │
└─┴──────┴──► └─┴────────►
In machine learning, this numerical artifact manifests as algorithmic instability. As Muehlebach et al. demonstrate in Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectives, standard numerical integration methods introduce artificial energy growth that destabilizes continuous-time optimization models. Instead of smoothly minimizing a loss function, the discrete iterations can oscillate violently or diverge entirely.
The Mechanism of Symplectic Integration
Symplectic integrators are specialized numerical schemes designed to preserve phase space volume (a property known as symplecticity). Instead of updating position and momentum simultaneously using current values, a basic symplectic integrator (such as Symplectic Euler) updates them sequentially:
$$ q_{k+1} = q_k + \Delta t \cdot f(p_k) $$ $$ p_{k+1} = p_k + \Delta t \cdot g(q_{k+1}) $$
Notice that the momentum update uses the newly computed position $q_{k+1}$ rather than the old $q_k$.
Why does this minor change work? While a symplectic integrator does not conserve the exact energy (Hamiltonian) of the original continuous system, it exactly conserves a slightly perturbed energy function known as a shadow Hamiltonian. Because the discrete steps are bound to the level sets of this shadow Hamiltonian, the system cannot drift to infinity or collapse inward. The discrete trajectory is mathematically constrained to remain stable over long horizons.
Preserving Acceleration in Optimization
This geometric preservation is crucial for translating continuous-time optimization proofs into practical, discrete algorithms. For example, Nesterov’s accelerated gradient descent can be modeled as a continuous-time differential equation. As Shi et al. show in Acceleration via Symplectic Discretization of High-Resolution Differential Equations, explicit discretization often fails to maintain acceleration, whereas a semi-implicit symplectic approach ensures that the discrete algorithm inherits the exact convergence characteristics of the continuous dynamical system.
This framework has been extended across several optimization domains:
- Time-Varying Systems: To handle acceleration methods where parameters evolve over time, Goto demonstrates in Fast symplectic integrator for Nesterov-type acceleration method how non-autonomous (time-dependent) optimization dynamics can be lifted into higher-dimensional Hamiltonian spaces to preserve stability via geometric integration.
- Lyapunov Stability: Suh et al. utilize symplectic Euler discretization in Continuous-Time Analysis of Accelerated Gradient Methods via Conservation Laws in Dilated Coordinate Systems to translate continuous conservation laws into discrete updates that outperform classical methods.
- Non-Convex Landscapes: When navigating complex engineering landscapes, Bolten et al. apply symplectic integrators to dissipative Hamiltonian flows in Non-convex shape optimization by dissipative Hamiltonian flows, preventing numerical errors from injecting artificial energy and allowing optimization paths to reliably bypass local minima.
- Practical Tuning: To make these methods viable for deep learning, Duruisseaux et al. introduce momentum restarting and temporal looping strategies in Practical Perspectives on Symplectic Accelerated Optimization, transforming highly oscillatory symplectic trajectories into robust algorithms that rival industry standards.
- Minimax Games: In adversarial settings like zero-sum games, standard simultaneous gradient descent-ascent causes player strategies to spiral away from equilibrium. Bailey et al. prove in Finite Regret and Cycles with Fixed Step-Size via Alternating Gradient Descent-Ascent that simply switching to alternating updates functions as a symplectic integrator, bounding regret and forcing stable, cyclic behavior around equilibria.
Structure-Preserving Deep Learning Architectures
The necessity of symplectic discretization extends directly to deep neural network architectures that model temporal or continuous processes.
In the context of Neural ODEs, Zhu et al. reveal a fundamental limitation of standard training in On Numerical Integration in Neural Ordinary Differential Equations. When trained with standard explicit solvers, a Neural ODE does not learn the true continuous physical dynamics; instead, it learns a solver-specific, perturbed system. Without a symplectic integrator, the model is mathematically incapable of learning exact physical conservation laws (like energy preservation) from Hamiltonian data.
Training Data ──► Naive Solver ──────► Deviates from physical laws
Training Data ──► Symplectic Solver ──► Preserves exact conservation laws
Symplectic principles also offer solutions to classic training pathologies like vanishing and exploding gradients in Recurrent Neural Networks (RNNs). Rusch et al. introduce the UnICORNN architecture in UnICORNN: A recurrent model for learning very long time dependencies. By modeling hidden states as a system of undamped Hamiltonian oscillators and discretizing them via a symplectic Euler scheme, the network preserves phase space volume. This mathematical guarantee prevents gradients from exploding or vanishing, allowing the network to master sequences spanning thousands of time steps.
To scale these physical guarantees to complex systems, Yang et al. propose Hyperverlet: A Symplectic Hypersolver for Hamiltonian Systems. This approach pairs a traditional Velocity Verlet solver with a learned symplectic neural network corrector, ensuring long-term energy conservation that standard black-box neural networks fail to provide.
Broadening the Geometric Framework
Beyond optimization and neural architectures, structure-preserving discretization serves as a foundational bridge across control theory, reinforcement learning, and sampling:
- Optimal Control: From a theoretical perspective, Ohsawa et al. establish a complete framework in Discrete Hamilton-Jacobi Theory, showing that discrete-time evolutions preserving geometric phase space structure naturally generalize both the discrete Riccati equation and the Bellman equation.
- Reinforcement Learning: In practical RL, Lyu et al. develop Relativistic Adaptive Gradient Descent in Conformal Symplectic Optimization for Stable Reinforcement Learning. They demonstrate that preserving the geometric structure of the optimization phase space ensures predictable energy dissipation, drastically reducing the outlier states that cause RL agents to diverge.
- Particle Sampling: For particle-based sampling from complex distributions, Tan et al. implement symplectic Euler discretization in Accelerated Regularized Wasserstein Proximal Sampling Algorithms to maintain the geometric structure of deterministic particle trajectories over extended simulation periods.
- Large Time-Stepping: To allow for larger step sizes without losing physical invariants, Li et al. introduce NySALT in NySALT: Nyström-type inference-based schemes adaptive to large time-stepping, learning data-driven parameters for traditional integrators while strictly enforcing underlying symplectic constraints.
- Gradient Consistency: Finally, Hühnerbein et al. show in Learning Adaptive Regularization for Image Labeling Using Geometric Assignment that using symplectic numerical integration ensures that optimization and system discretization commute, providing exact gradients for stable model training.
By ensuring that the discrete algorithms respect the geometric invariants of their continuous counterparts, symplectic discretization provides a mathematically rigorous methodology for building stable, accelerated, and structure-preserving machine learning systems.
Key Foundational Work
Discrete Hamilton-Jacobi Theory (2011) — contributed 1 concept elements to this direction.
Acceleration via Symplectic Discretization of High-Resolution Differential Equations (2019) — contributed 1 concept elements to this direction.