Sparse Symplectically Integrated Neural Networks

Daniel M. DiPietro, Shiying Xiong, Bo Zhu

Predicting the behavior of complex physical systems—from a swinging pendulum to chaotic stellar motion—is a classic challenge in machine learning. Currently, researchers face a frustrating trade-off. Black-box models (like standard neural networks) are excellent at curve-fitting but offer zero insight into why a system moves the way it does, often violating fundamental laws of physics like energy conservation over time. Conversely, symbolic models attempt to find human-readable equations but struggle with noisy data and the error-prone process of calculating derivatives.

Sparse Symplectically Integrated Neural Networks (SSINNs) bridge this gap. They represent a new class of neural networks designed to learn the underlying "source code" of physical systems. By combining sparse symbolic regression with high-order symplectic integration, SSINNs are interpretable, physically grounded, and highly robust to noise.

How SSINNs Work

The core innovation of SSINNs relies on three integrated components:

  1. Hamiltonian Formalism: Instead of learning raw velocities, the model learns the Hamiltonian ($H$)—a single scalar function representing the total energy of the system. All equations of motion can be derived directly from this function.
  2. Sparse Regression via Function Spaces: Rather than using deep, opaque layers of weights, the network consists of a library of mathematical basis terms (e.g., $q^2, \sin(q), p^3$). During training, L1 regularization "zeros out" irrelevant terms, leaving only the essential pieces of the governing equation.
  3. Symplectic Integration: This is the architecture's secret weapon. During training, the model uses a high-accuracy numerical integrator to predict the next state. Because the integrator is symplectic, it mathematically guarantees that the system's geometric structure—and thus its energy conservation—is baked into the learning process.
SSINN Architecture

By embedding a symplectic integrator directly into the training loop, the model is forced to find a Hamiltonian that is stable over time. This acts as a powerful inductive bias, preventing the model from "cheating" with physically impossible short-term solutions. Furthermore, because SSINNs use integration rather than differentiation (which amplifies noise), they are significantly more robust to jittery, real-world data.

Performance and Key Results

The authors evaluated SSINNs on four classical Hamiltonian systems: the chaotic Hénon-Heiles system, nonlinearly coupled oscillators, a complex mass-spring system, and a simple pendulum. They tested these using both clean simulated data and noisy data to mimic real-world sensors.

The results demonstrate massive improvements over traditional approaches:

  • Extreme Parameter Efficiency: Because the model learns a concise symbolic equation rather than relying on massive weight matrices, an SSINN can operate with as few as 32 parameters. In contrast, state-of-the-art black-box models like Symplectic Recurrent Neural Networks (SRNNs) often require over 1,000,000 parameters to achieve worse results.
  • Superior Accuracy and True Discovery: On chaotic systems, SSINNs outperformed SRNNs by one to three orders of magnitude in prediction error. Remarkably, SSINNs successfully recovered the exact governing equations from as few as 200 noisy data points, learning coefficients to $10^{-6}$ precision on clean data.
  • Perfect Energy Conservation: While black-box models eventually drift—losing or gaining energy over long trajectories—SSINNs maintain near-perfect energy conservation.
Trajectory and Energy Conservation Comparison

Limitations and Future Directions

While highly effective, SSINNs have a few current limitations. First, they rely on a user-defined basis; a human must provide the library of functions. If the true physics requires a function not in the library, the model can only approximate it. Second, the architecture assumes the Hamiltonian is separable ($H = T(p) + V(q)$), which covers most classical mechanics but excludes systems with magnetic fields. Finally, the model requires data to be in canonical position and momentum coordinates.

Future work could address these constraints by using autoencoders to automatically learn canonical coordinates from raw video or sensor data, further expanding the ability of SSINNs to discover the hidden laws of physics from observation alone.

Summary updated

Abstract

We introduce Sparse Symplectically Integrated Neural Networks (SSINNs), a novel model for learning Hamiltonian dynamical systems from data. SSINNs combine fourth-order symplectic integration with a learned parameterization of the Hamiltonian obtained using sparse regression through a mathematically elegant function space. This allows for interpretable models that incorporate symplectic inductive biases and have low memory requirements. We evaluate SSINNs on four classical Hamiltonian dynamical problems: the Hénon-Heiles system, nonlinearly coupled oscillators, a multi-particle mass-spring system, and a pendulum system. Our results demonstrate promise in both system prediction and conservation of energy, often outperforming the current state-of-the-art black-box prediction techniques by an order of magnitude. Further, SSINNs successfully converge to true governing equations from highly limited and noisy data, demonstrating potential applicability in the discovery of new physical governing equations.
BibTeX
@inproceedings{dipietro2020sparse,
  author = {Daniel M. DiPietro and Shiying Xiong and Bo Zhu},
  title = {{Sparse Symplectically Integrated Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure