Stiffness-aware neural network for learning Hamiltonian systems

Senwei Liang, Zhongzhan Huang, Hong ZhangICLR 2022 · OpenReview

In the quest to bridge deep learning and classical mechanics, researchers have made significant strides in learning Hamiltonian systems—physical systems where total energy is conserved, such as a swinging pendulum or orbiting planets. However, standard neural network models often fall apart when the physics gets "stiff."

A paper from ICLR 2022, "Stiffness-aware Neural Network for Learning Hamiltonian Systems," introduces a clever way to handle these volatile moments. The proposed method ensures that neural networks don't just learn smooth, predictable orbits, but also capture the high-stakes "close encounters" that define complex physical dynamics.

Executive Summary

The authors propose the Stiffness-Aware Neural Network (SANN), which uses a novel metric called the Stiffness-Aware Index (SAI) to identify volatile "stiff" moments in physical data. By resampling these rare events and using adaptive time-integration during training, SANN significantly outperforms state-of-the-art models in stability and energy conservation for complex systems like the three-body problem.

The Challenge: Stiff Dynamics

Most neural networks have an implicit bias toward learning smooth, low-frequency functions. In physics, this is a major problem when dealing with stiff dynamics.

Imagine two planets orbiting each other. Most of the time, they move in smooth, predictable curves. But if they pass very close to one another, the gravitational forces spike instantly. This is a "stiff" moment. Standard Hamiltonian Neural Networks (HNNs) often fail here; they "smooth over" these spikes, leading to massive errors where the planets fly off into deep space and the system's total energy is no longer conserved.

Standard HNNs fail to conserve energy during "stiff" close encounters in the three-body problem.

Furthermore, these stiff moments are rare. In a typical three-body simulation, they might occur during only 4% of the total time. This means a standard training process will naturally ignore them in favor of the easier, "nonstiff" data.

The SANN Approach

The authors introduce SANN, which treats stiff and nonstiff data differently through three key innovations:

  1. Stiffness-Aware Index (SAI): Because calculating the theoretical "Stiffness Index" requires knowing the underlying physics equations (which the model is trying to learn), the authors created SAI. It is a simple proxy calculated directly from the training data: the ratio of the rate of change of the system's state to the magnitude of the state itself.
  2. Resampling: To solve the "imbalanced data" problem, SANN replicates the stiff intervals in the training set. This forces the neural network to prioritize learning the difficult, high-force interactions that it would otherwise ignore.
  3. Adaptive Integration: During training, SANN splits the data. For "nonstiff" intervals, it uses standard integration steps. For "stiff" intervals (identified by a high SAI), it uses a much finer "partition"—essentially taking many tiny steps to capture the rapid changes in the vector field.
The SANN architecture incorporates stiffness classification, resampling, and adaptive integration.

The model relies on a physics-informed architecture where the neural network approximates the Hamiltonian (energy) function, and the gradients of that function dictate the system's motion.

Experimental Validation and Results

The researchers evaluated SANN on notoriously difficult physical benchmarks, including the Billiard Model (a ball bouncing inside a ring with a stiff boundary layer) and the Three-Body Problem (chaotic interactions via Newtonian gravity). They also tested an N-Chain Pendulum to demonstrate efficacy on systems with inseparable Hamiltonians.

SANN demonstrated a massive leap in accuracy and "physicality" compared to standard HNNs and SRNNs (Symplectic Recurrent Neural Networks):

  • Billiard Model: SANN reduced the relative energy error by 56% compared to SRNN and a staggering 92% compared to HNN. Visually, the SANN orbits remained perfectly on track while others drifted into chaos.
  • Three-Body Problem: In long-term predictions, SANN's Mean Squared Error (MSE) was 63% lower than SRNN and 92% lower than HNN.
  • Energy Conservation: While other models saw their "conserved" energy values spike and fluctuate wildly during close encounters, SANN maintained a nearly flat energy profile, proving it had truly captured the underlying physical laws.

Key Insights

  • Stiffness is the "Aha!" Moment: The paper proves that SAI is a reliable proxy for the theoretical Stiffness Index. Ranking data by SAI effectively highlights the parts of the trajectory where the physics is most likely to break a standard model.
  • Spectral Bias Mitigation: By forcing the network to look at resampled, finely-integrated stiff data, SANN overcomes the natural tendency of neural networks to ignore high-frequency components.
  • Activation Functions Matter: The authors found that Rational activation functions (which look like fractions of polynomials) performed better than ReLU or Tanh. This makes intuitive sense, as many physical laws (like gravity) are defined by inverse-square relationships that rational functions can mimic more easily.

Limitations and Future Work

While SANN is a powerful step forward, the authors note a few limitations:

  • Coordinate Sensitivity: The value of SAI can change if you shift the coordinate system. While the authors mitigate this by using rankings rather than absolute values, a truly coordinate-invariant metric would be ideal.
  • Noise Sensitivity: Like most physics-informed models, SANN struggles when the training data is highly noisy, which can cause the learned dynamics to become unstable.
  • Generalization: While tested on 2D and 3D systems, scaling this approach to extremely high-dimensional systems (like molecular dynamics with thousands of atoms) remains an open challenge.

For researchers in scientific machine learning, SANN provides a robust blueprint for how to handle the "edge cases" of the physical world that standard deep learning often misses.

Summary updated

Abstract

We propose stiffness-aware neural network (SANN), a new method for learning Hamiltonian dynamical systems from data. SANN identifies and splits the training data into stiff and nonstiff portions based on a stiffness-aware index, a simple, yet effective metric we introduce to quantify the stiffness of the dynamical system. This classification along with a resampling technique allows us to apply different time integration strategies such as step size adaptation to better capture the dynamical characteristics of the Hamiltonian vector fields. We evaluate SANN on complex physical systems including a three-body problem and billiard model. We show that SANN is more stable and can better preserve energy when compared with the state-of-the-art methods, leading to significant improvement in accuracy.
BibTeX
@inproceedings{liang2022stiffness,
  author = {SENWEI Liang and Zhongzhan Huang and Hong Zhang},
  title = {{Stiffness-aware neural network for learning Hamiltonian systems}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=uVXEKeqJbNa},
}
Figure