Fast-Slow Neural Networks for Learning Singularly Perturbed Dynamical Systems

Daniel A. Serino, Allen Alvarez Loya, J. W. Burby, Ioannis G. Kevrekidis, Qi Tang

Abstract

Singularly perturbed dynamical systems play a crucial role in climate dynamics and plasma physics. A powerful and well-known tool to address these systems is the Fenichel normal form, which significantly simplifies fast dynamics near slow manifolds through a transformation. However, this normal form is difficult to realize in conventional numerical algorithms. In this work, we explore an alternative way of realizing it through structure-preserving machine learning. Specifically, a fast-slow neural network (FSNN) is proposed for learning data-driven models of singularly perturbed dynamical systems with dissipative fast timescale dynamics. Our method enforces the existence of a trainable, attracting invariant slow manifold as a hard constraint. Closed-form representation of the slow manifold enables efficient integration on the slow time scale and significantly improves prediction accuracy beyond the training data. We demonstrate the FSNN on examples including the Grad moment system, two-scale Lorenz96 equations, and Abraham-Lorentz dynamics modeling radiation reaction of electrons.

Introduction: The Challenge of Timescale Separation

In complex physical systems—ranging from atmospheric currents to fusion plasmas—dynamics often unfold across vastly different timescales. These "singularly perturbed" systems feature fast processes that quickly dissipate and slow processes that dictate long-term evolution. Mathematically, trajectories in these systems are rapidly attracted to a "slow manifold," a lower-dimensional subspace where the interesting long-term physics reside.

While Fenichel theory provides a rigorous framework (the Fenichel normal form) to describe these dynamics, implementing it in traditional numerical algorithms is notoriously difficult. These systems are "stiff," requiring prohibitively small time steps to resolve fast scales even when only the slow behavior is of interest. Furthermore, discovering these manifolds from raw data without prior knowledge of the governing equations remains a significant hurdle in model reduction.

The FSNN Architecture

To address these challenges, the Fast-Slow Neural Network (FSNN) architecture is explicitly structured to mirror the Fenichel normal form, baking physical constraints directly into the model. Instead of a "black box" approach, the FSNN uses a structured pipeline to separate and solve multiscale dynamics.

The FSNN architecture integrates coordinate transformation via invertible networks with a physics-informed integrator based on the Fenichel normal form.

The architecture consists of several key components:

  1. Invertible Coupling Flow Network: An Invertible Neural Network (INN) learns a coordinate transformation that maps raw data into a latent space where fast and slow variables are explicitly separated.
  2. Negative Schur Form Network: This layer parameterizes the linear part of the dynamics. By constraining eigenvalues to the left-half of the complex plane, it mathematically guarantees that the fast dynamics are dissipative and stable.
  3. Low-Rank Bilinear Map: This component captures complex interactions between variables using a memory-efficient, low-rank approximation.
  4. Adaptive IMEX Integrator: The model employs Implicit-Explicit (IMEX) time-stepping. By treating stiff linear parts implicitly and non-stiff parts explicitly, it maintains numerical stability even with large time steps, effectively "stepping over" expensive fast-scale calculations.

Experimental Validation

The FSNN was evaluated across several demanding physical benchmarks, demonstrating its ability to learn underlying physics rather than just fitting curves:

  • Linear Grad Moment System: A 30-dimensional system from kinetic theory where the FSNN recovered physical eigenvalues with high precision (errors often $< 10^{-3}$).
  • Two-Scale Lorenz96: A standard benchmark for multiscale atmospheric dynamics. The FSNN demonstrated remarkable generalization, predicting dynamics up to $T=3000$ despite being trained on a window of only $T=3$.
  • Abraham-Lorentz Dynamics: This system, modeling electron radiation reaction, is notoriously unstable forward in time. By training on reverse-time data, the FSNN successfully captured the stable "guiding center" motion where standard numerical integrators typically fail due to exponential error growth.

Key Insights and Future Directions

A defining feature of the FSNN is its use of hard constraints. Rather than using loss functions to "encourage" stability, the architecture itself makes instability impossible for the fast scales through the Schur form. This structural guarantee makes the model exceptionally robust and allows it to discover attractors even from short trajectory snippets.

Furthermore, once the slow manifold is learned, the fast-scale equations can be discarded, creating a high-fidelity "closure" model that is orders of magnitude faster than the original system. Currently, the FSNN requires the dimension of the slow manifold to be known a priori. Future research aims to automate this dimensionality discovery and extend the framework to hyperbolic systems where some fast scales may be unstable.

Summary updated

Abstract

Singularly perturbed dynamical systems play a crucial role in climate dynamics and plasma physics. A powerful and well-known tool to address these systems is the Fenichel normal form, which significantly simplifies fast dynamics near slow manifolds through a transformation. However, this normal form is difficult to realize in conventional numerical algorithms. In this work, we explore an alternative way of realizing it through structure-preserving machine learning. Specifically, a fast-slow neural network (FSNN) is proposed for learning data-driven models of singularly perturbed dynamical systems with dissipative fast timescale dynamics. Our method enforces the existence of a trainable, attracting invariant slow manifold as a hard constraint. Closed-form representation of the slow manifold enables efficient integration on the slow time scale and significantly improves prediction accuracy beyond the training data. We demonstrate the FSNN on examples including the Grad moment system, two-scale Lorenz96 equations, and Abraham-Lorentz dynamics modeling radiation reaction of electrons.
BibTeX
@article{serino2025fast,
  author = {Daniel A. Serino and Allen Alvarez Loya and J. W. Burby and Ioannis G. Kevrekidis and Qi Tang},
  title = {{Fast-Slow Neural Networks for Learning Singularly Perturbed Dynamical Systems}},
  journal = {Journal of Computational Physics},
  year = {2025},
  doi = {10.1016/j.jcp.2025.114090},
}
Figure