Lyapunov Spectral Analysis for Neural Network Dynamics

19 concepts16 papers1 proposal

When we train a recurrent neural network (RNN) or observe a circuit of spiking neurons in the brain, we are looking at a high-dimensional dynamical system. As inputs flow through these networks, their internal states trace out intricate trajectories in phase space. Understanding whether these trajectories are stable, chaotic, or collapsing is critical: it determines whether an artificial network can retain memories without exploding gradients, or whether a biological network can process sensory information reliably without dissolving into epileptic seizures.

To peer inside this black box, researchers use Lyapunov spectral analysis. Instead of looking at a single trajectory, this approach examines how an infinitesimal perturbation—a tiny "nudge" to the network's state—evolves over time. By measuring the exponential rates at which these perturbations expand or contract along different geometric directions, the Lyapunov spectrum provides a rigorous mathematical fingerprint of a network's computational capacity.

The Geometry of Perturbations

Imagine a sphere of initial network states centered around a baseline trajectory. As the system evolves in time, this sphere deforms into an ellipsoid. The rate of growth or contraction of the ellipsoid's principal axes is governed by the network's Lyapunov Exponents (LEs).

Mathematically, if we perturb a hidden state $h_t$ by a tiny vector $\delta h_t$, the next state's perturbation is given by the linearization of the network's transition function:

$$\delta h_{t+1} \approx J_t \delta h_t$$

where $J_t$ is the Jacobian matrix of the hidden state transition at time $t$. Over $T$ timesteps, the cumulative perturbation is multiplied by a sequence of Jacobians:

$$\delta h_T \approx \left( \prod_{t=1}^T J_t \right) \delta h_1$$

The Lyapunov exponents are the logarithms of the eigenvalues of this long-term product matrix as $T \to \infty$.

  Stable (λ < 0)             Edge of Chaos (λ ≈ 0)         Chaotic (λ > 0)
    ⊙ ───> •                       ⊙ ───> ⊙                  ⊙ ───> ───⊙───>
Perturbations shrink.         Information persists.     Perturbations explode.
Memory is lost.               Optimal computation.      System becomes unpredictable.

If the largest exponent ($\lambda_{\max}$) is positive, nearby trajectories diverge exponentially, meaning the system is chaotic and highly sensitive to initial conditions. If it is negative, trajectories converge toward a shared path, meaning the system eventually forgets its initial state.

Tracking Information Flow in Recurrent Networks

In machine learning, calculating this spectrum helps solve the notorious problem of vanishing and exploding gradients. Because backpropagation through time relies on the exact same chain rule of Jacobians as forward perturbation tracking, the Lyapunov spectrum directly mirrors gradient behavior.

Historically, practitioners relied on static metrics like the spectral radius (the largest eigenvalue of the raw weight matrix) to tune architectures. However, as shown by Jin (2014) in the context of reservoir computing, static metrics fail to capture the true behavior of an active network because they ignore actual neuron firing patterns. Instead, activity-dependent metrics like Lyapunov exponents serve as far more accurate gauges of a network's processing capabilities.

Furthermore, real-world RNNs are non-autonomous dynamical systems—they are constantly driven by external time-series inputs. Vogt et al. (2022) formalized this by tracking the Jacobian across successive timesteps under input drive. They discovered a remarkable property: the Lyapunov spectrum of an RNN stabilizes incredibly early during training—often within the first few epochs.

This early convergence means we do not have to wait for a model to finish training to know if it will succeed. As developed in the AeLLE (Autoencoder for Lyapunov Latent Embedding) framework by Vogt et al. (2024), the full spectrum of Lyapunov exponents acts as a robust diagnostic signature. It predicts final task performance far better than standard hyperparameters, capturing the underlying nonlinear dynamics that drive convergence and generalization.

The Analytical Frontier in Spiking Networks

While calculating Jacobians is straightforward in continuous RNNs, biological brains and neuromorphic hardware rely on Spiking Neural Networks (SNNs). Spikes are discontinuous, all-or-nothing events. If you nudge a spiking neuron's membrane potential slightly, its trajectory doesn't just shift continuously—it fires its spike slightly earlier or later. This temporal shift alters the input timing for every downstream neuron, creating a cascade.

To calculate the Lyapunov spectrum for these systems, researchers had to reinvent how we track perturbations across discrete events. Jahnke et al. (2009) and Engelken et al. (2013) resolved this by deriving an analytical expression for the single-spike Jacobian. Instead of tracking continuous state coordinates, this matrix maps how a small perturbation in the timing of a spike propagates to subsequent spiking events, explicitly incorporating the network's coupling matrix and topology.

This framework allowed neuroscientists to settle a long-standing debate about the "balanced state"—the regime in cortical circuits where excitatory and inhibitory inputs cancel each other out on average, leaving highly irregular, noisy spiking activity. Is this irregularity driven by deterministic chaos or external noise?

Using Lyapunov spectral analysis, the answers proved nuanced:

  • Jahnke et al. (2009) demonstrated that purely inhibitory networks with delays actually converge to stable periodic orbits ($\lambda_{\max} < 0$), proving that chaos is not a prerequisite for irregular neural firing.
  • Gu et al. (2018) showed that current-based integrate-and-fire networks under Poisson drive remain non-chaotic, meaning their irregular trajectories are stable, robust features rather than a consequence of extreme sensitivity to initial conditions.
  • Conversely, introducing specific biophysical properties can shatter this stability. Manz et al. (2019) showed that adding even a single "anti-leaky" neuron (which accelerates toward its threshold) can inject full-blown chaos into an otherwise stable network, altering its covariant Lyapunov vectors and opening up new modes of spike-based computation.

By calculating the full spectrum across arbitrary network topologies, Engelken et al. (2013) and Engelken et al. (2023) demonstrated how structural motifs shape a network's dynamical entropy. This provides a direct link between the physical wiring diagram of a circuit and its capacity to compress or expand information.

Diagnosing and Guiding Deep Learning

Lyapunov analysis has expanded beyond recurrent sequences into the training dynamics of deep architectures and continual learning. Here, the concepts are applied in two distinct ways: analyzing the state space across deep layers, and analyzing the optimization trajectory in parameter space.

State Space Stability and Continual Learning

When networks are stacked deeply, their internal representations can degrade. Gallicchio et al. (2017) proved that stacking recurrent layers inherently pushes a network toward more chaotic regimes even before training begins, highlighting why deep architectures require careful initialization to avoid losing control of their state trajectories.

In continual learning—where a network must learn thousands of tasks sequentially—networks often suffer from "Loss of Plasticity" (LoP), becoming unable to adapt to new data. To diagnose why this happens, Wang et al. (2025) utilized Finite-Time Lyapunov Exponents (FTLEs). Unlike asymptotic LEs, FTLEs measure divergence over a bounded window, making them ideal for tracking non-equilibrium transitions.

Negative FTLE (λ < 0)                      Positive FTLE (λ > 0)
Representation Collapse                    Chaotic Boundaries
⊙ ───> ───> •                              ⊙ ───> ───> 💥
States squashed into a point;              States hyper-sensitive;
Network loses ability to differentiate.    Network forgets past tasks instantly.

The authors discovered that LoP is a dual-natured failure mode. Negative FTLEs signal representation collapse (where the network squashes distinct inputs into a single point), while highly positive FTLEs signal chaotic boundaries (where the network becomes hyper-sensitive and overwrites past knowledge). Monitoring these exponents allows algorithms to dynamically balance stability and plasticity.

Optimization Trajectories in Parameter Space

We can also conceptualize the training process itself as a dynamical system, where the coordinates are the network's weights and the velocity is determined by the optimizer.

The authors of Adversarial Data Augmentation... (2025) leveraged this perspective to improve domain generalization. They introduced LEAwareSGD, an optimization method that estimates Lyapunov exponents in parameter space using the loss function's Hessian and the learning rate. By actively guiding the optimization toward the "edge of chaos" ($\lambda \approx 0$), the algorithm prevents the model from getting trapped in overly rigid local minima, ensuring the learned features remain flexible enough to generalize to unseen domains.

Similarly, analyzing early training trajectories using spectral signatures can reveal long-term learning outcomes. Redman et al. (2024) utilized Koopman operator theory to show that the presence of unstable eigenvalues in the very first iterations of training serves as a predictive indicator for complex behaviors like grokking (delayed generalization), distinguishing models destined for high performance from those that will remain stagnant.

A Unified Toolkit

Whether applied to the millisecond-by-millisecond voltage fluctuations of a cortical circuit, the hidden states of an industrial LSTM, or the weight updates of a modern optimizer, Lyapunov spectral analysis operates on the same core principle. By formalizing how networks respond to infinitesimal changes, it translates the abstract geometry of phase space into actionable metrics for stability, memory, and learning health.

Key Foundational Work

How chaotic is the balanced state? (2009) — contributed 1 concept elements to this direction.
What Matters in Neuronal Locking? (1996) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_lyapunov_spectral_analysis_fd8b7511,
  title = {Lyapunov spectral analysis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lyapunov-spectral-analysis/rdr_fd8b7511f2984f8ab7d564ac5fa4ee25},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_lyapunov_spectral_analysis_fd8b7511_v1,
  title = {Lyapunov spectral analysis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lyapunov-spectral-analysis/rdr_fd8b7511f2984f8ab7d564ac5fa4ee25/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake