On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools

Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, Guillaume Lajoie

Introduction

Recurrent Neural Networks (RNNs) are powerful tools for sequence modeling but are notoriously difficult to train. Small changes in initialization or hyperparameters can lead to vanishing or exploding gradients, causing model collapse. While empirical "rules of thumb" exist, a rigorous framework for measuring training stability has been lacking. Vogt et al. propose using Lyapunov Exponents (LEs) from dynamical systems theory to quantify how information propagates through an RNN, providing a precise "readout" of network stability.

The Challenge of RNN Stability

The primary hurdle in RNN training is the long-term dependency problem. For a network to learn from distant time steps, gradients must be preserved during backpropagation. Traditional analysis often treats networks as autonomous systems or simplifies data distributions. However, real-world RNNs are non-autonomous; their internal states are driven by complex, time-varying inputs. This paper bridges the gap between theory and practice by applying the Lyapunov spectrum to these driven systems.

Methodology: The Lyapunov Spectrum

The authors adapt a standard QR-decomposition-based algorithm to calculate the full Lyapunov spectrum for various RNN architectures, including Vanilla RNNs, GRUs, and LSTMs.

  1. Jacobian Derivation: Analytical expressions for the hidden state Jacobian are derived to describe how perturbations evolve.
  2. Orthonormal Tracking: The algorithm tracks a set of orthonormal vectors, applying the Jacobian and re-orthonormalizing via QR-decomposition at each step.
  3. Exponent Calculation: The "R" matrix provides expansion/contraction factors. Averaging their logarithms over time yields the Lyapunov Exponents ($\lambda_i$).
  4. Efficiency: This process is integrated into the training loop with minimal overhead (~4%), allowing for real-time monitoring.

Experimental Setup

The approach was evaluated on two distinct tasks:

  • CharRNN: Character-level language modeling using War and Peace, requiring the capture of low-dimensional, long-term temporal correlations.
  • CMU Mocap: Predicting human motion capture data, a higher-dimensional task involving complex sensor correlations.

Key Insights

  • Data-Driven Analysis: The "stochastic" Lyapunov spectrum accounts for actual input data, reflecting real-world network behavior.
  • Early Convergence: The spectrum stabilizes within the first few epochs, enabling early detection of training issues and faster hyperparameter tuning.
  • Task-Specific Signatures: Optimal performance correlates with specific spectral properties. For text modeling, the best models had a maximum exponent ($\lambda_{max}$) near zero—the "edge of stability." For motion capture, the mean of the exponents was a stronger predictor of success.

Results and Stability Regimes

The study identifies three distinct dynamical regimes based on the Lyapunov spectrum, which correlate strongly with validation loss.

Stability regimes and Lyapunov spectra: (a) Stable ($\lambda_{max} < 0$), (b) Marginally Stable ($\lambda_{max} \approx 0$), (c) Chaotic ($\lambda_{max} > 0$), and (d-e) corresponding spectral plots.
  • Stable Regime ($\lambda_{max} < 0$): Information contracts, leading to vanishing gradients and poor long-term memory.
  • Marginally Stable Regime ($\lambda_{max} \approx 0$): Information is preserved over time, ideal for learning complex dependencies.
  • Chaotic Regime ($\lambda_{max} > 0$): Information expands rapidly, causing exploding gradients and unstable training.

Conclusion

By integrating Lyapunov exponents into RNN analysis, this work moves toward a more principled approach to network design. These tools provide a window into the "black box" of RNN dynamics, offering a theoretically grounded method for hyperparameter tuning. Future work may extend this analysis to the internal gating mechanisms of LSTMs and the influence of input projection matrices on the driving force of data.

Summary updated

BibTeX
@article{vogt2022lyapunov,
  author = {Ryan Vogt and Maximilian Puelma Touzel and Eli Shlizerman and Guillaume Lajoie},
  title = {{On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools}},
  journal = {Frontiers Appl. Math. Stat.},
  year = {2022},
  doi = {10.3389/fams.2022.818799},
}
Figure