Forecasting of Spatio-temporal Chaotic Dynamics with Recurrent Neural Networks: a comparative study of Reservoir Computing and Backpropagation Algorithms

Pantelis R. Vlachas, Jaideep Pathak, Brian R. Hunt, Themistoklis P. Sapsis, Michelle Girvan, Edward Ott, Petros Koumoutsakos

Abstract

This study examines the efficiency of Recurrent Neural Networks (RNNs) in forecasting the spatiotemporal dynamics of high-dimensional and reduced-order complex systems. We compare Reservoir Computing (RC) and Backpropagation Through Time (BPTT) for gated network architectures, highlighting the advantages and limitations of each. Using the Lorenz-96 and Kuramoto-Sivashinsky (KS) equations as benchmarks, we find that RC outperforms BPTT in terms of predictive performance and training speed when full state dynamics are available. However, for reduced-order data, gated RNNs (LSTM and GRU) trained via BPTT demonstrate superior robustness and forecasting ability. Furthermore, this study quantifies for the first time the Lyapunov Spectrum of the KS equation using BPTT, achieving accuracy comparable to RC.

Introduction

Predicting the evolution of chaotic systems—such as atmospheric dynamics, fluid turbulence, or financial markets—is a fundamental challenge in computational science. Because small errors in these systems grow exponentially, traditional numerical solvers require exact physical equations and high computational costs. Data-driven RNNs offer a powerful alternative, but the field is currently divided between two paradigms: Reservoir Computing (RC), which utilizes fixed internal weights for speed, and Backpropagation Through Time (BPTT), which optimizes all weights for flexibility. This paper provides a systematic comparison of these methods, particularly regarding their scalability to high-dimensional spatial data and performance on partial observations.

Architectural Comparison

The researchers evaluated four distinct architectures to understand the trade-offs between training efficiency and model robustness.

Reservoir Computing (RC): These networks, specifically Echo State Networks, utilize a large, fixed, and random internal reservoir. Only the output layer is trained, making the process exceptionally fast.

Schematic of a Reservoir Computing (RC) cell.

Gated RNNs (LSTM & GRU): These architectures are designed to mitigate the "vanishing gradient" problem during BPTT training. By using gating mechanisms to control information flow, they can capture long-term dependencies more effectively than standard RNNs, though they require significantly more training time.

Architectures of (c) LSTM and (d) GRU cells used in BPTT training.

Methodology and Parallel Strategy

To handle high-dimensional systems like the 512-node Kuramoto-Sivashinsky (KS) grid, the authors implemented a parallel "divide and conquer" strategy. Instead of a single monolithic network, an ensemble of smaller RNNs was used. Each network predicts a local spatial region while communicating with neighbors to capture cross-spatial correlations. This approach allows the models to scale to dimensions that would otherwise exceed GPU memory limits.

Benchmarks included:

  • Lorenz-96: A 40-variable system simulating atmospheric dynamics.
  • KS Equation: A partial differential equation modeling turbulence, discretized to 512 nodes.
  • Reduced-Order Observables: Using Singular Value Decomposition (SVD), the authors created datasets where models only see the most energetic components, mimicking real-world sensor limitations.

Key Results and Insights

Full-State vs. Reduced-Order Performance

When the full state of the system is visible, RC is highly effective, achieving a Valid Prediction Time (VPT) of ~2.3 Lyapunov times in Lorenz-96 benchmarks. However, RC proved brittle when dealing with reduced-order data, frequently diverging. In contrast, BPTT-trained LSTMs and GRUs remained stable, maintaining accuracy for nearly 2 Lyapunov times where RC failed within 1.

Training Efficiency and Memory

RC training is orders of magnitude faster; for the KS equation, RC trained in seconds while BPTT required hours. However, RC often requires massive reservoirs (thousands of nodes) to match the accuracy of a small LSTM, leading to significantly higher RAM requirements during inference.

Capturing the Signature of Chaos

A major contribution of this work is the demonstration that GRUs trained via BPTT can accurately reconstruct the Lyapunov Spectrum—the mathematical fingerprint of a chaotic system's stability. This matches the performance of RC and establishes BPTT-based models as a potent framework for learning complex dynamics.

Conclusion

While RC remains the "speed king" for full-state predictions, gated RNNs trained via BPTT are significantly more robust for reduced-order systems. The study suggests that future work should explore hybrid approaches—perhaps using RC-based heuristics to initialize weights before fine-tuning with BPTT—to combine the speed of reservoirs with the robustness of deep learning.

Summary updated

Abstract

We examine the efficiency of Recurrent Neural Networks in forecasting the spatiotemporal dynamics of high dimensional and reduced order complex systems using Reservoir Computing (RC) and Backpropagation through time (BPTT) for gated network architectures. We highlight advantages and limitations of each method and discuss their implementation for parallel computing architectures. We quantify the relative prediction accuracy of these algorithms for the longterm forecasting of chaotic systems using as benchmarks the Lorenz-96 and the Kuramoto-Sivashinsky (KS) equations. We find that, when the full state dynamics are available for training, RC outperforms BPTT approaches in terms of predictive performance and in capturing of the long-term statistics, while at the same time requiring much less training time. However, in the case of reduced order data, large scale RC models can be unstable and more likely than the BPTT algorithms to diverge. In contrast, RNNs trained via BPTT show superior forecasting abilities and capture well the dynamics of reduced order systems. Furthermore, the present study quantifies for the first time the Lyapunov Spectrum of the KS equation with BPTT, achieving similar accuracy as RC. This study establishes that RNNs are a potent computational framework for the learning and forecasting of complex spatiotemporal systems.
BibTeX
@inproceedings{vlachas2020forecasting,
  author = {Pantelis R. Vlachas and Jaideep Pathak and Brian R. Hunt and Themistoklis P. Sapsis and Michelle Girvan and Edward Ott and Petros Koumoutsakos},
  title = {{Forecasting of Spatio-temporal Chaotic Dynamics with Recurrent Neural Networks: a comparative study of Reservoir Computing and Backpropagation Algorithms}},
  booktitle = {Neural Networks},
  year = {2020},
  url = {https://openreview.net/forum?id=ltIhtbJlBeM},
}
Figure