On the Benefits of Memory for Modeling Time-Dependent PDEs

Ricardo Buitrago, Tanya Marwah, Albert Gu, Andrej Risteski

In the realm of scientific computing, researchers are increasingly turning to "neural operators"—deep learning models capable of solving complex partial differential equations (PDEs) orders of magnitude faster than traditional numerical solvers. However, most of these models operate under a strict "Markovian" assumption: to predict the future state of a system, you only need to know its current state.

Recent research, "On the Benefits of Memory for Modeling Time-Dependent PDEs," argues that this assumption is often a critical flaw. By drawing on classical physics theory and modern sequence modeling, the authors demonstrate that explicitly "remembering" the past is essential when dealing with the messy, low-resolution, or noisy data typical of real-world engineering applications.

The Problem with "Memoryless" Models

Most state-of-the-art neural PDE solvers, such as the Fourier Neural Operator (FNO), are Markovian. They take the state of a system at time $t$ and predict the state at time $t+1$. This approach works beautifully if you have perfect, high-resolution data that fully captures the system's dynamics.

In reality, we rarely possess "perfect" data. Sensors introduce noise, and computational constraints frequently force us to observe systems at low resolutions. When information is lost in this way—a state known as partial observability—the system's underlying physics effectively becomes non-Markovian. To understand where a fluid is moving, you can no longer rely solely on a blurry, low-resolution snapshot of its current position; you need to know where it was to fill in the missing physical details.

Bridging Physics and Deep Learning

To address this, the authors bridge the gap between deep learning and classical physics using Mori-Zwanzig theory. This theory provides a mathematical framework for "model reduction," demonstrating that when you only observe a subset of a system's variables, a "memory term" naturally emerges to account for the unobserved, high-frequency parts.

Translating this theory into a practical architecture, the authors propose the Memory Neural Operator (MemNO). Instead of a simple feed-forward step, MemNO interleaves spatial layers with a temporal sequence model:

  • Spatial Dynamics: The model utilizes the Factorized Fourier Neural Operator (FFNO) to handle the complex spatial patterns inherent in PDEs.
  • Temporal Memory: It employs S4 (Structured State Space models) to maintain a compressed, continuous representation of all past states.

Unlike LSTMs or Transformers, S4 scales linearly with sequence length and is specifically designed to handle continuous signals, making it a natural fit for physical simulations.

Key Insights: The "Resolution Cutoff"

The most striking insight from the study is that memory provides a "regime-dependent" benefit. Memory acts as a compensator for missing high-frequency information. If a model cannot resolve fine-grained details in a single frame, it can reconstruct them by analyzing how those blurry shapes evolved over time.

At very high resolutions, Markovian models and MemNO perform similarly because the current state contains almost all necessary information. However, as resolution drops, Markovian models suffer a catastrophic drop in accuracy, while MemNO remains robust.

Key Results and Performance

The authors evaluated their approach on several challenging benchmarks, including the Kuramoto-Sivashinsky (KS) Equation, Burgers’ Equation, and 2D Navier-Stokes, specifically generating datasets with low viscosity to create highly complex, "wiggly" high-frequency solutions.

Table comparing nRMSE across different resolutions and models. S4FFNO (MemNO) significantly outperforms memoryless baselines at lower resolutions.
  • 6x Error Reduction: As shown in the results table, on the Kuramoto-Sivashinsky dataset at a low resolution of 32 points, the MemNO architecture (S4FFNO) achieved a massive reduction in test error compared to the standard FFNO (e.g., dropping from 0.187 to 0.031 for $\nu=0.125$).
  • Noise Resilience: In simulations with added observation noise, MemNO consistently outperformed all baselines. At high noise levels, the Markovian FFNO's error was often 50% higher than the memory-augmented version.
  • Computational Efficiency: Despite the addition of memory, the S4 layers remain computationally efficient. The authors demonstrate that an S4 memory layer requires less computation than an additional spatial FFNO layer, while providing substantially greater accuracy gains.

Limitations and Future Directions

While the results strongly advocate for memory-augmented solvers, the authors note areas for further exploration:

  • Complex Noise Profiles: The study primarily utilized i.i.d. Gaussian noise. Real-world sensor noise or aliasing effects might be more structured, potentially requiring tailored memory handling.
  • Alternative Sequence Models: While S4 performed best in this study, newer sequence models like Mamba or xLSTM could be integrated into the MemNO framework for potentially greater performance.
  • Long-term Rollouts: The current work focuses on predicting the next step given the past. Investigating how memory impacts the stability of very long-term rollouts (predicting hundreds of steps into the future) remains an open challenge.

By proving that "the past is prologue" in PDE modeling, this research provides a clear roadmap for building more resilient, accurate AI tools for weather forecasting, aerodynamics, and climate modeling.

Summary updated

Abstract

Data-driven techniques have emerged as a promising alternative to traditional numerical methods for solving PDEs. For time-dependent PDEs, many approaches are Markovian---the evolution of the trained system only depends on the current state, and not the past states. In this work, we investigate the benefits of using memory for modeling time-dependent PDEs: that is, when past states are explicitly used to predict the future. Motivated by the Mori-Zwanzig theory of model reduction, we theoretically exhibit examples of simple (even linear) PDEs, in which a solution that uses memory is arbitrarily better than a Markovian solution. Additionally, we introduce Memory Neural Operator (MemNO), a neural operator architecture that combines recent state space models (specifically, S4) and Fourier Neural Operators (FNOs) to effectively model memory. We empirically demonstrate that when the PDEs are supplied in low resolution or contain observation noise at train and test time, MemNO significantly outperforms the baselines without memory---with up to $6 \times$ reduction in test error. Furthermore, we show that this benefit is particularly pronounced when the PDE solutions have significant high-frequency Fourier modes (e.g., low-viscosity fluid dynamics) and we construct a challenging benchmark dataset consisting of such PDEs.
BibTeX
@inproceedings{buitrago2025benefits,
  author = {Ricardo Buitrago and Tanya Marwah and Albert Gu and Andrej Risteski},
  title = {{On the Benefits of Memory for Modeling Time-Dependent PDEs}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=o9kqa5K3tB},
}
Figure