On the Benefits of Memory for Modeling Time-Dependent PDEs

Ricardo Buitrago Ruiz, Tanya Marwah, Albert Gu, Andrej Risteski

Abstract

Data-driven techniques have emerged as a promising alternative to traditional numerical methods for solving partial differential equations (PDEs). These techniques frequently offer a better trade-off between computational cost and accuracy for many PDE families of interest. For time-dependent PDEs, existing methodologies typically treat PDEs as Markovian systems, i.e., the evolution of the system only depends on the "current state", and not the past states. However, distortion of the input signals—e.g., due to discretization or low-pass filtering—can render the evolution of the distorted signals non-Markovian. In this work, motivated by the Mori-Zwanzig theory of model reduction, we investigate the impact of architectures with memory for modeling PDEs: that is, when past states are explicitly used to predict the future. We introduce Memory Neural Operator (MemNO), a network based on the recent SSM architectures and Fourier Neural Operator (FNO). We empirically demonstrate on a variety of PDE families of interest that when the input is given on a low-resolution grid, MemNO significantly outperforms the baselines without memory, achieving more than 6 times less error on unseen PDEs. Via a combination of theory and experiments, we show that the effect of memory is particularly significant when the solution of the PDE has high frequency Fourier components (e.g., low-viscosity fluid dynamics), and it also increases robustness to observation noise.

Introduction

In the rapidly evolving field of scientific machine learning, neural operators have become a cornerstone for solving partial differential equations (PDEs) with greater efficiency than traditional numerical solvers. Most current state-of-the-art models operate under a Markovian assumption: they predict the next state of a system based solely on its current state.

However, this assumption often breaks down in real-world applications where we only have access to partial observations. Whether due to low-resolution sensors or inherent measurement noise, missing information—such as high-frequency physical oscillations—can make a system appear non-Markovian. Drawing from the Mori-Zwanzig theory of statistical mechanics, this research demonstrates that when only a subset of a system's variables is observed, the evolution of that subset depends on its history. By explicitly modeling this "memory," we can recover lost information and significantly improve predictive accuracy.

Methodology: The Memory Neural Operator (MemNO)

To bridge the gap between spatial dynamics and temporal history, we propose the Memory Neural Operator (MemNO). MemNO is a modular framework designed to interleave spatial operator layers with temporal sequence models.

  1. Spatial Modeling: The framework utilizes standard operators, such as the Fourier Neural Operator (FNO) or U-Net, to capture the complex spatial dependencies of the PDE.
  2. Temporal Memory: Between spatial passes, we insert a sequential layer—specifically the Structured State Space (S4) model. S4 is uniquely suited for this task due to its ability to handle long-range dependencies efficiently and its inherent bias toward continuous signals, which aligns well with physical systems.
  3. Hybrid Architecture: Our flagship implementation, S4FFNO, integrates a Factorized FNO with S4 to maintain a compressed, evolving representation of the system's history.
Architecture of the MemNO framework in 1D. The memory layer (green) is inserted between spatial neural operator layers to maintain a history of the system's evolution.

Experimental Setup and Datasets

We evaluated the MemNO framework across several classic time-dependent PDEs that represent various physical challenges:

  • Kuramoto-Sivashinsky (KS) Equation (1D): A chaotic system used to model fluid instabilities.
  • Burgers’ Equation (1D): A fundamental PDE for fluid mechanics and shock wave modeling.
  • Navier-Stokes Equation (2D): The standard for modeling incompressible fluid flow.

To rigorously test the limits of memoryless models, we developed a more challenging version of the KS dataset with lower viscosity. This introduces significant high-frequency Fourier modes that are particularly difficult to capture at low resolutions without historical context.

Key Insights and Results

The "Cutoff" Resolution

Our research identifies a critical resolution threshold for PDE modeling. Above this threshold (high resolution), Markovian models perform adequately. However, below this threshold, memory-augmented models like MemNO provide a massive accuracy boost. In these "lossy" regimes, memory allows the model to infer missing high-frequency dynamics from the history of observed low-frequency modes.

Performance Gains

  • 6x Error Reduction: On the Kuramoto-Sivashinsky equation at low resolution (32 points), S4FFNO achieved a 6x lower test error (nRMSE) compared to the memoryless FFNO baseline.
  • Scaling Limits: We found that simply increasing the parameter count or depth of Markovian models does not close the performance gap. The improvement is fundamentally tied to the temporal integration of history.
  • Robustness to Noise: Memory acts as a natural temporal filter. In 2D Navier-Stokes simulations with added Gaussian noise, MemNO-2D consistently outperformed baselines, maintaining roughly 50% lower error across various noise levels.

Conclusion and Future Work

This work establishes that memory is not just a luxury but a necessity for modeling PDEs from partial or noisy observations. By leveraging the Mori-Zwanzig theory, MemNO provides a robust framework for capturing the non-Markovian nature of distorted physical signals.

Future research will explore the application of newer sequence models like Mamba or xLSTM within this framework and investigate how memory handles more complex, structured noise patterns. As these solvers move toward real-world engineering and climate modeling, ensuring their robustness in low-data regimes will be paramount.

Summary updated

Abstract

Data-driven techniques have emerged as a promising alternative to traditional numerical methods for solving partial differential equations (PDEs). These techniques frequently offer a better trade-off between computational cost and accuracy for many PDE families of interest. For time-dependent PDEs, existing methodologies typically treat PDEs as Markovian systems, i.e., the evolution of the system only depends on the ``current state'', and not the past states. However, distortion of the input signals -- e.g., due to discretization or low-pass filtering -- can render the evolution of the distorted signals non-Markovian. In this work, motivated by the Mori-Zwanzig theory of model reduction, we investigate the impact of architectures with memory for modeling PDEs: that is, when past states are explicitly used to predict the future. We introduce Memory Neural Operator (MemNO), a network based on the recent SSM architectures and Fourier Neural Operator (FNO). We empirically demonstrate on a variety of PDE families of interest that when the input is given on a low-resolution grid, MemNO significantly outperforms the baselines without memory, achieving more than 6 times less error on unseen PDEs. Via a combination of theory and experiments, we show that the effect of memory is particularly significant when the solution of the PDE has high frequency Fourier components (e.g., low-viscosity fluid dynamics), and it also increases robustness to observation noise.
BibTeX
@inproceedings{ruiz2025benefits,
  author = {Ricardo Buitrago Ruiz and Tanya Marwah and Albert Gu and Andrej Risteski},
  title = {{On the Benefits of Memory for Modeling Time-Dependent PDEs}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=o9kqa5K3tB},
}
Figure