Fourier-RNNs for Modelling Noisy Physics Data

Vignesh Gopakumar, Stanislas Pamela, Lorenzo ZanisiarXiv 2023 · DOI · arXiv

Abstract

Modeling physical systems governed by Partial Differential Equations (PDEs) is critical for modern engineering, yet current deep learning approaches like Fourier Neural Operators (FNOs) often struggle with temporal memory and noise sensitivity. This paper introduces Fourier-RNNs (F-RNNs), a hybrid architecture that embeds Fourier layers within a recurrent framework. By leveraging the hidden state as a temporal filter in frequency space, F-RNNs demonstrate superior robustness and accuracy in predicting complex fluid dynamics from noisy data compared to standard FNOs and convolutional RNNs.

Introduction

Modeling physical systems described by Partial Differential Equations (PDEs) is a cornerstone of modern engineering. Recently, Fourier Neural Operators (FNOs) have emerged as a powerful tool because they map between infinite-dimensional function spaces rather than finite vectors, making them highly efficient at learning underlying physical operators.

However, FNOs possess two significant limitations. First, they are Markovian, typically predicting the next state based solely on the current state without memory of the distant past. Second, they exhibit high noise sensitivity; without temporal context, they struggle to distinguish between actual physical signals and random noise. While standard Recurrent Neural Networks (RNNs) provide memory via a hidden state, they operate in Euclidean space, which often fails to capture the complex, periodic behaviors inherent in physics data. To address these gaps, researchers from the UK Atomic Energy Authority have introduced the Fourier-RNN (F-RNN).

Methodology: The Fourier-RNN Cell

The core innovation of this work is the Fourier-RNN cell. In a standard RNN, the hidden state is updated using linear transformations. The F-RNN replaces these transformations with Fourier Layers to process data in the frequency domain.

Architecture of the Fourier-RNN (F-RNN) cell, showing the integration of Fourier layers for both input (x) and hidden state (h) updates.

As shown in the architecture above, each F-RNN cell utilizes Fourier layers that perform a Fourier Transform, apply a learnable weight matrix to the frequency modes, and then perform an Inverse Fourier Transform back to the spatial domain. By embedding these operators within a recurrent structure, the model learns to update its "memory" (hidden state) in Fourier space. This allows the hidden state to act as a temporal filter, using historical context to filter out noise that does not align with learned physical patterns.

Experimental Setup

The model was evaluated on three synthetic physics benchmarks with varying levels of Gaussian noise (0% to 25% variance):

  1. 2D Wave Equation: Modeling wave propagation over time.
  2. 2D Navier-Stokes (Laminar): Smooth fluid flow with kinematic viscosity $\nu=10^{-3}$.
  3. 2D Navier-Stokes (Turbulent): Chaotic fluid flow with lower viscosity $\nu=10^{-5}$.

The models were trained on 800 simulations and tested on 200 using an Nvidia V100 GPU, maintaining a parameter count (~4.2M) similar to a standard FNO for fair comparison.

Key Results and Discussion

The F-RNN demonstrated significant superiority in handling noisy data. In the Navier-Stokes laminar case with 25% noise, the F-RNN achieved a Mean Squared Error (MSE) of 0.001499, nearly ten times lower than the standard FNO's error of 0.01368. In the highly challenging turbulent regime, the F-RNN maintained stable predictions while Convolutional-RNNs (C-RNNs) failed to converge.

Qualitative comparison of model predictions across time steps (Initial, Middle, Final) for noisy physics data. F-RNN maintains clarity and matches the ground truth solution more closely than FNO or C-RNN.

Qualitative analysis (see above) reveals that FNO predictions become increasingly "grainy" and distorted as noise levels rise. In contrast, the F-RNN produces sharp, accurate predictions. This confirms the "aha moment" of the research: the hidden state effectively filters the noisy input against the system's learned trajectory. Furthermore, because it uses neural operators, the F-RNN learns the underlying physical operator, making it robust across different grid discretizations.

Conclusion and Future Work

Fourier-RNNs represent a major step toward building "digital twins" for complex physical systems, such as fusion reactors, where sensors provide noisy, real-time data. While this study utilized synthetic Gaussian noise, future work will focus on testing F-RNNs on experimental data from real-world systems. Additionally, further optimization of hidden state initialization could enhance the model's sensitivity to initial field distributions.

Summary updated

Abstract

Classical sequential models employed in time-series prediction rely on learning the mappings from the past to the future instances by way of a hidden state. The Hidden states characterise the historical information and encode the required temporal dependencies. However, most existing sequential models operate within finite-dimensional Euclidean spaces which offer limited functionality when employed in modelling physics relevant data. Alternatively recent work with neural operator learning within the Fourier space has shown efficient strategies for parameterising Partial Differential Equations (PDE). In this work, we propose a novel sequential model, built to handle Physics relevant data by way of amalgamating the conventional RNN architecture with that of the Fourier Neural Operators (FNO). The Fourier-RNN allows for learning the mappings from the input to the output as well as to the hidden state within the Fourier space associated with the temporal data. While the Fourier-RNN performs identical to the FNO when handling PDE data, it outperforms the FNO and the conventional RNN when deployed in modelling noisy, non-Markovian data.
BibTeX
@misc{gopakumar2023fourier,
  author = {Vignesh Gopakumar and Stanislas Pamela and Lorenzo Zanisi},
  title = {{Fourier-RNNs for Modelling Noisy Physics Data}},
  year = {2023},
  eprint = {2302.06534},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.06534},
}
Figure