Learning continuous-time PDEs from sparse data with graph neural networks

Valerii Iakovlev, Markus Heinonen, Harri Lähdesmäki

Researchers from Aalto University have developed a novel framework for learning Partial Differential Equations (PDEs) directly from "messy" real-world data. By combining Message Passing Neural Networks (MPNNs) with continuous-time solvers, their model can learn complex physical dynamics from sparse, noisy observations recorded at irregular intervals and unstructured spatial locations.

The Problem: Spatial and Temporal Rigidity

Partial Differential Equations are the mathematical backbone of physics, describing everything from fluid flow to heat dissipation. Traditionally, scientists derive these equations manually. Recently, machine learning has attempted to "discover" these equations directly from data.

However, most existing "neural PDE" solvers rely on Convolutional Neural Networks (CNNs), which creates two major bottlenecks:

  1. Spatial Rigidity: CNNs require data to be on a perfect, uniform grid. Real-world sensors (like weather stations or ocean buoys) are rarely placed so neatly.
  2. Temporal Rigidity: Most models use discrete-time approximations, assuming observations arrive at fixed intervals. They struggle if data is missing or if the time gap between snapshots varies.

The authors aim to bridge this gap by creating a model that treats space as a flexible graph and time as a continuum.

Methods: Bridging Graphs and Continuous Time

The paper introduces a general continuous-time differential model parameterized by Graph Neural Networks (GNNs). The approach relies on three key pillars:

  • Method of Lines (MOL): Instead of solving the PDE all at once, the spatial domain is discretized into a set of nodes (a graph). This transforms the PDE into a system of coupled Ordinary Differential Equations (ODEs).
  • Message Passing Neural Networks (MPNNs): The authors use MPNNs to learn the function governing the system's evolution. Because MPNNs operate on local neighborhoods and use relative distances, the model is position-invariant. It learns the underlying physical "rules" of interaction rather than just memorizing spatial coordinates.
  • The Adjoint Method: To train the model efficiently, they use the continuous-time adjoint method (popularized by Neural ODEs). This allows them to compute gradients through the ODE solver with a constant memory cost, regardless of how many steps the solver takes.
Spatial discretization using Delaunay triangulation

Datasets and Key Insights

The model was evaluated on synthetic data generated from three classic physical systems: the Convection-Diffusion Equation, the Heat Equation, and Burgers' Equations. The data was intentionally downsampled to create "sparse" scenarios, including irregular time steps and unstructured spatial grids generated via Delaunay triangulation.

Several key insights emerged from the experiments:

  • Continuous Time is a Superpower: Unlike discrete models that degrade when the time step between observations increases, this model remains stable. It learns the rate of change, allowing it to generalize to any arbitrary time point.
  • Graphs > Grids: By representing the domain as a graph, the model handles unstructured data naturally. Using Delaunay triangulation to define neighbors ensures that the graph captures the most relevant local spatial relationships.
  • Relative Positioning is Crucial: Encoding relative positions as edge features was vital. Without this spatial context, the model failed to learn directed dynamics like convection.
  • Zero-Shot Generalization: A model trained on a coarse grid (750 nodes) could still perform reasonably well when applied to a much finer grid (3000 nodes). This suggests the network truly learned the local physical laws rather than just a mapping of the training grid.
Zero-shot generalization across different spatial resolutions

Key Results

In benchmarks against existing models like PDE-Net and DPGN, the proposed framework demonstrated superior robustness. It maintained consistent accuracy even as the time interval between training snapshots quadrupled—a scenario where discrete models saw their error rates skyrocket.

Furthermore, the model achieved nearly identical performance on irregular time grids (where snapshots are perturbed by noise) as it did on regular ones. It also proved highly robust to additive noise in the training data, maintaining predictive power even when observations were significantly "jittered." On complex tasks like the Burgers' equations, the model accurately captured nonlinear propagation effects that simpler models missed.

Limitations and Future Work

While powerful, the framework has a few limitations. The current model assumes that while spatial locations are arbitrary, the state is observed at all those locations at every time point. While masking could potentially handle missing data at specific nodes, it was not the primary focus of this work.

Additionally, the model assumes the underlying physics are homogeneous everywhere in the domain (position invariance). It might struggle with systems containing position-dependent fields or varying material properties. Finally, while the Method of Lines is efficient for 2D, scaling to very large 3D simulations with millions of nodes will require further optimizations in graph processing and ODE solving.

Summary updated

Abstract

The behavior of many dynamical systems follow complex, yet still unknown partial differential equations (PDEs). While several machine learning methods have been proposed to learn PDEs directly from...
BibTeX
@inproceedings{iakovlev2021learning,
  author = {Valerii Iakovlev and Markus Heinonen and Harri Lähdesmäki},
  title = {{Learning continuous-time PDEs from sparse data with graph neural networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=rPiRlJyqfr},
}
Figure