Learning Neural PDE Solvers with Convergence Guarantees

Jun-Ting Hsieh, Shengjia Zhao, Stephan Eismann, Lucia Mirabella, Stefano Ermon

TL;DR

This paper introduces a hybrid approach to solving Partial Differential Equations (PDEs) that combines the speed of deep learning with the mathematical rigor of classical iterative solvers. By learning a neural "correction term" for existing solvers, the authors achieve a 2-3x speedup while providing theoretical guarantees that the model will always converge to the correct solution.

The Challenge with Existing Solvers

Partial Differential Equations (PDEs) are the backbone of physics and engineering, used to model everything from heat flow to quantum mechanics. Traditionally, these are solved using hand-crafted iterative methods (like the Jacobi or Multigrid methods) that slowly refine a candidate solution.

While these classical solvers are mathematically "safe"—meaning they are guaranteed to find the right answer eventually—they are often slow and general-purpose. Conversely, recent "pure" deep learning approaches can be incredibly fast but act as black boxes: they offer no guarantees of correctness, often fail when the geometry of the problem changes (e.g., moving from a square to a cylinder), and can produce physically impossible results. The challenge is: How can we make PDE solvers faster using machine learning without losing the convergence guarantees of classical mathematics?

A Hybrid Approach: Neural PDE Solvers

The authors propose a framework called Learning Neural PDE Solvers. Instead of replacing the classical solver, they build on top of it.

  1. The Base Iterator: They start with a standard linear iterative solver (like Jacobi). Every step, this solver takes a current guess $u$ and produces a new guess $u'$.
  2. The Neural Correction: They introduce a learned linear operator, $H$, parameterized as a deep convolutional neural network (either a simple ConvNet or a U-Net).
  3. The Update Rule: The new update becomes: $$\Phi_H(u) = \text{Standard\_Update}(u) + \text{Correction}(H, \text{Residual})$$
  4. Guarantees: Crucially, they design the correction term such that if the standard solver reaches the correct solution (the "fixed point"), the correction term becomes zero. This ensures that the neural network cannot "pull" the solver away from the true solution once it's found.
  5. Optimization: They train the network to minimize the error after a fixed number of iterations, effectively teaching the network to approximate the inverse of the PDE operator.

Datasets and Evaluation

The paper doesn't use standard computer vision datasets. Instead, it uses synthetic data generated from the 2D Poisson equation:

  • Training Set: The model is trained on a very simple 64x64 square domain with the Laplace equation ($\nabla^2 u = 0$) and random Dirichlet boundary conditions.
  • Evaluation/Generalization: To test if the model actually learned physics rather than just memorizing a shape, they evaluated it on larger grids ($256 \times 256$), complex geometries (L-shaped domains and Cylindrical domains with holes), and non-zero source terms (Poisson equation $\nabla^2 u = f$).
Evaluation Geometries

Key Insights

  • Fixed Point Preservation: By structuring the neural network as a modifier to a residual, the authors ensure that the neural solver inherits the fixed point of the classical solver. If the classical solver is "valid," the neural one is too.
  • Generalization through Locality: Because the model uses convolutional layers (which look at local neighborhoods), a model trained on a simple square can generalize to complex, curved, or sharp geometries it has never seen before.
  • Linearity is a Feature, Not a Bug: The authors use linear networks (no ReLU or Sigmoid). While this sounds counter-intuitive for "deep learning," it allows them to use linear algebra proofs to guarantee convergence via the spectral radius of the update matrix.
  • U-Net as a Learned Multigrid: The authors show that a U-Net architecture naturally mimics the "V-cycle" of the classical Multigrid method, allowing information to propagate across the grid much faster than local methods like Jacobi.

Key Results

  • Speedup: The neural solvers achieved a 2-3x speedup in terms of computational operations compared to state-of-the-art classical baselines.
  • Generalization: Despite being trained only on squares, the models converged perfectly on L-shapes and cylinders with no loss in performance.
  • Hardware Efficiency: Compared to the highly optimized FEniCS library (a standard tool in scientific computing), the neural solver was comparable on CPU but provided an additional 30x speedup when run on a GPU, thanks to the inherent parallelism of convolutional layers.
  • Convergence: In all test cases, the model successfully reached the ground truth solution, proving the theoretical guarantees hold in practice.

Limitations and Future Directions

  • Linear PDEs Only: The current framework is specifically designed for linear PDEs. Extending this to non-linear equations (like the Navier-Stokes equations for fluid dynamics) is a significant next step.
  • Dependence on Base Solver: The method requires an existing iterative solver to build upon. If a "good" base solver doesn't exist for a specific niche PDE, this method cannot be easily applied.
  • Training Complexity: While the solver is fast at inference, training requires backpropagating through iterative steps, which can be memory-intensive for very large problems.

Summary updated

Abstract

Partial differential equations (PDEs) are widely used across the physical and computational sciences. Decades of research and engineering went into designing fast iterative solution methods. Existing solvers are general purpose, but may be sub-optimal for specific classes of problems. In contrast to existing hand-crafted solutions, we propose an approach to learn a fast iterative solver tailored to a specific domain. We achieve this goal by learning to modify the updates of an existing solver using a deep neural network. Crucially, our approach is proven to preserve strong correctness and convergence guarantees. After training on a single geometry, our model generalizes to a wide variety of geometries and boundary conditions, and achieves 2-3 times speedup compared to state-of-the-art solvers.
BibTeX
@inproceedings{hsieh2019learning,
  author = {Jun-Ting Hsieh and Shengjia Zhao and Stephan Eismann and Lucia Mirabella and Stefano Ermon},
  title = {{Learning Neural PDE Solvers with Convergence Guarantees}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=nKihqwoBgOE},
}
Figure