Learning data-driven discretizations for partial differential equations

Yohai Bar‐Sinai, Stephan Hoyer, Jason Hickey, Michael P. Brenner

TL;DR

This paper introduces "data-driven discretization," a novel method that uses neural networks to learn optimized approximations of partial differential equations (PDEs) directly from high-resolution simulation data. This approach allows for accurate numerical integration of complex PDEs at significantly coarser resolutions than traditional methods, opening doors for more efficient simulations of physical systems.

What Problem Does This Paper Address?

Solving partial differential equations (PDEs) numerically often requires resolving extremely fine spatial and temporal features. This can be computationally prohibitive, especially for complex systems like turbulent fluid flows. To overcome this, researchers often resort to "coarse-graining" – developing simplified, approximate equations that capture the essential long-wavelength dynamics while accounting for the effects of unresolved small-scale physics. However, deriving these coarse-grained equations is typically a difficult, ad hoc process. This paper addresses the challenge of systematically and accurately deriving these coarse-grained numerical methods by learning them directly from data.

Methods

The core of the proposed method, "data-driven discretization," involves training neural networks to approximate spatial derivatives within a PDE. Instead of relying on fixed, pre-computed finite difference formulas, the neural network learns equation-specific coefficients that best satisfy the PDE on a low-resolution grid.

Key aspects of their approach include:

  • Neural Network Architecture: A fully convolutional neural network is used to estimate spatial derivatives. This network takes low-resolution cell-averaged values as input and outputs coefficients for a generalized finite-difference formula.
  • Pseudo-linear Representation: The neural network's output is a list of coefficients that form a "pseudo-linear filter," generalizing standard finite difference schemes. This allows for the imposition of polynomial accuracy constraints.
  • End-to-End Optimization: The neural network is trained to minimize the error in the predicted time derivative of the PDE, directly optimizing for the accuracy of the integrated solution rather than just the spatial derivatives.
  • Physical Constraints: The method incorporates physical constraints, such as conservation laws (using a finite-volume scheme) and formal polynomial accuracy, to improve stability and accuracy.
  • Burgers' Equation as a Testbed: The authors demonstrate their method using Burgers' equation, a classic nonlinear PDE known for shock formation, which serves as a good model for understanding the behavior of more complex systems.

Datasets

The paper does not introduce new benchmark datasets. Instead, it generates training and evaluation datasets by running high-resolution simulations of the PDEs under study (Burgers', Korteweg-de Vries (KdV), and Kuramoto-Sivashinsky (KS) equations). These simulations are then subsampled to create the low-resolution data used for training and testing the neural network models.

Key Insights and Ideas

  • Learning Physics from Data: The most significant insight is that neural networks can effectively learn the complex, often non-intuitive, relationships governing the behavior of PDEs, even at resolutions far coarser than what's typically required. This bypasses the need for manual derivation of coarse-grained models.
  • Equation-Specific Discretizations: The learned discretizations are highly specific to the PDE being studied. This allows them to capture unique features, like shock structures in Burgers' equation, far more effectively than generic finite difference methods.
  • Interpretable Coefficients: While the neural network itself is a black box, the learned coefficients can sometimes offer physical insights. For instance, in Burgers' equation, the coefficients exhibit an "upwind bias," which aligns with physical intuition about fluid flow direction.
  • Generalization Beyond Training Domain: The trained models demonstrate remarkable generalization capabilities, performing well on spatial domains significantly larger than those used during training.

Key Results

  • Significant Resolution Improvement: The data-driven discretization method allows for accurate integration of PDEs at resolutions 4x to 8x coarser than standard finite-difference methods.
  • Outperforming Traditional Methods: For Burgers' equation, the neural network approach significantly outperforms baseline finite volume methods and even WENO schemes at coarse resolutions.
  • Stability and Accuracy: The method demonstrates improved numerical stability, particularly in regions with sharp features like shocks, where traditional methods often diverge.
  • Robustness Across PDEs: The approach is shown to be effective for a range of nonlinear PDEs, including Burgers', KdV, and KS equations, which represent different types of physical phenomena (dissipative, integrable, and chaotic, respectively).
  • Reduced Computational Cost: By enabling simulations at coarser resolutions, the method offers a substantial reduction in computational cost for achieving comparable accuracy.

Limitations and Future Work

  • Computational Cost of Training: While inference is fast, training the neural networks can be computationally intensive.
  • Generalization to Higher Dimensions and Complex Grids: The current work focuses on 1D problems with regular grids. Scaling to higher dimensions and irregular or adaptive grids is a significant future challenge.
  • Interpretability: While some coefficients offer insights, extracting deeper abstract physical understanding from the learned models remains a difficult problem.
  • Numerical Stability Guarantees: While the method shows improved stability, formal guarantees for all scenarios are still an area of active research.
  • Speed Improvements: The authors suggest exploring alternative machine learning architectures (e.g., using banks of pre-trained linear filters) to further accelerate the inference process.

Summary updated

Abstract

The numerical solution of partial differential equations (PDEs) is challenging because of the need to resolve spatiotemporal features over wide length- and timescales. Often, it is computationally intractable to resolve the finest features in the solution. The only recourse is to use approximate coarse-grained representations, which aim to accurately represent long-wavelength dynamics while properly accounting for unresolved small-scale physics. Deriving such coarse-grained equations is notoriously difficult and often ad hoc. Here we introduce data-driven discretization, a method for learning optimized approximations to PDEs based on actual solutions to the known underlying equations. Our approach uses neural networks to estimate spatial derivatives, which are optimized end to end to best satisfy the equations on a low-resolution grid. The resulting numerical methods are remarkably accurate, allowing us to integrate in time a collection of nonlinear equations in 1 spatial dimension at resolutions 4× to 8× coarser than is possible with standard finite-difference methods.
BibTeX
@inproceedings{barsinai2019learning,
  author = {Yohai Bar‐Sinai and Stephan Hoyer and Jason Hickey and Michael P. Brenner},
  title = {{Learning data-driven discretizations for partial differential equations}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2019},
  doi = {10.1073/pnas.1814058116},
}
Figure