Physics-informed deep learning and compressive collocation for high-dimensional diffusion-reaction equations: practical existence theory and numerics

Simone Brugiapaglia, Nick C. Dexter, Samir Karam, Weiqi WangarXiv 2024 · DOI · arXiv

Abstract

Solving high-dimensional partial differential equations (PDEs) is a fundamental challenge in scientific computing due to the "curse of dimensionality," where computational costs grow exponentially with the number of dimensions. This paper establishes a rigorous theoretical and numerical framework for solving steady-state diffusion-reaction equations in high dimensions using Physics-Informed Neural Networks (PINNs) and Compressive Fourier Collocation (CFC). We provide a "practical existence theorem" demonstrating that PINNs can achieve accurate solutions with sample complexity scaling only logarithmically or linearly with dimension. Furthermore, we introduce an adaptive CFC method that achieves near-machine precision for sparse solutions. Numerical experiments in up to 30 dimensions validate the efficiency and robustness of these approaches.

Introduction

The "curse of dimensionality," a term coined by Richard Bellman, describes the exponential explosion of computational requirements as the dimensionality ($d$) of a problem increases. In the context of Partial Differential Equations (PDEs), traditional grid-based methods become intractable for $d \gg 3$. While Deep Learning (DL) has emerged as a powerful tool for high-dimensional problems, its application to PDEs often lacks a solid theoretical foundation regarding sample complexity and trainability.

This work, published by Simone Brugiapaglia and colleagues, bridges the gap between empirical DL success and mathematical sparse approximation theory. By focusing on steady-state diffusion-reaction equations on a $d$-dimensional torus, the authors demonstrate that high-dimensional PDEs can be solved efficiently if the solution exhibits sparsity in the Fourier domain.

Methodology

Periodic Physics-Informed Neural Networks (PINNs)

To handle periodic boundary conditions naturally, the authors employ a specialized PINN architecture. This network incorporates a "periodic layer" that transforms input coordinates using sine and cosine functions before passing them to subsequent hidden layers.

Architecture of a Periodic Physics-Informed Neural Network (PINN) featuring a specialized periodic layer to handle boundary conditions in high-dimensional spaces.

The network is trained by minimizing the PDE residual at a set of random collocation points. The use of periodic layers ensures that the boundary conditions are satisfied by construction, allowing the optimization process to focus entirely on the interior physics.

Compressive Fourier Collocation (CFC)

In addition to PINNs, the paper explores Compressive Fourier Collocation (CFC), a non-DL approach rooted in Compressive Sensing. CFC assumes the solution is a sparse linear combination of Fourier basis functions. To navigate the massive search space of Fourier indices in high dimensions, the authors propose Adaptive Lower OMP, a greedy algorithm that iteratively expands the set of basis functions while maintaining a "lower set" structure.

Theoretical Framework: Practical Existence

The core contribution is a "practical existence theorem." Unlike standard universal approximation theorems, this theory accounts for sample complexity—the number of training points ($m$) required to achieve a target accuracy.

The authors prove that a class of Deep Neural Networks using Rectified Power Units (RePU) can emulate products of sines and cosines. This allows them to show that for a solution with sparsity $s$, the required sample size scales as $O(s \cdot \text{polylog}(s, d))$. This result is significant because it replaces exponential scaling with a mild logarithmic or linear dependence on the physical dimension $d$.

Numerical Evaluation

The methodologies were tested on three synthetic benchmark problems in dimensions ranging from $d=6$ to $d=30$, including highly sparse, anisotropic, and full-dimensional smooth solutions.

Key Findings

  • Robustness vs. Precision: PINNs proved highly robust, consistently achieving errors between $10^{-2}$ and $10^{-3}$ across all test cases. In contrast, the adaptive CFC method achieved extraordinary precision (up to $10^{-10}$) for truly sparse solutions, effectively reaching machine precision where PINNs typically plateau.
  • Scaling Success: Both methods successfully solved PDEs in 30 dimensions. Empirical results confirmed that the sample size required to maintain a target error grew linearly, rather than exponentially, as the dimension doubled from 10 to 20.

Discussion and Future Work

While the theoretical framework provides a strong foundation, an "activation gap" remains: the theory utilizes RePU activations for exact Fourier emulation, whereas numerical success is often achieved with tanh or other smooth activations. Additionally, the theory assumes the existence of a global minimizer, which may not always be reached by standard optimization algorithms like Adam or SGD.

Future research will focus on extending these methods to time-dependent PDEs and non-periodic domains. Handling Dirichlet or Neumann boundary conditions will require more sophisticated analysis of the underlying Riesz systems to ensure convergence in high-dimensional settings.

Summary updated

Abstract

On the forefront of scientific computing, Deep Learning (DL), i.e., machine learning with Deep Neural Networks (DNNs), has emerged a powerful new tool for solving Partial Differential Equations (PDEs). It has been observed that DNNs are particularly well suited to weakening the effect of the curse of dimensionality, a term coined by Richard E. Bellman in the late `50s to describe challenges such as the exponential dependence of the sample complexity, i.e., the number of samples required to solve an approximation problem, on the dimension of the ambient space. However, although DNNs have been used to solve PDEs since the `90s, the literature underpinning their mathematical efficiency in terms of numerical analysis (i.e., stability, accuracy, and sample complexity), is only recently beginning to emerge. In this paper, we leverage recent advancements in function approximation using sparsity-based techniques and random sampling to develop and analyze an efficient high-dimensional PDE solver based on DL. We show, both theoretically and numerically, that it can compete with a novel stable and accurate compressive spectral collocation method for the solution of high-dimensional, steady-state diffusion-reaction equations with periodic boundary conditions. In particular, we demonstrate a new practical existence theorem, which establishes the existence of a class of trainable DNNs with suitable bounds on the network architecture and a sufficient condition on the sample complexity, with logarithmic or, at worst, linear scaling in dimension, such that the resulting networks stably and accurately approximate a diffusion-reaction PDE with high probability.
BibTeX
@misc{brugiapaglia2024physics,
  author = {Simone Brugiapaglia and Nick C. Dexter and Samir Karam and Weiqi Wang},
  title = {{Physics-informed deep learning and compressive collocation for high-dimensional diffusion-reaction equations: practical existence theory and numerics}},
  year = {2024},
  eprint = {2406.01539},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.01539},
}
Figure