Error Estimates for the Variational Training of Neural Networks with Boundary Penalty

Johannes Müller, Marius ZeinhoferarXiv 2021 · arXiv · OpenReview

Abstract

We estimate the error of the Deep Ritz Method for linear elliptic equations. For Dirichlet boundary conditions, we estimate the error when the boundary values are imposed through the boundary penalty method. Our results apply to arbitrary sets of ansatz functions and estimate the error in dependence of the optimization accuracy, the approximation capabilities of the ansatz class and — in the case of Dirichlet boundary values — the penalization strength $\lambda$. To the best of our knowledge, our results are presently the only ones in the literature that treat the case of Dirichlet boundary conditions in full generality, i.e., without a lower order term that leads to coercivity on all of $H^1(\Omega)$. Further, we discuss the implications of our results for ansatz classes which are given through ReLU networks and the relation to existing estimates for finite element functions. For high dimensional problems our results show that the favourable approximation capabilities of neural networks for smooth functions are inherited by the Deep Ritz Method.

Introduction

The Deep Ritz Method (DRM) has emerged as a powerful "physics-informed" framework for solving partial differential equations (PDEs) by minimizing a variational energy functional. While DRM is highly flexible, enforcing Dirichlet boundary conditions—where the solution is constrained to specific values on the boundary—remains a significant challenge for standard neural network architectures.

In practice, researchers often employ the boundary penalty method. This approach adds a term to the loss function that penalizes deviations from the boundary values. Despite its widespread use, a rigorous theoretical understanding of this penalty was previously limited to specific cases. This paper provides the first comprehensive mathematical error analysis for DRM using boundary penalties for linear elliptic PDEs, filling a critical gap in the theoretical foundation of neural PDE solvers.

The Variational Challenge

Traditional proofs for the convergence of variational methods often rely on "coercivity" (a form of mathematical stability) across the entire Sobolev space $H^1(\Omega)$. Unfortunately, many fundamental equations, such as the Poisson equation, are only coercive on the subspace of functions that vanish at the boundary ($H^1_0$).

The authors overcome this hurdle by relaxing the requirements for the PDE operator. By proving the method works for operators that are coercive only on $H^1_0$, they extend the theory to a much broader class of real-world problems. This "coercivity breakthrough" ensures that the Deep Ritz Method is mathematically sound even for the most common physical models.

Methodology

To dissect the error of the DRM, the authors utilize several advanced tools from functional analysis:

  • Nonlinear Céa’s Lemma: The authors adapt this classic result from Finite Element Method (FEM) theory to accommodate non-linear sets, such as neural networks. This allows them to bound the distance between the neural network solution and the true solution.
  • Steklov Eigenfunctions: To specifically address the boundary penalty, the researchers use a Fourier-like expansion based on Steklov eigenfunctions. This technique allows them to precisely quantify the "leakage" at the boundary caused by using a finite penalty parameter ($\lambda$).
  • Quantitative Universal Approximation: By combining PDE error estimates with existing theorems on ReLU network approximation, the authors derive explicit convergence rates for specific architectures.

The Three-Way Error Decomposition

A central contribution of this work is the identification of a three-way trade-off that determines the total error in the $H^1$ norm. The error is composed of:

  1. Optimization Error ($\delta$): This represents how close the training algorithm (e.g., Adam or SGD) got to the global minimum of the loss function.
  2. Approximation Error: This reflects the inherent capacity of the chosen neural network architecture to represent the true solution.
  3. Penalty Error ($1/\lambda$): This is the error introduced by the penalty method itself. Because the boundary condition is not enforced exactly, there is a residual error that scales inversely with the penalty strength.

Through a 1D example, the authors demonstrate that the $1/\lambda$ convergence rate for the boundary penalty is sharp, meaning it represents the fundamental limit of the penalty approach.

Overcoming the Curse of Dimensionality

One of the most promising aspects of neural network-based PDE solvers is their performance in high-dimensional spaces. The paper establishes that if the underlying solution to a high-dimensional PDE is smooth, neural networks can approximate it far more efficiently than traditional grid-based methods.

For a solution with $r$-order smoothness in $d$ dimensions, the authors show that with an optimal choice of penalty $\lambda$, the error scales approximately as $n^{-(2r+3)/4d}$, where $n$ is the number of parameters. Unlike the Finite Element Method, which requires the complex construction of higher-order elements to exploit smoothness, the Deep Ritz Method inherits this "adaptation to smoothness" naturally through the neural network architecture.

Limitations and Future Outlook

While this work provides a rigorous foundation for DRM, several challenges remain:

  • The Optimization Gap: The theory assumes an optimization accuracy $\delta$, but it does not provide a strategy for achieving a small $\delta$ given the non-convex nature of neural network training.
  • Regularity Requirements: The proofs for Dirichlet conditions require the solution to possess $H^2$ regularity. This may not hold for problems involving singular data or highly irregular boundaries.
  • Numerical Integration: The current analysis assumes perfect computation of the integrals in the loss function. In practice, these are estimated via sampling or quadrature. Future work could integrate "generalization error" analysis to account for these sampling effects.

In conclusion, this paper establishes that the Deep Ritz Method is a theoretically sound and powerful tool for solving elliptic PDEs, particularly when the problem's smoothness can be leveraged to bypass the curse of dimensionality.

Summary updated

Abstract

We estimate the error of the Deep Ritz Method for linear elliptic equations. For Dirichlet boundary conditions, we estimate the error when the boundary values are imposed through the boundary penalty method. Our results apply to arbitrary sets of ansatz functions and estimate the error in dependence of the optimization accuracy, the approximation capabilities of the ansatz class and -- in the case of Dirichlet boundary values -- the penalization strength $\lambda$. To the best of our knowledge, our results are presently the only ones in the literature that treat the case of Dirichlet boundary conditions in full generality, i.e., without a lower order term that leads to coercivity on all of $H^1(\Omega)$. Further, we discuss the implications of our results for ansatz classes which are given through ReLU networks and the relation to existing estimates for finite element functions. For high dimensional problems our results show that the favourable approximation capabilities of neural networks for smooth functions are inherited by the Deep Ritz Method.
BibTeX
@misc{muller2021error,
  author = {Johannes Müller and Marius Zeinhofer},
  title = {{Error Estimates for the Variational Training of Neural Networks with Boundary Penalty}},
  year = {2021},
  eprint = {2103.01007},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.01007},
}
Figure