Error Estimates for the Deep Ritz Method with Boundary Penalty

Johannes Müller, Marius ZeinhoferMSML 2022 · OpenReview

Abstract

This paper establishes a rigorous mathematical framework for estimating the error of the Deep Ritz Method (DRM) applied to linear elliptic equations. Specifically, it addresses the common practice of imposing Dirichlet boundary conditions via a boundary penalty method. The derived estimates account for optimization accuracy, the approximation power of the chosen ansatz class (such as ReLU networks), and the penalization strength $\lambda$. Notably, the results extend to general cases without requiring coercivity on the full $H^1(\Omega)$ space, demonstrating that the DRM preserves the advantageous scaling of neural networks in high-dimensional settings for smooth solutions.

Introduction

The Deep Ritz Method (DRM) has emerged as a powerful tool for solving Partial Differential Equations (PDEs) by reformulating them as energy minimization problems. However, enforcing Dirichlet boundary conditions—where the solution must take specific values at the boundary—remains a significant practical challenge. Unlike traditional Finite Element Methods (FEM), standard neural networks do not inherently satisfy these constraints.

Practitioners typically resort to the boundary penalty method, adding a weighted term to the loss function that penalizes deviations at the boundary with a factor of $\lambda$. While this approach is widely used, its theoretical convergence for general elliptic problems, such as the prototypical Poisson equation, has lacked a complete error analysis. This paper fills that gap, providing the first comprehensive estimates that account for optimization error, network expressivity, and penalty strength.

Theoretical Framework

To derive these estimates, the authors employ several sophisticated mathematical techniques:

  • Nonlinear Céa’s Lemma: The authors adapt this classical result from numerical analysis to provide a curvature-based bound on the distance between the neural network approximation and the exact solution.
  • Steklov Eigenfunction Expansion: To isolate the error introduced specifically by the boundary penalty, the researchers use a Fourier-like expansion based on Steklov eigenfunctions. This allows them to quantify exactly how the error decays as the penalty parameter $\lambda$ increases.
  • ReLU Network Theory: By combining general PDE results with quantitative universal approximation theorems for Rectified Linear Unit (ReLU) networks, the paper demonstrates how error scales with the number of neurons and the dimensionality of the problem.

Key Insights: The Penalty Trade-off

A central finding of the research is the "$\lambda$ trade-off." Increasing the penalty parameter $\lambda$ reduces the boundary error at a rate of $O(1/\lambda)$. However, a higher $\lambda$ also increases the "curvature" or stiffness of the energy landscape. This makes the optimization problem significantly harder for gradient descent algorithms, creating a practical ceiling on the optimal penalty strength. Through a 1D ODE example, the authors prove that this $1/\lambda$ rate is sharp, meaning it is the best possible convergence rate mathematically achievable for this method.

High-Dimensional Scaling

One of the primary promises of neural PDE solvers is their ability to bypass the "curse of dimensionality." The authors show that if the underlying solution is sufficiently smooth (in the Sobolev sense), the Deep Ritz Method effectively inherits the powerful approximation capabilities of neural networks. For a solution with $r+2$ degrees of smoothness in $d$ dimensions, the error scales roughly as $O(n^{-(2r+3)/4d})$, where $n$ is the number of parameters. This result confirms that for smooth functions, DRM remains efficient even as the number of dimensions increases.

Conclusion and Limitations

While this work provides a necessary theoretical foundation for the DRM, it also highlights the "optimization gap." The derived bounds include an optimization error term ($\delta$), but the paper acknowledges that reaching this global minimum in a non-convex landscape remains a challenge for practitioners. Furthermore, the analysis assumes perfect numerical integration, whereas real-world implementations use Monte Carlo sampling, which introduces its own statistical error.

Despite these hurdles, the paper bridges the gap between classical Finite Element theory and modern Deep Learning, proving that the Deep Ritz Method is a mathematically sound and potent approach for high-dimensional elliptic PDEs.

Summary updated

Abstract

We estimate the error of the Deep Ritz Method for linear elliptic equations. For Dirichlet boundary conditions, we estimate the error when the boundary values are imposed through the boundary penalty method. Our results apply to arbitrary sets of ansatz functions and estimate the error in dependence of the optimization accuracy, the approximation capabilities of the ansatz class and – in the case of Dirichlet boundary values – the penalization strength $\lambda$. To the best of our knowledge, our results are presently the only ones in the literature that treat the case of Dirichlet boundary conditions in full generality, i.e., without a lower order term that leads to coercivity on all of $H^1(\Omega)$. Further, we discuss the implications of our results for ansatz classes which are given through ReLU networks and the relation to existing estimates for finite element functions. For high dimensional problems our results show that the favourable approximation capabilities of neural networks for smooth functions are inherited by the Deep Ritz Method.
BibTeX
@inproceedings{muller2022error,
  author = {Johannes Müller and Marius Zeinhofer},
  title = {{Error Estimates for the Deep Ritz Method with Boundary Penalty}},
  booktitle = {MSML},
  year = {2022},
  url = {https://openreview.net/forum?id=QHtfk9YDxS},
}
Figure