Abstract
Recently, neural networks have been widely applied for solving partial differential equations (PDEs). Although such methods have been proven remarkably successful on practical engineering problems, they have not been shown, theoretically or empirically, to converge to the underlying PDE solution with arbitrarily high accuracy. The primary difficulty lies in solving the highly non-convex optimization problems resulting from the neural network discretization, which are difficult to treat both theoretically and practically. It is our goal in this work to take a step toward remedying this. For this purpose, we develop a novel greedy training algorithm for shallow neural networks. Our method is applicable to both the variational formulation of the PDE and also to the residual minimization formulation pioneered by physics informed neural networks (PINNs). We analyze the method and obtain a priori error bounds when solving PDEs from the function class defined by shallow networks, which rigorously establishes the convergence of the method as the network size increases. Finally, we test the algorithm on several benchmark examples, including high dimensional PDEs, to confirm the theoretical convergence rate. Although the method is expensive relative to traditional approaches such as finite element methods, we view this work as a proof of concept for neural network-based methods, which shows that numerical methods based upon neural networks can be shown to rigorously converge.
Introduction
Solving Partial Differential Equations (PDEs) with neural networks has initiated a paradigm shift in computational science. Frameworks such as Physics-Informed Neural Networks (PINNs) and the Deep Ritz method have demonstrated significant potential, particularly in high-dimensional spaces where traditional Finite Element Methods (FEM) succumb to the "curse of dimensionality."
However, a critical gap remains: while these neural solvers perform well in empirical settings, they often lack the rigorous convergence guarantees central to classical numerical analysis. In standard training, increasing the number of neurons does not necessarily guarantee that the error will vanish. Optimization routines like Adam or SGD frequently encounter local minima or plateaus, preventing the model from reaching the true PDE solution. This research introduces a novel greedy training algorithm for shallow neural networks designed to provide these missing theoretical and practical guarantees.
The Greedy Training Framework
The core innovation of this work is the move away from simultaneous weight updates in favor of an incremental, greedy strategy. By adding neurons one by one, the complex non-convex optimization problem is decomposed into a sequence of more manageable sub-problems.
Algorithmic Variants
- Relaxed Greedy Algorithm (RGA): This method builds the network incrementally. In each step, it identifies a single new "basis function" (a neuron) that best reduces the current residual or energy functional. This neuron is then added to the ensemble with a specific weight.
- Orthogonal Greedy Algorithm (OGA): A more robust iteration of the RGA. After selecting a new neuron, the algorithm performs an orthogonal projection to optimize the coefficients of all currently selected neurons simultaneously, ensuring the best possible approximation at each step.
- Dictionary-Based Selection: Instead of optimizing over an infinite parameter space, neurons are selected from a "dictionary" of functions (e.g., ReLU² or ReLU³). This allows the application of approximation theory to prove that the error decreases at a predictable rate as the network width increases.
Theoretical Convergence and Adaptivity
By utilizing greedy selection, the authors establish a priori error bounds. This rigorously proves that the neural network approximation converges to the true PDE solution as the number of neurons ($n$) grows. Specifically, the OGA can match theoretical convergence rates of $O(n^{-1})$ or $O(n^{-2})$, depending on the problem structure.
Beyond pure convergence, the algorithm exhibits inherent adaptivity. In numerical tests, the greedy selection process automatically placed neurons in regions where the solution exhibited the highest curvature. Much like an adaptive mesh in traditional FEM, the neural network "discovered" where high-resolution detail was required without manual intervention.
Numerical Benchmarks
The algorithm was validated against several challenging PDE benchmarks:
- High-Dimensional Elliptic Equations: The method successfully solved a 10-dimensional PDE using only 256 neurons. A traditional grid-based method would require an astronomical number of points ($N^{10}$) to achieve similar results, demonstrating a clear circumvention of the curse of dimensionality.
- High-Order and Non-linear PDEs: The RGA and OGA were tested on fourth-order equations and the non-linear Poisson-Boltzmann equation, proving the versatility of the greedy approach across different functional forms.
- Precision Comparison: In 1D and 2D benchmarks, the OGA achieved high-precision results with errors as low as $10^{-10}$. In contrast, standard training methods like Adam and SGD often saw error rates flatten out as the network grew, failing to exploit the increased capacity of the model.
Limitations and Conclusion
While the greedy algorithm provides a path to provable convergence, it comes with a computational trade-off. Solving the "argmax" sub-problem—finding the optimal next neuron—is itself a non-convex optimization task. Although more tractable than training a full deep network, it remains computationally expensive compared to traditional linear solvers used in FEM.
Furthermore, this study focuses on shallow (one-hidden-layer) networks. Extending these greedy strategies to deep architectures remains an open area for future research. Nevertheless, this work serves as a vital proof of concept, demonstrating that neural network-based numerical methods can be held to the same rigorous standards of convergence as classical techniques, paving the way for more reliable scientific machine learning.
Summary updated