On the approximation of the solution of partial differential equations by artificial neural networks trained by a multilevel Levenberg-Marquardt method

Henri Calandra, Serge Gratton, Elisa Riccietti, Xavier VasseurarXiv 2019 · arXiv

Solving Partial Differential Equations (PDEs) is fundamental to physics and engineering. However, traditional numerical methods like finite elements or finite differences often struggle with high-dimensional or highly nonlinear problems due to the "curse of dimensionality." Artificial Neural Networks (ANNs) have emerged as a promising alternative because they are mesh-free and capable of approximating complex high-dimensional functions.

The primary challenge in using ANNs for PDEs lies in the training process. Training involves minimizing a loss function based on the PDE's residual, which is a massive, non-convex optimization problem. While second-order methods like the Levenberg-Marquardt (LM) algorithm are robust, they are computationally expensive for large networks. This paper introduces a multilevel Levenberg-Marquardt (MLM) method that leverages algebraic multigrid (AMG) theory to accelerate training without compromising accuracy.

Methodology

The authors propose a multilevel optimization strategy that creates a hierarchy of "coarser" versions of the optimization problem. By performing expensive calculations on smaller, cheaper versions of the problem and prolongating those corrections back to the fine-scale network, the total computational cost is significantly reduced.

A key innovation is the use of an Algebraic Multigrid (AMG) Heuristic to handle the lack of physical geometry in neural network parameters. The authors analyze the Hessian matrix to identify strongly coupled parameters. They treat parameters as atomic "triples" consisting of an input weight ($w_i$), an output weight ($v_i$), and a bias ($b_i$). Specifically, they employ the Ruge and Stüben method to select a subset of these triples to form a coarse network.

Architecture of the single hidden layer feedforward neural network used for PDE approximation, illustrating the parameters $w_i, v_i, b_i$ grouped into triples for the coarsening process.

The network architecture, as shown above, is a feedforward ANN with a single hidden layer. This choice is motivated by the Universal Approximation Theorem, which states that such networks can approximate any continuous function to arbitrary precision.

Experimental Setup

The MLM method was evaluated on several PDE benchmark problems:

  • 1D and 2D Poisson Equations: Standard benchmarks for elliptic PDEs.
  • 1D and 2D Helmholtz Equations: Known for being difficult due to oscillatory solutions.
  • Nonlinear PDEs: Including the Liouville equation and equations with sine-based nonlinearities.

The performance was measured by comparing the ANN's solution to known analytical solutions or high-accuracy finite difference references.

Key Results and Discussion

The MLM method consistently outperformed the standard one-level LM method. Key findings include:

  • Computational Efficiency: The MLM method required 1.5x to 4.3x fewer floating-point operations (FLOPs) to reach the same accuracy as the standard approach. While the number of iterations might not always decrease, the cost per iteration on coarser levels is significantly lower.
  • Robustness: The method proved effective across various PDE types, including highly oscillatory and nonlinear cases.
  • Accuracy Preservation: The Root Mean Square Error (RMSE) remained identical to the standard approach, indicating that the coarsening heuristic preserves vital information.

The study highlights that even though neural network weights lack an explicit physical grid, the Hessian matrix reveals an underlying algebraic structure that can be exploited for multilevel optimization.

Conclusion and Future Work

While successful, this study focused on networks with a single hidden layer. Future research will explore extending the AMG heuristic to deep (multilayer) architectures, which are the industry standard. Additionally, the authors suggest that developing a "matrix-free" version of this multilevel strategy could further enhance performance for very large-scale problems, avoiding the need to explicitly build large Jacobian or Hessian matrices.

Summary updated

Abstract

This paper is concerned with the approximation of the solution of partial differential equations by means of artificial neural networks. Here a feedforward neural network is used to approximate the solution of the partial differential equation. The learning problem is formulated as a least squares problem, choosing the residual of the partial differential equation as a loss function, whereas a multilevel Levenberg-Marquardt method is employed as a training method. This setting allows us to get further insight into the potential of multilevel methods. Indeed, when the least squares problem arises from the training of artificial neural networks, the variables subject to optimization are not related by any geometrical constraints and the standard interpolation and restriction operators cannot be employed any longer. A heuristic, inspired by algebraic multigrid methods, is then proposed to construct the multilevel transfer operators. Numerical experiments show encouraging results related to the efficiency of the new multilevel optimization method for the training of artificial neural networks, compared to the standard corresponding one-level procedure.
BibTeX
@misc{calandra2019approximation,
  author = {Henri Calandra and Serge Gratton and Elisa Riccietti and Xavier Vasseur},
  title = {{On the approximation of the solution of partial differential equations by artificial neural networks trained by a multilevel Levenberg-Marquardt method}},
  year = {2019},
  eprint = {1904.04685},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.04685},
}
Figure