Solving multiscale elliptic problems by sparse radial basis function neural networks

Zhiwen Wang, Minxin Chen, Jingrun ChenJournal of Computational Physics 2023

Introduction

Multiscale elliptic partial differential equations (PDEs) are fundamental to modeling physical phenomena across diverse fields, including fluid dynamics in porous media and the structural analysis of composite materials. These systems are characterized by features that manifest across vastly different scales, often represented by a small parameter $\epsilon$. Traditional numerical approaches, such as Finite Element Methods (FEM) or Finite Difference Methods (FDM), face significant computational hurdles in these contexts. To accurately resolve the smallest scales, these methods require extremely fine meshes, leading to a computational cost that grows exponentially as $\epsilon$ decreases.

While deep learning has emerged as a promising alternative, standard architectures like Physics-Informed Neural Networks (PINNs) often struggle with multiscale problems. These networks frequently fail to capture high-frequency oscillations or require prohibitively large architectures to achieve acceptable accuracy. This paper introduces Sparse Radial Basis Function Neural Networks (SRBFNN), a specialized framework designed to overcome these limitations by combining local basis functions with sparsity-promoting regularization.

Methodology

The SRBFNN approach diverges from standard deep learning methods through four primary innovations:

1. First-Order System Reformulation

Rather than solving the second-order elliptic PDE directly, the authors reformulate the problem into a system of first-order equations. This technique, inspired by the Deep Mixed Residual Method, introduces auxiliary variables for the derivatives. By reducing the order of the differential operators, the optimization landscape becomes more manageable for the neural network, facilitating more stable training.

2. Radial Basis Function (RBF) Neurons

Standard neural networks typically utilize global activation functions like ReLU or Tanh, where a change in a single weight can influence the entire domain. In contrast, SRBFNN employs Gaussian-like RBFs with local support. This locality is crucial for multiscale problems, as it allows the network to approximate high-frequency, localized oscillations without affecting the global solution.

3. $\ell_1$ Regularization and Pruning

To ensure computational efficiency and prevent overfitting, the framework incorporates an $\ell_1$ penalty into the loss function. This encourages the network to identify a sparse set of essential neurons. A pruning algorithm is implemented during training to remove neurons with negligible contributions, resulting in a compact model that retains high representational power.

4. Homogenization-Informed Initialization

A key challenge in neural network training is hyperparameter selection. SRBFNN leverages mathematical homogenization theory to intelligently initialize the width (shape) of the RBFs based on the scale $\epsilon$. This physics-informed initialization ensures the network is pre-configured to capture the specific frequency characteristics of the multiscale problem from the outset.

Experimental Evaluation

The performance of SRBFNN was evaluated against eight benchmark PDE problems, ranging from 1D periodic coefficients to complex 3D domains. The method was compared against traditional FDM and state-of-the-art neural architectures, including PINN, the Deep Galerkin Method (DGM), the Deep Ritz Method (DRM), and MscaleDNN.

Accuracy and Efficiency

In 1D and 2D benchmarks, SRBFNN consistently achieved $L_2$ errors between $10^{-4}$ and $10^{-5}$, frequently outperforming PINNs and MscaleDNN by one to two orders of magnitude. The efficiency gains were particularly striking: in a two-scale test case, SRBFNN achieved higher accuracy than MscaleDNN using only 351 parameters, compared to the 277,751 parameters required by the latter.

Robustness and Scalability

The method demonstrated significant robustness in handling discontinuous coefficients, a scenario where standard networks often produce blurred results. Furthermore, the authors successfully applied SRBFNN to a 3D periodic problem. In this high-dimensional setting, traditional mesh-based methods become computationally unaffordable due to the millions of grid points required, whereas SRBFNN provided a reliable, convergent solution with a fraction of the memory footprint.

Discussion and Future Work

The success of SRBFNN highlights the importance of "locality" in approximating multiscale features. By using RBFs, the network can refine the solution in specific sub-domains independently. Mathematically, the authors demonstrate that the number of RBFs required scales as $O(\epsilon^{-n\tau})$ (where $\tau < 1$), which is superior to the $O(\epsilon^{-n})$ scaling typical of classical methods.

Despite these advantages, challenges remain. The selection of penalty parameters and the initial number of RBFs requires careful tuning. Future research will focus on developing faster sparse optimization algorithms and extending the framework to handle complex, non-rectangular geometries. Nonetheless, SRBFNN represents a significant step forward in the application of theory-driven AI to complex physical systems.

Summary updated

BibTeX
@article{wang2023solving,
  author = {Zhiwen Wang and Minxin Chen and Jingrun Chen},
  title = {{Solving multiscale elliptic problems by sparse radial basis function neural networks}},
  journal = {Journal of Computational Physics},
  year = {2023},
}
Figure