An improved data-free surrogate model for solving partial differential equations using deep neural networks

Xinhai Chen, Rongliang Chen, Qian Wan, Rui Xu, Jie LiuScientific 2021 · DOI

Abstract: Partial differential equations (PDEs) are ubiquitous in natural science and engineering problems. Traditional discrete methods for solving PDEs are usually time-consuming and labor-intensive due to the need for tedious mesh generation and numerical iterations. Recently, deep neural networks have shown new promise in cost-effective surrogate modeling because of their universal function approximation abilities. In this paper, we borrow the idea from physics-informed neural networks (PINNs) and propose an improved data-free surrogate model, DFS-Net. Specifically, we devise an attention-based neural structure containing a weighting mechanism to alleviate the problem of unstable or inaccurate predictions by PINNs. The proposed DFS-Net takes expanded spatial and temporal coordinates as the input and directly outputs the observables (quantities of interest). It approximates the PDE solution by minimizing the weighted residuals of the governing equations and data-fit terms, where no simulation or measured data are needed. The experimental results demonstrate that DFS-Net offers a good trade-off between accuracy and efficiency. It outperforms the widely used surrogate models in terms of prediction performance on different numerical benchmarks, including the Helmholtz, Klein-Gordon, and Navier-Stokes equations.

Introduction

Partial Differential Equations (PDEs) serve as the mathematical foundation for describing physical phenomena ranging from acoustic propagation to fluid dynamics. While traditional mesh-based methods like Finite Element Analysis (FEA) are the industry standard, they are often computationally prohibitive for complex geometries or high-dimensional problems.

Recently, Physics-Informed Neural Networks (PINNs) have emerged as a "data-free" alternative, leveraging the underlying physical laws as a loss function to train deep neural networks. However, standard PINNs frequently suffer from training instabilities, particularly near domain boundaries, and struggle to capture sharp nonlinearities. This paper introduces DFS-Net, an evolved surrogate model designed to overcome these pathologies through architectural innovations in weighting and feature attention.

Methodology: The DFS-Net Architecture

DFS-Net improves upon the standard PINN framework by incorporating three primary technical enhancements: an expanded input layer, attention-based excitation blocks, and a specialized spatial weighting mechanism.

Spatial Weighting Mechanism

A critical challenge in neural PDE solvers is the uneven distribution of error across the physical domain. Standard models often diverge near boundaries where physical constraints are most rigid. DFS-Net addresses this by implementing a dual-weighting strategy:

  1. Interior Weighting: A piecewise function assigns higher importance to "pivotal" points in the center of the domain, ensuring the network captures the core physical behavior.
  2. Boundary Weighting: Parabolic functions are applied to boundary data to eliminate singular values caused by discontinuities, which are common in fluid dynamics problems like lid-driven cavity flow.
Spatial weighting strategies in DFS-Net: (a) 2D domain, (b) interior piecewise weighting, and (c) boundary parabolic weighting.

Attention and Feature Engineering

Beyond spatial weighting, DFS-Net utilizes Attention-based Excitation Blocks. Inspired by the "Squeeze-and-Excitation" concept in computer vision, these blocks act as a self-gating mechanism that recalibrates internal features, allowing the network to prioritize the most relevant physical signals.

Furthermore, the model employs an Expanded Input Layer. Instead of feeding raw coordinates $(x, t)$ directly, DFS-Net transforms them into a higher-dimensional feature vector including polynomial ($x^2, t^2$) and trigonometric ($\sin, \cos$) terms. This manual feature enhancement allows the network to approximate complex nonlinear patterns much earlier in the training process.

Experimental Evaluation

As a data-free model, DFS-Net is trained solely on the residuals of the governing equations. Its performance was evaluated against three classic numerical benchmarks:

  • Helmholtz Equation: A standard test for acoustics and electromagnetism.
  • Klein-Gordon Equation: A nonlinear PDE relevant to quantum physics.
  • Navier-Stokes (Lid-driven Cavity Flow): A rigorous test of fluid dynamics involving a square cavity with a moving lid, tested at Reynolds numbers ($Re$) of 100, 300, and 600.

Key Results and Discussion

The experimental results indicate that DFS-Net significantly outperforms existing surrogate models like the Deep Galerkin Method (DGM) and standard PINNs.

  • Accuracy: In the Helmholtz benchmark, DFS-Net achieved errors approximately two orders of magnitude lower than standard PINNs.
  • Robustness: In complex fluid flow scenarios ($Re=100$), DFS-Net maintained stability and reached a relative $L^2$ error of ~1.34%, whereas standard PINNs often produced massive errors near the right boundary walls.
  • Efficiency: While the attention blocks add slight computational overhead per epoch, the model's superior convergence rate makes it more cost-effective overall, reaching higher accuracy in fewer total iterations.

Conclusion

DFS-Net represents a significant step forward in making neural PDE solvers robust enough for real-world engineering. By combining spatial weighting with attention mechanisms and feature expansion, it mitigates the common stability issues of PINNs. Future work will focus on extending this framework to 3D problems and irregular geometries, as well as optimizing performance for higher Reynolds numbers where turbulence becomes a dominant factor.

Summary updated

Abstract

Partial differential equations (PDEs) are ubiquitous in natural science and engineering problems. Traditional discrete methods for solving PDEs are usually time-consuming and labor-intensive due to the need for tedious mesh generation and numerical iterations. Recently, deep neural networks have shown new promise in cost-effective surrogate modeling because of their universal function approximation abilities. In this paper, we borrow the idea from physics-informed neural networks (PINNs) and propose an improved data-free surrogate model, DFS-Net. Specifically, we devise an attention-based neural structure containing a weighting mechanism to alleviate the problem of unstable or inaccurate predictions by PINNs. The proposed DFS-Net takes expanded spatial and temporal coordinates as the input and directly outputs the observables (quantities of interest). It approximates the PDE solution by minimizing the weighted residuals of the governing equations and data-fit terms, where no simulation or measured data are needed. The experimental results demonstrate that DFS-Net offers a good trade-off between accuracy and efficiency. It outperforms the widely used surrogate models in terms of prediction performance on different numerical benchmarks, including the Helmholtz, Klein-Gordon, and Navier-Stokes equations.
BibTeX
@inproceedings{chen2021improved,
  author = {Xinhai Chen and Rongliang Chen and Qian Wan and Rui Xu and Jie Liu},
  title = {{An improved data-free surrogate model for solving partial differential equations using deep neural networks}},
  booktitle = {Scientific},
  year = {2021},
  doi = {10.1038/s41598-021-99037-x},
}
Figure