Self-adaptive loss balanced Physics-informed neural networks for the incompressible Navier-Stokes equations

Zixue Xiang, Wei Peng, Xiaohu Zheng, Xiaoyu Zhao, Wen Yao

Abstract

There have been several efforts to Physics-informed neural networks (PINNs) in the solution of the incompressible Navier-Stokes fluid. The loss function in PINNs is a weighted sum of multiple terms, including the mismatch in the observed velocity and pressure data, the boundary and initial constraints, as well as the residuals of the Navier-Stokes equations. In this paper, we observe that the weighted combination of competitive multiple loss functions plays a significant role in training PINNs effectively. We establish Gaussian probabilistic models to define the loss terms, where the noise collection describes the weight parameter for each loss term. We propose a self-adaptive loss function method, which automatically assigns the weights of losses by updating the noise parameters in each epoch based on the maximum likelihood estimation. Subsequently, we employ the self-adaptive loss balanced Physics-informed neural networks (lbPINNs) to solve the incompressible Navier-Stokes equations, including two-dimensional steady Kovasznay flow, two-dimensional unsteady cylinder wake, and three-dimensional unsteady Beltrami flow. Our results suggest that the accuracy of PINNs for effectively simulating complex incompressible flows is improved by adaptively appropriate weights in the loss terms. The outstanding adaptability of lbPINNs is not irrelevant to the initialization choice of noise parameters, which illustrates the robustness. The proposed method can also be employed in other problems where PINNs apply besides fluid problems.

Introduction

Physics-Informed Neural Networks (PINNs) have emerged as a transformative mesh-free approach for solving partial differential equations (PDEs), particularly in the domain of Computational Fluid Dynamics (CFD). By embedding physical laws directly into the neural network's loss function, PINNs can approximate solutions to complex fluid problems without the need for traditional discretization. However, a significant hurdle in their practical application is the "balancing act" required during training.

A standard PINN loss function typically aggregates several disparate terms: PDE residuals, initial conditions (IC), boundary conditions (BC), and experimental data. These terms often have vastly different scales and gradients, leading to a competitive optimization landscape. If the weights assigned to these terms are not meticulously tuned, the network may prioritize one constraint at the expense of others. Historically, this tuning has been a manual, trial-and-error process. This paper introduces lbPINNs, a self-adaptive framework designed to automate this balancing process through a principled probabilistic approach.

Methodology

The core innovation of the self-adaptive loss balanced method (lbPINN) is the treatment of the training process as a multi-task learning problem. Instead of a simple weighted sum, the authors frame the loss terms within a Gaussian probabilistic framework.

Probabilistic Modeling and MLE

Each component of the total loss (PDE, BC, IC, and data) is modeled as a Gaussian likelihood. In this formulation, the "weight" of each task is inversely proportional to a learnable "noise" parameter ($\epsilon$). The total loss function is derived using Maximum Likelihood Estimation (MLE), resulting in a form that includes both the weighted residuals and a regularization term ($\log \epsilon$). This regularization term is critical; it prevents the optimizer from simply increasing the noise parameters to infinity to trivially minimize the loss.

Architecture of the lbPINN framework with learnable noise parameters.

Dynamic Weight Adaptation

Unlike static weighting schemes, these noise parameters are updated dynamically using the Adam optimizer in every epoch, alongside the network's weights and biases. This allows the model to automatically identify which constraints are currently the most difficult to satisfy and adjust their relative importance on the fly. This dynamic adjustment helps navigate the complex optimization landscape of the Navier-Stokes equations more effectively than fixed weights.

Experimental Evaluation

The lbPINN method was rigorously tested against three benchmark fluid dynamics problems:

  1. 2D Steady Kovasznay Flow: Used to verify the model's ability to resolve steady-state solutions with analytical benchmarks.
  2. 2D Unsteady Cylinder Wake: A dynamic scenario involving vortex shedding, where data was generated via high-fidelity spectral element methods.
  3. 3D Unsteady Beltrami Flow: A complex three-dimensional test case to evaluate the scalability and robustness of the adaptive weighting scheme.

Results and Discussion

The lbPINN approach demonstrated significant performance gains over traditional PINNs across all benchmarks. In the Kovasznay flow, relative L2 errors were reduced by an order of magnitude. The most striking results were observed in the unsteady cylinder wake, where lbPINN achieved a velocity error of $4.8 \times 10^{-6}$, compared to $5.3 \times 10^{-4}$ for the baseline PINN.

Comparison of convergence rates and dynamic weight adjustment between PINNs and lbPINNs.

As shown in the convergence plots, the self-adaptive weights ($\omega$) and noise parameters ($\epsilon$) stabilize relatively quickly, typically within the first 10,000 iterations. This rapid adaptation leads to faster overall convergence and a more accurate final state. Furthermore, the method proved robust to the initial settings of the noise parameters, consistently converging to high-accuracy solutions regardless of the starting point. This robustness addresses one of the primary criticisms of manual weight tuning.

Conclusion

The introduction of self-adaptive loss balancing via lbPINNs represents a significant step forward in making PINNs a reliable tool for fluid dynamics. By replacing manual hyperparameter tuning with an automated, probabilistic framework, the authors have improved both the accuracy and the ease of use for simulating complex incompressible flows. Future work will focus on providing deeper theoretical insights into the optimization dynamics and extending the method to handle more complex geometries and moving boundaries.

Summary updated

Abstract

There have been several efforts to Physics-informed neural networks (PINNs) in the solution of the incompressible Navier-Stokes fluid. The loss function in PINNs is a weighted sum of multiple terms, including the mismatch in the observed velocity and pressure data, the boundary and initial constraints, as well as the residuals of the Navier-Stokes equations. In this paper, we observe that the weighted combination of competitive multiple loss functions plays a significant role in training PINNs effectively. We establish Gaussian probabilistic models to define the loss terms, where the noise collection describes the weight parameter for each loss term. We propose a self-adaptive loss function method, which automatically assigns the weights of losses by updating the noise parameters in each epoch based on the maximum likelihood estimation. Subsequently, we employ the self-adaptive loss balanced Physics-informed neural networks (lbPINNs) to solve the incompressible Navier-Stokes equations,\hspace{-1pt} including\hspace{-1pt} two-dimensional\hspace{-1pt} steady Kovasznay flow, two-dimensional unsteady cylinder wake, and three-dimensional unsteady Beltrami flow. Our results suggest that the accuracy of PINNs for effectively simulating complex incompressible flows is improved by adaptively appropriate weights in the loss terms. The outstanding adaptability of lbPINNs is not irrelevant to the initialization choice of noise parameters, which illustrates the robustness. The proposed method can also be employed in other problems where PINNs apply besides fluid problems.
BibTeX
@inproceedings{xiang2022self,
  author = {Zixue Xiang and Wei Peng and Xiaohu Zheng and Xiaoyu Zhao and Wen Yao},
  title = {{Self-adaptive loss balanced Physics-informed neural networks for the incompressible Navier-Stokes equations}},
  booktitle = {Neurocomputing},
  year = {2022},
  url = {https://openreview.net/forum?id=ef5zrNfcGQ},
}
Figure