Improved Physics-Informed Neural Networks Loss Function Regularization with a Variance-Based Term

John M. Hanna, Irène Vignon-Clémentel, Hugues TalbotSSRN 2025 · DOI

TL;DR

This paper introduces a novel loss function for Physics-Informed Neural Networks (PINNs) that incorporates the standard deviation of the error alongside the mean. This variance-based approach effectively reduces the impact of outliers and localized high-error regions, leading to more accurate solutions for complex physics problems compared to traditional mean-based loss functions.

What Problem Does This Paper Address?

Deep learning, particularly through Physics-Informed Neural Networks (PINNs), has shown great promise in solving partial differential equations (PDEs). PINNs leverage the physics governing a problem to regularize the neural network's predictions, reducing the need for large datasets. However, a common challenge in PINNs is the presence of localized errors, especially in regions with sharp gradients or discontinuities. These outliers can significantly degrade the overall accuracy of the solution, even when the average error is minimized. Standard loss functions, like Mean Squared Error (MSE) or Mean Absolute Error (MAE), primarily focus on minimizing the mean error and are not robust to these localized high-error regions. This paper aims to address this limitation by proposing a loss function that explicitly accounts for and reduces the variance of the error.

Methods

The core innovation of this paper is a new loss function that augments the traditional mean-based error with a term representing the standard deviation of the error. The proposed loss function is defined as:

$L = \alpha \bar{e} + (1-\alpha) \sqrt{\frac{1}{N} \sum_{i=1}^{N} (e_i - \bar{e})^2}$

where:

  • $L$ is the total loss.
  • $\alpha$ is a hyperparameter (between 0 and 1) that balances the contribution of the mean error ($\bar{e}$) and the standard deviation of the error.
  • $\bar{e}$ is the mean of the chosen error function $e_i$.
  • $\sqrt{\frac{1}{N} \sum_{i=1}^{N} (e_i - \bar{e})^2}$ is the standard deviation of the error.

By minimizing this combined loss, the PINN is encouraged to not only reduce the average error but also to distribute the error more uniformly across the domain, thereby mitigating the impact of outliers. The authors implement this by adding the standard deviation term to both the data loss ($L_{data}$) and the PDE residual loss ($L_{PDE}$).

Datasets

The paper does not introduce new datasets. Instead, it evaluates the proposed loss function on three established benchmark problems:

  1. Burger's equation: A classic nonlinear PDE used to model phenomena like traffic flow and fluid dynamics.
  2. 2D linear elastic solid mechanics: A problem in solid mechanics involving stress and strain analysis.
  3. 2D steady-state Navier-Stokes equations: A fundamental set of equations describing fluid flow.

These problems are solved using PINNs, and the results are compared against solutions obtained with standard MSE loss.

Key Insights and Ideas

  • Variance as a Regularizer: The key insight is that explicitly penalizing the variance of the error acts as a powerful regularizer, forcing the neural network to learn solutions that are more uniformly accurate, rather than just minimizing the average error.
  • Outlier Mitigation: The variance-based loss function demonstrably reduces the impact of localized high-error regions, which are common in PINNs due to sharp gradients or discontinuities in the underlying physics.
  • Improved Solution Quality: By addressing outliers, the proposed method leads to a better approximation of the true solution, especially in challenging regions.
  • Computational Efficiency: The added computational cost of calculating the standard deviation is minimal, making the proposed loss function a practical and efficient improvement over standard methods.

Key Results

The paper demonstrates the effectiveness of the variance-based loss function across three different physics problems:

  • Burger's Equation: The maximum absolute residual was significantly reduced compared to MSE loss, leading to a better overall solution approximation.
  • Solid Mechanics: For displacement in the x-direction, the maximum absolute error was reduced by an order of 20. For displacement in the y-direction, the maximum error was reduced by an order of 10. The variance-based loss also eliminated localized error regions observed with MSE.
  • Fluid Mechanics (Navier-Stokes): The variance-based loss captured a shifted maximum velocity profile at the outlet due to centrifugal forces, which was not accurately represented by MSE loss. The maximum error in pressure estimation was also reduced by approximately half compared to MSE loss.

In terms of computational time, the variance-based loss function incurred only a small overhead compared to MSE loss (e.g., 25.1 seconds vs. 24.9 seconds for Burger's equation, and 225 seconds vs. 222 seconds for solid mechanics).

Limitations and Future Work

  • Hyperparameter Tuning ($\alpha$): The paper used $\alpha = 0.5$ for all experiments, giving equal weight to the mean and standard deviation. The authors suggest that a proper assessment of the effect of changing $\alpha$ is needed for future work.
  • Optimizer: While Adam optimizer was sufficient for Burgers and linear elasticity, it was not optimal for the Navier-Stokes example. The authors suggest exploring more powerful optimizers, possibly in conjunction with Adam, for complex problems.
  • Computational Cost Estimation: A more precise estimation of the added computational cost is recommended for future study.
  • Implementation: The paper notes that the variance term is simple to implement in popular deep learning frameworks like TensorFlow and PyTorch.

Summary updated

BibTeX
@misc{hanna2025improved,
  author = {John M. Hanna and Irène Vignon-Clémentel and Hugues Talbot},
  title = {{Improved Physics-Informed Neural Networks Loss Function Regularization with a Variance-Based Term}},
  year = {2025},
  doi = {10.2139/ssrn.5269341},
  howpublished = {SSRN},
}

Loading related content...

Figure