Robust Regression with Highly Corrupted Data Via Physics Informed Neural Networks

Wei Peng, Wen Yao, Weien Zhou, Xiaoya Zhang, Weijie Yao

TL;DR

This paper introduces LAD-PINN and a two-stage MAD-PINN framework to address the challenge of training Physics-Informed Neural Networks (PINNs) with highly corrupted or noisy data. By replacing the standard L2 loss with an L1 loss for data fitting, these methods demonstrate significantly improved robustness in reconstructing solutions and identifying parameters of PDEs, even when a large portion of the observations are erroneous.

What Problem Does This Paper Address?

Physics-Informed Neural Networks (PINNs) have emerged as a powerful tool for solving and discovering partial differential equations (PDEs). They leverage both data and the governing physical laws to learn solutions. However, the effectiveness of standard PINNs, which typically rely on the L2 norm (mean squared error) for data fitting, is severely compromised when the observational data is heavily corrupted by outliers or spurious values. This is analogous to the well-known issue of Ordinary Least Squares (OLS) regression being sensitive to outliers. The paper aims to develop robust PINN variants that can reliably handle such corrupted data, enabling their application in real-world scenarios where data quality is often imperfect.

Methods

The core innovation lies in adapting robust regression techniques to the PINN framework.

  • LAD-PINN (Least Absolute Deviation PINN): This method replaces the L2 loss term for data fitting in the standard PINN objective function with an L1 loss (sum of absolute errors). This change makes the model less sensitive to large deviations caused by outliers. The PDE loss term remains the same, ensuring that the physical constraints are still enforced.
  • Two-Stage MAD-PINN (Median Absolute Deviation PINN): Motivated by robust linear regression methods, this approach uses a two-stage process for enhanced accuracy.
  • Outlier Detection: LAD-PINN is first used to obtain a preliminary reconstruction. This reconstruction then serves as a reference to identify and filter out highly corrupted data points. Two filtering strategies are proposed: one based on a fixed percentage of maximum errors and another using the Median Absolute Deviation (MAD) to estimate the standard deviation of non-outlier data.
  • Robust Reconstruction: The remaining "cleaned" data is then used to train a standard PINN (OLS-PINN) or its variants. The LAD-PINN trained in the first stage is also used to pre-train the network in the second stage, potentially accelerating convergence.

Datasets

The paper does not introduce new benchmark datasets. Instead, it evaluates the proposed methods on synthetic datasets generated from known solutions to various PDEs. These datasets are intentionally corrupted with different types of noise to simulate real-world scenarios. The types of data corruption considered include:

  • Gaussian noise
  • Contaminated (mixed Gaussian) noise
  • Cauchy noise
  • Fixed outliers
  • Mixed noise (spurious values plus Gaussian noise)

The experiments cover problems with known PDE parameters (Poisson's equation, steady Navier-Stokes) and unknown parameters (wave equation, unsteady Navier-Stokes).

Key Insights and Ideas

  • L1 Loss for Robustness: The fundamental insight is that replacing the L2 loss with an L1 loss in the data fitting component of PINNs is a simple yet effective way to achieve robustness against outliers, mirroring the success of LAD in robust linear regression.
  • Two-Stage Filtering is Crucial for High Accuracy: While LAD-PINN offers robustness, the two-stage MAD-PINN framework significantly improves the accuracy of the reconstruction, especially for complex problems or when data corruption is severe. The data cleaning step is critical for the subsequent OLS-PINN to perform well.
  • MAD as a Robust Estimator for Filtering: The use of Median Absolute Deviation (MAD) for estimating the standard deviation of non-outlier data proves to be a robust and effective strategy for identifying and filtering outliers, particularly when prior knowledge about the noise distribution is limited.
  • Generalizability Across PDEs: The proposed methods demonstrate their effectiveness across a range of PDEs, including Poisson, wave, and Navier-Stokes equations, highlighting their broad applicability.

Key Results

  • Superior Performance on Corrupted Data: In scenarios with significant data corruption (Contaminated, Cauchy, Outlier, Mixed noise), LAD-PINN consistently outperforms standard OLS-PINN, often by orders of magnitude in terms of relative error. For instance, in the Poisson equation experiment with outliers, OLS-PINN's error can be over 100%, while LAD-PINN's error remains below 1%.
  • Two-Stage MAD-PINN Achieves High Accuracy: The two-stage MAD-PINN framework consistently achieves the best results, especially for reconstructing pressure fields in the Navier-Stokes equations, where single-stage methods often fail or yield very low accuracy. For example, in the unsteady N-S pressure reconstruction, MAD-PINN achieves errors below 5% in most cases, while OLS-PINN's errors can exceed 100%.
  • Robustness to Hyperparameter Choice: The MAD-based filtering in the two-stage approach is shown to be relatively insensitive to the choice of filter parameters (e.g., k=2.0, 2.5, 3.0), suggesting good generalizability.
  • Parameter Recovery: The methods are also effective in recovering unknown parameters of PDEs, even under corrupted data conditions.

Limitations and Future Work

  • Loss Weight Selection: The paper acknowledges that selecting the appropriate weight for the PDE loss term when transforming the problem into an unconstrained optimization remains a challenge, especially with varying noise scales. While the chosen weight (w=1.0) worked well in their experiments, more systematic methods for weight selection could be beneficial.
  • Non-Smoothness of LAD Loss: The non-smooth nature of the L1 loss function can pose computational challenges. The authors mention that reweighting methods from robust analysis could be used to address this, but this would introduce additional hyperparameters, which they aimed to minimize.
  • Computational Cost: While not explicitly quantified, the two-stage approach inherently involves more computational steps than a single-stage method.
  • Future Work: The authors suggest exploring methods to avoid explicit hyperparameter tuning for loss weights, potentially through penalty methods or augmented Lagrangian multipliers, although this might introduce other algorithmic hyperparameters. They also note the potential for handling the non-smoothness of the LAD loss more effectively.

Summary updated

Abstract

Physics-informed neural networks (PINNs) have been proposed to solve two main classes of problems: data-driven solutions and data-driven discovery of partial differential equations. This task becomes prohibitive when such data is highly corrupted due to the possible sensor mechanism failing. We propose the Least Absolute Deviation based PINN (LAD-PINN) to reconstruct the solution and recover unknown parameters in PDEs - even if spurious data or outliers corrupt a large percentage of the observations. To further improve the accuracy of recovering hidden physics, the two-stage Median Absolute Deviation based PINN (MAD-PINN) is proposed, where LAD-PINN is employed as an outlier detector followed by MAD screening out the highly corrupted data. Then the vanilla PINN or its variants can be subsequently applied to exploit the remaining normal data. Through several examples, including Poisson's equation, wave equation, and steady or unsteady Navier-Stokes equations, we illustrate the generalizability, accuracy and efficiency of the proposed algorithms for recovering governing equations from noisy and highly corrupted measurement data.
BibTeX
@misc{peng2022robust,
  author = {Wei Peng and Wen Yao and Weien Zhou and Xiaoya Zhang and Weijie Yao},
  title = {{Robust Regression with Highly Corrupted Data Via Physics Informed Neural Networks}},
  year = {2022},
  eprint = {2210.10646},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.10646},
}
Figure