Robust subsampling-based sparse Bayesian inference to tackle four challenges (large noise, outliers, data integration, and extrapolation) in the discovery of physical laws from data

Sheng Zhang, Guang Lin

Abstract

Automated discovery of governing differential equations from experimental data is a cornerstone of modern data-driven physics. However, real-world data is often corrupted by high-variance noise, outliers, and inconsistent experimental conditions. We present SubTSBR (Subsampling-based Threshold Sparse Bayesian Regression), a robust framework that combines hierarchical Bayesian model selection with an ensemble subsampling strategy. By iteratively evaluating small subsets of data, SubTSBR effectively filters out corruptions to identify the true underlying physical laws. Our results across various physical systems—including predator-prey dynamics and fluid flow—demonstrate superior performance in noise robustness, outlier resistance, and long-term extrapolation compared to standard regression techniques.

Introduction

The quest to derive physical laws, such as the Heat Equation or the Navier-Stokes equations, directly from observations has led to the development of sparse identification methods like SINDy and LASSO. While effective on clean, synthetic datasets, these methods often falter when faced with "dirty" data. In industrial and laboratory settings, measurements are frequently plagued by Gaussian noise, sensor-induced outliers, and varying initial conditions across pooled datasets. Standard regression tends to overfit these errors, leading to the discovery of spurious terms that lack physical meaning.

Methodology: SubTSBR

The proposed SubTSBR algorithm addresses these challenges by integrating Threshold Sparse Bayesian Regression (TSBR) with a robust subsampling ensemble strategy.

Threshold Sparse Bayesian Regression (TSBR)

At its core, TSBR utilizes a hierarchical Bayesian framework, specifically the Relevance Vector Machine (RVM), to estimate the coefficients (weights) of potential terms in a candidate library of differential equations.

Hierarchical Bayesian model for Threshold Sparse Bayesian Regression (TSBR).

As shown in Figure 1, the model treats the weights $w_i$ as random variables governed by hyperparameters $\alpha_i$. Through Automatic Relevance Determination (ARD), hyperparameters associated with non-essential terms are driven to infinity, effectively pruning those terms from the model. This ensures a sparse and interpretable representation of the physical system.

Subsampling Ensemble Strategy

The "Sub" in SubTSBR introduces a strategy to handle outliers and high noise:

  1. Random Subsampling: Instead of fitting the model to the entire dataset, the algorithm selects a small subset of size $S$.
  2. Iteration: This process is repeated $L$ times, generating a library of candidate models.
  3. Model Selection: A specific criterion—the ratio of estimated variance to the square of the mean for each weight—is used to evaluate model confidence.
  4. Final Selection: The candidate model with the lowest criterion (highest confidence) is selected.

The underlying logic is statistical: while the full dataset may be heavily corrupted, there is a high probability that at least one small subset remains relatively "clean," allowing the Bayesian framework to identify the true physics.

Experimental Evaluation

The authors evaluated SubTSBR across four diverse physical models:

  • Lotka-Volterra (Predator-Prey): Testing robustness against high-variance Gaussian noise.
  • Shallow Water Equations: A 2D system used to test resistance to 2% uniform outliers.
  • 1D Heat Diffusion: Testing data integration from 20 independent experiments with varying initial/boundary conditions.
  • Fish-Harvesting Problem: Evaluating the ability to generalize across bifurcations and qualitative regime shifts.

Key Results and Insights

SubTSBR demonstrated significant advantages over traditional sparse regression:

  • Outlier and Noise Robustness: In the shallow water equations with 2% outliers, SubTSBR maintained coefficient accuracy within 1%, whereas standard TSBR errors reached 50%. In the predator-prey model, SubTSBR avoided "ghost" terms that standard methods incorrectly identified.
  • Extrapolation Capability: Because SubTSBR discovers the fundamental differential equation rather than just fitting a curve, it excels at predicting future states. In the heat diffusion test, SubTSBR achieved a Mean Squared Error (MSE) of $10^{-5}$ for long-term predictions, while traditional least-squares regression failed with an MSE of $10^7$.
  • Data Integration: The method successfully pooled data from multiple experiments with unknown initial conditions, provided the underlying physical law remained constant.

Conclusion

SubTSBR provides a robust pathway for discovering physical laws from imperfect real-world data. By leveraging the "clean subset" hypothesis through Bayesian model selection, it overcomes the sensitivity to noise and outliers that hampers traditional sparse identification methods. Future work will focus on reducing the computational overhead of the ensemble process and extending the framework to handle structured or systematic experimental biases.

Reference: Zhang, S., & Lin, G. (2020). SubTSBR to tackle high noise and outliers for data-driven discovery of differential equations. arXiv:1907.07788v4.

Summary updated

Abstract

Data-driven discovery of differential equations has been an emerging research topic. We propose a novel algorithm subsampling-based threshold sparse Bayesian regression (SubTSBR) to tackle high noise and outliers. The subsampling technique is used for improving the accuracy of the Bayesian learning algorithm. It has two parameters: subsampling size and the number of subsamples. When the subsampling size increases with fixed total sample size, the accuracy of our algorithm goes up and then down. When the number of subsamples increases, the accuracy of our algorithm keeps going up. We demonstrate how to use our algorithm step by step and compare our algorithm with threshold sparse Bayesian regression (TSBR) for the discovery of differential equations. We show that our algorithm produces better results. We also discuss the merits of discovering differential equations from data and demonstrate how to discover models with random initial and boundary condition as well as models with bifurcations. The numerical examples are: (1) predator-prey model with noise, (2) shallow water equations with outliers, (3) heat diffusion with random initial and boundary condition, and (4) fish-harvesting problem with bifurcations.
BibTeX
@misc{zhang2019robust,
  author = {Sheng Zhang and Guang Lin},
  title = {{Robust subsampling-based sparse Bayesian inference to tackle four challenges (large noise, outliers, data integration, and extrapolation) in the discovery of physical laws from data}},
  year = {2019},
  eprint = {1907.07788},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.07788},
}
Figure