Abstract
Data-driven discovery of differential equations has been an emerging research topic. We propose a novel algorithm subsampling-based threshold sparse Bayesian regression (SubTSBR) to tackle high noise and outliers. The subsampling technique is used for improving the accuracy of the Bayesian learning algorithm. It has two parameters: subsampling size and the number of subsamples. When the subsampling size increases with fixed total sample size, the accuracy of our algorithm goes up and then down. When the number of subsamples increases, the accuracy of our algorithm keeps going up. We demonstrate how to use our algorithm step by step and compare our algorithm with threshold sparse Bayesian regression (TSBR) for the discovery of differential equations. We show that our algorithm produces better results. We also discuss the merits of discovering differential equations from data and demonstrate how to discover models with random initial and boundary condition as well as models with bifurcations.
Introduction
The "holy grail" of data-driven science is the automated discovery of governing physical laws from raw observational data. Typically, these laws are expressed as differential equations. While techniques like Sparse Identification of Nonlinear Dynamics (SINDy) and LASSO have made great strides, they often struggle with "dirty" data—datasets characterized by high sensor noise or extreme outliers. Standard regression techniques attempt to minimize error across all data points, which means even a small percentage of outliers can significantly bias the resulting model, leading to the inclusion of spurious "ghost" terms that lack physical basis.
Methodology: Subsampling-based Threshold Sparse Bayesian Regression (SubTSBR)
To address these challenges, this work introduces Subsampling-based Threshold Sparse Bayesian Regression (SubTSBR). This algorithm integrates a robust Bayesian framework with a strategic subsampling approach to enhance model selection in the presence of noise.
Threshold Sparse Bayesian Regression (TSBR)
The foundational component of the method is TSBR, a hierarchical Bayesian framework that treats the weights of candidate functions as random variables. It employs Automatic Relevance Determination (ARD) to drive the weights of irrelevant functions to zero, effectively selecting the parsimonious model that best explains the data.
In this model, $\alpha_1, \alpha_2, \dots, \alpha_M$ are hyperparameters that control the precision of the weights $w_1, w_2, \dots, w_M$. The variable $\sigma^2$ represents the noise variance, and $\eta$ represents the observed data. By iteratively updating these parameters, the algorithm identifies which basis functions in a candidate library are truly significant.
Random Subsampling Strategy
The "Sub" in SubTSBR refers to the use of $L$ random subsets of the data, each of size $S$. The workflow proceeds as follows:
- Independent Inference: TSBR is executed on each of the $L$ subsets independently.
- Model Selection: Each resulting model is evaluated using a criterion based on the posterior variance of the weights.
- Final Selection: The model with the lowest criterion score—indicating the highest confidence—is selected as the final discovery.
The statistical intuition is that by sampling many small subsets, the algorithm is highly likely to encounter at least one subset that is relatively uncontaminated by outliers or extreme noise.
Experimental Evaluation
The SubTSBR algorithm was evaluated against four classic physical models corrupted with noise and outliers:
- Predator-Prey (Lotka-Volterra): Tested against high Gaussian noise levels.
- 2D Shallow Water Equations: Challenged with 2% extreme outliers (uniform distribution corruption).
- 1D Heat Diffusion: Demonstrated discovery capabilities under random initial and boundary conditions.
- Fish-Harvesting Model: Showcased the ability to handle systems with bifurcations, where qualitative behavior changes with parameter shifts.
Key Insights and Discussion
The Impact of Subsample Parameters
The performance of SubTSBR is sensitive to the subsample size ($S$). If $S$ is too small, the algorithm lacks sufficient information to distinguish signal from regular noise. Conversely, if $S$ is too large, the probability of including a "bad" outlier in every subset increases, degrading performance. There exists a "Goldilocks" zone for $S$ that maximizes the success rate of basis selection. In contrast, increasing the number of subsamples ($L$) consistently improves accuracy, albeit at a higher computational cost.
Discovery vs. Interpolation
A significant advantage of discovering the underlying differential equation rather than just fitting a curve is the ability to extrapolate. While black-box models like neural networks may interpolate well within the training time range, the discovered ODE/PDE allows for stable predictions far into the future and under varying initial conditions.
Conclusion
SubTSBR provides a robust alternative to traditional sparse regression methods for discovering physical laws from imperfect data. By leveraging the statistical power of subsampling within a Bayesian framework, it effectively filters out outliers that would otherwise derail the discovery process. While the computational overhead is higher due to repeated inference, the process is highly parallelizable. Future work will explore combining this approach with advanced denoising techniques and broader candidate libraries to tackle even more complex real-world systems.
Summary updated