Abstract
Data-driven discovery of differential equations has been an emerging research topic. We propose a novel algorithm, subsampling-based threshold sparse Bayesian regression (SubTSBR), to tackle high noise and outliers. The subsampling technique is used for improving the accuracy of the Bayesian learning algorithm. It has two parameters: subsampling size and the number of subsamples. When the subsampling size increases with fixed total sample size, the accuracy of our algorithm goes up and then down. When the number of subsamples increases, the accuracy of our algorithm keeps going up. We demonstrate how to use our algorithm step by step and compare our algorithm with threshold sparse Bayesian regression (TSBR) for the discovery of differential equations. We show that our algorithm produces better results. We also discuss the merits of discovering differential equations from data and demonstrate how to discover models with random initial and boundary conditions as well as models with bifurcations. The numerical examples include a predator-prey model with noise, shallow water equations with outliers, heat diffusion with random initial and boundary conditions, and a fish-harvesting problem with bifurcations.
Introduction
The "holy grail" of data-driven science is to transform raw observations into clean, interpretable physical laws. While methods like SINDy (Sparse Identification of Nonlinear Dynamics) and Lasso have made significant strides, they often struggle with real-world data challenges. This paper addresses four primary "demons" that hinder traditional discovery methods: large noise in sensors, extreme non-Gaussian outliers, the need for data integration across disparate experiments, and the difficulty of extrapolation across system bifurcations. SubTSBR is designed to bridge the gap between idealized laboratory data and the "dirty" data encountered in industrial and field applications.
Methodology
The core innovation of this work is Subsampling-based Threshold Sparse Bayesian Regression (SubTSBR). The method rests on two pillars: sparse Bayesian inference and a strategic subsampling approach.
Sparse Bayesian Inference
Unlike standard regression techniques that provide point estimates for coefficients, the Bayesian approach calculates a posterior distribution. This allows the model to quantify uncertainty through "error bars" for each term in the candidate equation. The hierarchical structure of this framework is illustrated below.
The Subsampling Strategy
Instead of training on the entire dataset simultaneously, SubTSBR randomly selects a subset of data points and performs Bayesian regression over $L$ iterations. The algorithm defines a selection metric based on the normalized variance of the weights (the "error bar"). After $L$ loops, the model with the smallest error bar is selected. This strategy leverages subsampling for accuracy rather than speed; by examining many small subsets, the algorithm can identify a "clean" subset that is not dominated by outliers or extreme noise.
Experimental Setup
The authors evaluated SubTSBR using four numerical playgrounds:
- Predator-Prey (Lotka-Volterra): Testing robustness against high Gaussian noise.
- 2D Shallow Water Equations: Testing resistance to 2% extreme outliers (random values added to height and velocity).
- 1D Heat Diffusion: Testing data integration using 20 different solutions generated from random initial and boundary conditions.
- Fish-Harvesting Model: Testing extrapolation capabilities across bifurcations where qualitative system behavior changes abruptly.
Key Insights and Results
The "Goldilocks" Subsampling Size
Research indicated that the subsampling size $S$ must be carefully balanced. If $S$ is too small, there is insufficient information; if too large, it is guaranteed to include corrupted data. A moderate size is optimal for filtering out noise and outliers. The posterior variance serves as a "compass," allowing the algorithm to find the most confident (and likely correct) model without prior knowledge of the true law.
Performance and Robustness
- Noise and Outliers: In the predator-prey example, SubTSBR correctly identified the governing equations, whereas SINDy and Lasso produced over-complicated models. For the shallow water equations, SubTSBR perfectly recovered the physical structure despite significant data corruption.
- Extrapolation and Bifurcations: Because the method recovers the functional form of the law rather than just fitting a curve, it correctly predicts system collapses or shifts (bifurcations) even if the training data only showed stable behavior. In the fish-harvesting problem, SubTSBR achieved a near-zero Mean Squared Error ($MSE \approx 10^{-5}$), significantly outperforming traditional regression.
- Efficiency: Despite the iterative nature of the algorithm, discovering a complex dynamical system took approximately 15 seconds on a standard laptop.
Conclusion
SubTSBR offers a robust framework for discovering physical laws from imperfect data. While it introduces computational overhead by running Bayesian inference multiple times and remains dependent on numerical derivatives, the gains in accuracy and extrapolation capability are substantial. Future work may focus on automating hyperparameter tuning and improving derivative calculation for extremely sparse time-series data.
Summary updated