Introduction
The ability to automatically discover the governing differential equations of a physical system from observational data represents a significant leap in scientific computing. Techniques such as Sparse Identification of Nonlinear Dynamics (SINDy) have demonstrated success in identifying parsimonious models that balance complexity and accuracy. However, the practical application of these methods is often hindered by the realities of experimental data: high-frequency noise and intermittent outliers. While traditional smoothing techniques can mitigate low-level noise, they often fail or introduce bias when faced with significant corruption. This paper discusses SubTSBR (Subsampling-based Threshold Sparse Bayesian Regression), a novel framework designed to maintain robustness in these challenging environments.
The SubTSBR Framework
SubTSBR builds upon the foundation of Threshold Sparse Bayesian Regression (TSBR). The fundamental goal is to solve a linear system $\eta = \Theta w + \epsilon$, where $\eta$ represents the time derivatives of the state variables, $\Theta$ is a library of candidate nonlinear functions, $w$ is a vector of coefficients, and $\epsilon$ is noise. To achieve a "sparse" model—one that includes only the physically relevant terms—the algorithm employs a hierarchical Bayesian approach.
As shown in Figure 1, the model treats the weights $w$ as random variables governed by hyperparameters $\alpha$. By optimizing these hyperparameters, the algorithm can effectively "turn off" irrelevant terms in the library (setting their corresponding $w_i$ to near zero), thereby discovering the simplest mathematical structure that explains the data.
The Innovation: Random Subsampling
The "secret sauce" that distinguishes SubTSBR from its predecessors is the integration of a random subsampling strategy. Standard regression methods utilize the entire dataset simultaneously, which makes them vulnerable to outliers that can skew the entire optimization landscape. SubTSBR addresses this through a four-step iterative process:
- Subsampling: Instead of the full dataset of size $N$, the algorithm randomly selects a smaller subset of size $S$.
- Local Regression: Sparse Bayesian regression is performed exclusively on this subset.
- Evaluation: The resulting model is evaluated using a model-selection criterion (an "evidence" score) that measures the likelihood of the model given the data.
- Selection: This process is repeated $L$ times. The model that achieves the highest confidence (lowest score) across all trials is selected as the final governing equation.
The intuition behind this approach is statistical: by repeatedly looking at small windows of data, the algorithm increases the probability of encountering a "clean" subset that is relatively free of outliers or extreme noise. Once a clean subset is found, the underlying physical laws emerge clearly, allowing the algorithm to ignore the corrupted data points that would otherwise lead to model misspecification.
Experimental Validation and Results
The researchers validated SubTSBR across four distinct numerical simulations, each representing a specific challenge in system identification.
Predator-Prey and Shallow Water Dynamics
In the classic Lotka-Volterra predator-prey model, SubTSBR successfully identified the exact mathematical terms even under high Gaussian noise, where standard TSBR failed to eliminate spurious terms. More impressively, in a 2D Shallow Water Equation simulation corrupted with a 2% outlier rate, SubTSBR maintained high structural accuracy. Traditional least-squares methods in this scenario produced physically impossible predictions, whereas SubTSBR's subsampling effectively filtered out the "garbage" data points.
Generalization in Heat Diffusion
To test the algorithm's ability to discover partial differential equations (PDEs), the authors applied it to 1D heat diffusion. The model was trained on data with random initial and boundary conditions. SubTSBR discovered the governing PDE with such precision that it could extrapolate the system's state 10 seconds beyond the training window with a Mean Squared Error (MSE) of only $10^{-5}$. In contrast, standard regression methods failed to capture the true physics, leading to an MSE exceeding $28,000,000$ during extrapolation.
Key Insights: The "Sweet Spot"
The study revealed a critical trade-off regarding the subsampling size ($S$). If $S$ is too small, the model lacks sufficient information to identify the dynamics. If $S$ is too large, the probability of including outliers in every subset increases, degrading performance. The authors identified an optimal "sweet spot" for $S$ and provided an empirical formula for an adjusted model-selection criterion to help automate this tuning process. Furthermore, while increasing the number of trials ($L$) increases computational cost, it consistently improves the robustness of the discovery process.
Conclusion
SubTSBR represents a significant advancement in the data-driven discovery of physical laws. By combining the sparsity-promoting power of Bayesian inference with a robust subsampling strategy, it overcomes the limitations of traditional methods in the presence of high noise and outliers. While the computational overhead is higher than single-pass regression, the ability to discover the actual underlying physics—enabling reliable long-term extrapolation—makes it a powerful tool for researchers working with imperfect real-world data.
Summary updated