Scientific discovery is increasingly moving from the chalkboard to the computer. While we have centuries of physical laws expressed as differential equations, many complex modern systems—from climate patterns to biological networks—lack a known "source code." Data-driven equation discovery aims to find these governing equations automatically from observations. However, real-world data is messy: it’s often sparse, noisy, and expensive to collect.
A new framework, KBASS (Kernel learning and BAyesian Spike-and-Slab priors), addresses these challenges by combining the flexibility of kernel regression with the "gold standard" sparsity of Bayesian spike-and-slab priors. This approach achieves superior accuracy and uncertainty quantification while remaining significantly faster than existing Bayesian methods.
The Challenge of Equation Discovery
Discovering the underlying equations of a system usually involves picking the best terms from a "dictionary" of candidates (like $u, u^2, \partial_x u$, etc.). Current methods face a "trilemma" of issues:
- Sensitivity to Noise: Methods like SINDy rely on numerical differentiation, which amplifies noise and fails with sparse data.
- Optimization Complexity: Neural network-based approaches (like PINNs) are difficult to tune, computationally expensive, and lack uncertainty quantification.
- Computational Scaling: Existing Bayesian methods scale poorly ($O(N^3)$) as data volume increases.
The KBASS Framework
KBASS is built on three main pillars:
- Kernel Regression: Instead of neural networks, KBASS uses kernel regression to approximate the target function. Kernels are naturally smooth and allow for analytical derivatives, making them robust to noise and sparsity.
- Bayesian Spike-and-Slab Priors: This probabilistic method enforces sparsity. Each candidate term is assigned a "selection indicator." This allows the model to calculate the actual probability that a specific term belongs in the equation, providing built-in uncertainty quantification.
- The Kronecker Product Trick: To solve the $O(N^3)$ scaling issue, KBASS places function values on a mesh, allowing massive kernel matrices to be decomposed into Kronecker products. This reduces complexity from cubic to essentially linear relative to the number of mesh points.
Key Insights: Probabilistic Pruning
Unlike methods that require manual weight thresholds, KBASS uses posterior probabilities. This is crucial for equations like Allen-Cahn, where physical coefficients might be extremely small (e.g., $10^{-4}$). A hard threshold would likely delete such terms, but KBASS identifies them through their posterior distributions.
Experimental Results
The authors evaluated KBASS against benchmarks including the Van der Pol oscillator, Lorenz 96 system, and various PDEs.
- Sample Efficiency: KBASS recovered the Lorenz 96 system using only 12 training examples, whereas SINDy required 250.
- Robustness: In tests with 20% noise, KBASS successfully recovered equations where other methods failed.
- Speed: KBASS showed speedups of up to 7.3x over state-of-the-art Bayesian Spline Learning (BSL).
Conclusion
KBASS represents a significant step forward in automated scientific discovery. By integrating kernel methods with Bayesian sparsity and efficient tensor algebra, it provides a robust, fast, and interpretable framework for uncovering the laws of physics from limited and noisy data. Future work may involve dynamic dictionary expansion to discover functional forms not pre-specified by the user.
Full Paper: Equation Discovery with Bayesian Spike-and-Slab Priors and Efficient Kernels (2024). arXiv:2310.05387v2
Summary updated