Equation Discovery with Bayesian Spike-and-Slab Priors and Efficient Kernels

Da Long, Wei W. Xing, Aditi S. Krishnapriyan, Robert M. Kirby, Shandian Zhe, Michael W. Mahoney

Scientific discovery is increasingly moving from the chalkboard to the computer. While we have centuries of physical laws expressed as differential equations, many complex modern systems—from climate patterns to biological networks—lack a known "source code." Data-driven equation discovery aims to find these governing equations automatically from observations. However, real-world data is messy: it’s often sparse, noisy, and expensive to collect.

A new framework, KBASS (Kernel learning and BAyesian Spike-and-Slab priors), addresses these challenges by combining the flexibility of kernel regression with the "gold standard" sparsity of Bayesian spike-and-slab priors. This approach achieves superior accuracy and uncertainty quantification while remaining significantly faster than existing Bayesian methods.

The Challenge of Equation Discovery

Discovering the underlying equations of a system usually involves picking the best terms from a "dictionary" of candidates (like $u, u^2, \partial_x u$, etc.). Current methods face a "trilemma" of issues:

  1. Sensitivity to Noise: Methods like SINDy rely on numerical differentiation, which amplifies noise and fails with sparse data.
  2. Optimization Complexity: Neural network-based approaches (like PINNs) are difficult to tune, computationally expensive, and lack uncertainty quantification.
  3. Computational Scaling: Existing Bayesian methods scale poorly ($O(N^3)$) as data volume increases.

The KBASS Framework

KBASS is built on three main pillars:

  • Kernel Regression: Instead of neural networks, KBASS uses kernel regression to approximate the target function. Kernels are naturally smooth and allow for analytical derivatives, making them robust to noise and sparsity.
  • Bayesian Spike-and-Slab Priors: This probabilistic method enforces sparsity. Each candidate term is assigned a "selection indicator." This allows the model to calculate the actual probability that a specific term belongs in the equation, providing built-in uncertainty quantification.
  • The Kronecker Product Trick: To solve the $O(N^3)$ scaling issue, KBASS places function values on a mesh, allowing massive kernel matrices to be decomposed into Kronecker products. This reduces complexity from cubic to essentially linear relative to the number of mesh points.

Key Insights: Probabilistic Pruning

Unlike methods that require manual weight thresholds, KBASS uses posterior probabilities. This is crucial for equations like Allen-Cahn, where physical coefficients might be extremely small (e.g., $10^{-4}$). A hard threshold would likely delete such terms, but KBASS identifies them through their posterior distributions.

KBASS performance on the Allen-Cahn equation, demonstrating accurate recovery and posterior distributions for coefficients, including very small terms.

Experimental Results

The authors evaluated KBASS against benchmarks including the Van der Pol oscillator, Lorenz 96 system, and various PDEs.

  • Sample Efficiency: KBASS recovered the Lorenz 96 system using only 12 training examples, whereas SINDy required 250.
  • Robustness: In tests with 20% noise, KBASS successfully recovered equations where other methods failed.
  • Speed: KBASS showed speedups of up to 7.3x over state-of-the-art Bayesian Spline Learning (BSL).
Performance comparison on the Lorenz 96 system across varying noise levels and sample sizes, highlighting KBASS's superior sample efficiency.

Conclusion

KBASS represents a significant step forward in automated scientific discovery. By integrating kernel methods with Bayesian sparsity and efficient tensor algebra, it provides a robust, fast, and interpretable framework for uncovering the laws of physics from limited and noisy data. Future work may involve dynamic dictionary expansion to discover functional forms not pre-specified by the user.

Full Paper: Equation Discovery with Bayesian Spike-and-Slab Priors and Efficient Kernels (2024). arXiv:2310.05387v2

Summary updated

Abstract

Discovering governing equations from data is important to many scientific and engineering applications. Despite promising successes, existing methods are still challenged by data sparsity and noise issues, both of which are ubiquitous in practice. Moreover, state-of-the-art methods lack uncertainty quantification and/or are costly in training. To overcome these limitations, we propose a novel equation discovery method based on Kernel learning and BAyesian Spike-and-Slab priors (KBASS). We use kernel regression to estimate the target function, which is flexible, expressive, and more robust to data sparsity and noises. We combine it with a Bayesian spike-and-slab prior -- an ideal Bayesian sparse distribution -- for effective operator selection and uncertainty quantification. We develop an expectation-propagation expectation-maximization (EP-EM) algorithm for efficient posterior inference and function estimation. To overcome the computational challenge of kernel regression, we place the function values on a mesh and induce a Kronecker product construction, and we use tensor algebra to enable efficient computation and optimization. We show the advantages of KBASS on a list of benchmark ODE and PDE discovery tasks.
BibTeX
@inproceedings{long2024equation,
  author = {Da Long and Wei W. Xing and Aditi S. Krishnapriyan and Robert M. Kirby and Shandian Zhe and Michael W. Mahoney},
  title = {{Equation Discovery with Bayesian Spike-and-Slab Priors and Efficient Kernels}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=aoXvJ9bsc1},
}
Figure