Equation Discovery with Bayesian Spike-and-Slab Priors and Efficient Kernels

Da Long, Wei W. Xing, Aditi S. Krishnapriyan, Robert M. Kirby, Shandian Zhe, Michael W. Mahoney

Abstract

Discovering governing equations from data is critical for scientific and engineering applications, yet existing methods struggle with data sparsity, noise, and high computational costs. We propose KBASS (Kernel learning and BAyesian Spike-and-Slab priors), a novel framework that combines flexible kernel regression with Bayesian variable selection. By leveraging tensor algebra and Kronecker product constructions, KBASS enables efficient posterior inference and uncertainty quantification. Our results across various ODE and PDE benchmarks demonstrate superior robustness to noise and significantly faster training times compared to state-of-the-art methods. The code is available at https://github.com/long-da/KBASS.

Introduction

In fields ranging from fluid dynamics to ecology, researchers often possess observational data but lack the exact mathematical equations—typically Ordinary Differential Equations (ODEs) or Partial Differential Equations (PDEs)—that describe the system's underlying physics. Discovering these "governing equations" is a fundamental challenge in data-driven science.

Current state-of-the-art methods face three major hurdles:

  1. Data Quality: Methods like SINDy rely on numerical differentiation, which is highly sensitive to noise and sparsity.
  2. Computational Cost: Physics-Informed Neural Networks (PINNs) avoid numerical differentiation but are notoriously difficult to tune and expensive to train.
  3. Uncertainty Quantification: Most methods provide a single "best guess" without indicating confidence, which is problematic in scientific applications where understanding the margin of error is critical.

Methodology: The KBASS Framework

KBASS addresses these limitations through three core technical pillars:

Kernel Regression for Function Estimation

Instead of using deep neural networks, KBASS employs kernel regression to estimate target functions and their derivatives. Kernels act as global interpolants that are naturally robust to noise, providing a smoother and more reliable approximation of the underlying physics than local numerical differentiation.

Bayesian Spike-and-Slab Priors

To identify the simplest governing equation, the model must select a few relevant terms from a large dictionary of candidates (e.g., $u, u^2, u_{xx}$). KBASS utilizes a spike-and-slab prior, the gold standard for Bayesian variable selection. Each term is treated as a binary choice: it is either included in the equation (the "slab") or effectively zero (the "spike"). This allows the model to calculate the posterior probability of each term's relevance, eliminating the need for arbitrary manual thresholding.

Computational Efficiency via Tensor Algebra

Kernel methods typically suffer from $O(N^3)$ complexity due to matrix inversions. KBASS overcomes this by placing function values on a Cartesian mesh. This structure allows for the use of Kronecker products and tensor algebra, reducing computational costs from cubic to linear relative to the mesh size. Posterior inference is performed efficiently using a specialized Expectation-Propagation Expectation-Maximization (EP-EM) algorithm.

Experimental Results and Discussion

The authors evaluated KBASS against benchmarks including the Van der Pol oscillator, the chaotic Lorenz 96 system, and PDEs such as Burger’s, Kuramoto-Sivashinsky (KS), and Allen-Cahn equations.

Robustness and Accuracy

KBASS demonstrates exceptional resilience to noise. While traditional methods often fail when noise levels exceed 5-10%, KBASS successfully recovered governing equations with up to 20% noise.

Performance comparison on Burger's equation under varying noise levels. KBASS (bold) consistently achieves the lowest error and highest success rate.

Threshold-Free Discovery and Uncertainty

A significant advantage of the Bayesian approach is the ability to identify terms with very small physical constants. For instance, in the Allen-Cahn equation, the diffusion rate is approximately $10^{-4}$. While threshold-based methods might discard such terms as noise, KBASS identifies them based on their high posterior probability.

Discovery of the Allen-Cahn equation. KBASS accurately identifies terms with very small coefficients and provides full posterior distributions for uncertainty quantification.

Computational Speed

KBASS is consistently faster than competing Bayesian methods like BSL (1.9x to 7.3x faster). Compared to PINN-SR, the speedup is even more dramatic; on the KS equation, PINN-SR took 20 times longer and failed to converge to the correct solution.

Conclusion

KBASS provides a robust, efficient, and probabilistic framework for equation discovery. By integrating kernel learning with spike-and-slab priors, it handles sparse and noisy data where other methods fail. Currently, the method relies on a pre-specified dictionary of candidate operators. Future work may involve dynamically expanding this dictionary or incorporating symbolic regression to "invent" new terms, further reducing the reliance on prior domain knowledge.

Summary updated

Abstract

Discovering governing equations from data is important to many scientific and engineering applications. Despite promising successes, existing methods are still challenged by data sparsity and noise issues, both of which are ubiquitous in practice. Moreover, state-of-the-art methods lack uncertainty quantification and/or are costly in training. To overcome these limitations, we propose a novel equation discovery method based on Kernel learning and BAyesian Spike-and-Slab priors (KBASS). We use kernel regression to estimate the target function, which is flexible, expressive, and more robust to data sparsity and noises. We combine it with a Bayesian spike-and-slab prior — an ideal Bayesian sparse distribution — for effective operator selection and uncertainty quantification. We develop an expectation-propagation expectation-maximization (EP-EM) algorithm for efficient posterior inference and function estimation. To overcome the computational challenge of kernel regression, we place the function values on a mesh and induce a Kronecker product construction, and we use tensor algebra to enable efficient computation and optimization. We show the advantages of KBASS on a list of benchmark ODE and PDE discovery tasks. The code is available at \url{https://github.com/long-da/KBASS}.
BibTeX
@inproceedings{long2024equation,
  author = {Da Long and Wei W. Xing and Aditi S. Krishnapriyan and Robert M. Kirby and Shandian Zhe and Michael W. Mahoney},
  title = {{Equation Discovery with Bayesian Spike-and-Slab Priors and Efficient Kernels}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=aoXvJ9bsc1},
}
Figure