Sparsifying priors for Bayesian uncertainty quantification in model discovery

Seth M. Hirsh, David A. Barajas‐Solano, J. Nathan KutzRoyal 2022 · DOI

TL;DR

This paper introduces UQ-SINDy, a novel Bayesian framework for discovering governing equations of dynamical systems from noisy and sparse data. By employing sparsity-promoting priors, UQ-SINDy not only identifies the underlying equations but also quantifies the uncertainty in the model coefficients and the probability of each term's inclusion, leading to more robust and interpretable models.

What Problem Does This Paper Address?

Discovering the underlying mathematical equations that govern complex dynamical systems from observed data is a fundamental challenge in many scientific fields. Traditional methods like the Sparse Identification of Nonlinear Dynamics (SINDy) excel at finding parsimonious models by assuming that only a few terms from a library of candidate functions are needed. However, these methods often struggle with noisy data and limited observations, leading to inaccurate models and an inability to quantify the uncertainty associated with the identified equations. This paper addresses the critical need for a model discovery approach that is robust to noise and data scarcity, while simultaneously providing a rigorous quantification of uncertainty in both the model parameters and the model structure itself.

Methods

The core of this work is the UQ-SINDy (Uncertainty Quantification SINDy) framework. It builds upon the SINDy algorithm by integrating it with sparse Bayesian inference. Instead of finding point estimates for the coefficients of the candidate functions (the SINDy coefficients), UQ-SINDy estimates their full posterior distributions. This is achieved by employing sparsity-promoting priors, specifically the spike and slab prior and the regularized horseshoe prior. These priors encourage most coefficients to be exactly zero (or very close to it), effectively performing model selection by assigning probabilities to the inclusion of each candidate function.

The process involves:

  1. Constructing a library of candidate functions (e.g., polynomials of the system's state variables).
  2. Formulating a Bayesian model where the SINDy coefficients and initial conditions are treated as random variables with prior distributions.
  3. Defining a data likelihood that accounts for measurement noise (e.g., Gaussian or lognormal).
  4. Employing Markov Chain Monte Carlo (MCMC) methods (specifically NUTS for the regularized horseshoe prior) to sample from the posterior distribution of the SINDy coefficients and initial conditions.

This Bayesian approach allows for the quantification of uncertainty in the coefficient values and, crucially, provides inclusion probabilities (or pseudo-probabilities for the horseshoe prior) for each term in the SINDy library, indicating how likely it is that a particular term is part of the true governing equations.

Datasets

The paper evaluates UQ-SINDy on three datasets:

  • Synthetic Lotka-Volterra model data: Simulated predator-prey dynamics with lognormal multiplicative noise. This dataset is used to demonstrate the method's ability to identify the correct dynamics and quantify uncertainty in the presence of noise.
  • Synthetic nonlinear oscillator model data: Simulated damped nonlinear oscillator dynamics with normal noise. This example highlights the method's robustness to noise and its ability to handle challenging scenarios with sparse data sampling and a large library of candidate functions.
  • Real-world lynx-hare population data: Historical data on lynx and hare pelts, which is known to be noisy and limited in terms of the number of observed cycles. This application showcases the framework's utility for real-world ecological modeling.

Key Insights and Ideas

  • Bayesian inference unlocks uncertainty quantification: Moving beyond point estimates to full posterior distributions is key to understanding the reliability of discovered models and coefficients.
  • Sparsity-promoting priors are crucial for model selection: The spike-and-slab and regularized horseshoe priors effectively force irrelevant terms to zero, enabling the identification of parsimonious and interpretable models.
  • Inclusion probabilities provide model interpretability: Quantifying the probability of each candidate function being part of the true model offers a direct measure of model confidence and aids in understanding the underlying system.
  • Robustness to noise and limited data: UQ-SINDy demonstrates a significant improvement over traditional SINDy, accurately identifying dynamics even with orders-of-magnitude less data and substantial noise.
  • Model indeterminacy can be identified: The framework can reveal situations where multiple models might explain the data equally well, a common issue in complex systems.

Key Results

  • UQ-SINDy successfully identified the correct governing equations for all tested synthetic and real-world datasets, even in the presence of significant noise and limited data.
  • The method consistently outperformed the original SINDy algorithm, which struggled to identify accurate dynamics under noisy conditions.
  • For the Lotka-Volterra model, UQ-SINDy required significantly less data (as few as 21 time points) to identify a robust and sparse model compared to existing methods.
  • The inclusion probabilities (or pseudo-probabilities) provided by UQ-SINDy clearly distinguished relevant terms from irrelevant ones, offering a strong indication of model structure.
  • The posterior predictive distributions (PPDs) generated by UQ-SINDy provided accurate reconstructions and forecasts, with credibility intervals that effectively captured the uncertainty.

Limitations and Future Work

  • Scalability: The paper acknowledges that the computational cost of MCMC sampling can be a limitation for very large SINDy libraries (e.g., libraries of rational functions).
  • Variational Inference: As a potential solution for scalability, the authors suggest exploring variational inference methods, which have shown promise for high-dimensional models.
  • Coordinate Discovery: The current framework assumes the relevant coordinates for a sparse representation are known. Future work could extend UQ-SINDy to simultaneously learn parsimonious governing equations and the associated sparsity-inducing coordinate transformations, building on recent research in this area.

Summary updated

Abstract

We propose a probabilistic model discovery method for identifying ordinary differential equations governing the dynamics of observed multivariate data. Our method is based on the <i>sparse identification of nonlinear dynamics</i> (SINDy) framework, where models are expressed as sparse linear combinations of pre-specified candidate functions. Promoting parsimony through sparsity leads to interpretable models that generalize to unknown data. Instead of targeting point estimates of the SINDy coefficients, we estimate these coefficients via sparse Bayesian inference. The resulting method, <i>uncertainty quantification SINDy</i> (UQ-SINDy), quantifies not only the uncertainty in the values of the SINDy coefficients due to observation errors and limited data, but also the probability of inclusion of each candidate function in the linear combination. UQ-SINDy promotes robustness against observation noise and limited data, interpretability (in terms of model selection and inclusion probabilities) and generalization capacity for out-of-sample forecast. Sparse inference for UQ-SINDy employs Markov chain Monte Carlo, and we explore two sparsifying priors: the <i>spike and slab prior</i>, and the <i>regularized horseshoe prior</i>. UQ-SINDy is shown to discover accurate models in the presence of noise and with orders-of-magnitude less data than current model discovery methods, thus providing a transformative method for real-world applications which have limited data.
BibTeX
@inproceedings{hirsh2022sparsifying,
  author = {Seth M. Hirsh and David A. Barajas‐Solano and J. Nathan Kutz},
  title = {{Sparsifying priors for Bayesian uncertainty quantification in model discovery}},
  booktitle = {Royal},
  year = {2022},
  doi = {10.1098/rsos.211823},
}
Figure