General Bayesian updating and the loss-likelihood bootstrap

Simon Lyddon, Chris Holmes, Stephen G. Walker

TL;DR

This paper introduces the "loss-likelihood bootstrap," a novel method for generating Bayesian posterior samples that works even when the underlying data-generating model is unknown or misspecified. It achieves this by connecting a Bayesian nonparametric perspective to existing bootstrap techniques and then uses this to calibrate a more general Bayesian updating framework, ensuring it captures the right amount of information from the data.

What Problem Does This Paper Address?

Bayesian inference provides a powerful framework for reasoning under uncertainty, but its validity often relies on the assumption that the statistical model used to describe the data is "well-specified." In reality, models are frequently misspecified, meaning they don't perfectly capture the true data-generating process. This misspecification can lead to posterior distributions that concentrate on incorrect parameter values and can make it difficult to quantify uncertainty reliably.

Traditional Bayesian updating requires specifying a full probabilistic model for the data. However, this can be computationally prohibitive or even impossible if the true data-generating mechanism is unknown. The paper addresses this by proposing a method that can update beliefs about parameters that minimize an expected loss, without needing to construct a complete global probability model. A key challenge is then how to "calibrate" this general Bayesian updating approach so that it learns appropriately from the data, especially when the model is misspecified.

Methods

The paper introduces the loss-likelihood bootstrap, which extends the existing weighted likelihood bootstrap. The core idea is to view the parameter of interest not as the maximizer of a likelihood within a parametric family, but as the minimizer of an expected loss under an unknown sampling distribution. This perspective allows for a Bayesian nonparametric derivation of the weighted likelihood bootstrap, which then enables its extension to parameters defined by expected loss minimization.

The authors then connect this to general Bayesian updating, a framework that updates beliefs based on a loss function and a "loss scale" parameter ($w$). This parameter controls how much the data influences the posterior. The paper's main methodological contribution is a method to calibrate this loss scale ($w$) by matching the asymptotic Fisher information of the general Bayesian posterior to that of the loss-likelihood bootstrap. This calibration ensures that the general Bayesian posterior captures the same amount of information from the data as the more robust loss-likelihood bootstrap.

Datasets

The paper evaluates its methods on synthetic datasets and a real-world dataset:

  • Synthetic dataset: Used to demonstrate the method's performance in binary classification, comparing it to a standard support vector machine.
  • Statlog German Credit dataset: A real-world dataset with 24 covariates and 1000 observations, used for a more challenging binary classification task.

Key Insights and Ideas

  • Bridging Parametric and Nonparametric: The paper elegantly shows how a method (weighted likelihood bootstrap) traditionally derived from a parametric perspective can be re-interpreted and generalized from a nonparametric viewpoint, making it more robust to model misspecification.
  • Loss Minimization as a General Framework: Framing inference around minimizing expected loss provides a flexible way to define parameters of interest, moving beyond simple maximum likelihood estimation.
  • Calibrating General Bayesian Updates: The core challenge of general Bayesian updating is setting the loss scale ($w$). Matching asymptotic Fisher information with a robust bootstrap method provides a principled way to do this, ensuring appropriate learning from data.
  • Robustness to Model Misspecification: The loss-likelihood bootstrap and the calibrated general Bayesian posterior are shown to be more reliable when the assumed parametric model is incorrect, offering a way to perform inference honestly about model inadequacy.

Key Results

  • The loss-likelihood bootstrap provides a method for generating approximate Bayesian posterior samples for parameters minimizing expected loss, even when the data-generating distribution is unknown.
  • A method is proposed to calibrate the loss scale ($w$) in general Bayesian updating by matching asymptotic Fisher information with the loss-likelihood bootstrap.
  • In binary classification examples (synthetic and German Credit dataset), the calibrated general Bayesian posterior and the loss-likelihood bootstrap yield results comparable to standard support vector machines, with the Bayesian methods providing uncertainty quantification.
  • The calibration method for $w$ is shown to recover the correct Bayesian learning rate when the model is well-specified.

Limitations and Future Work

  • Computational Cost: While the loss-likelihood bootstrap is trivially parallelizable, the optimization required to find the parameter minimizing the expected loss can be computationally intensive, especially for complex loss functions or high-dimensional parameters.
  • Prior Specification: The loss-likelihood bootstrap itself is prior-free, which might be unappealing to subjective Bayesians. The paper discusses ideas for incorporating priors, such as importance weighting, but notes potential issues like density estimation and weight degeneracy.
  • Choice of Loss Function: The paper uses a smoothed hinge loss for classification. The choice and properties of the loss function are crucial, and the paper acknowledges that the hinge loss itself can be problematic due to its non-convexity.
  • Future Work: The paper suggests exploring alternative calibration methods and further investigating the properties of the loss-likelihood bootstrap and calibrated general Bayesian posteriors. The discussion also touches upon the potential for adapting these methods to cost-sensitive classification problems.

Summary updated

Abstract

In this paper we revisit the weighted likelihood bootstrap, a method that generates samples from an approximate Bayesian posterior of a parametric model. We show that the same method can be derived, without approximation, under a Bayesian nonparametric model with the parameter of interest defined through minimizing an expected negative loglikelihood under an unknown sampling distribution. This interpretation enables us to extend the weighted likelihood bootstrap to posterior sampling for parameters minimizing an expected loss. We call this method the loss-likelihood bootstrap, and we make a connection between it and general Bayesian updating, which is a way of updating prior belief distributions that does not need the construction of a global probability model, yet requires the calibration of two forms of loss function. The loss-likelihood bootstrap is used to calibrate the general Bayesian posterior by matching asymptotic Fisher information. We demonstrate the proposed method on a number of examples.
BibTeX
@inproceedings{lyddon2019general,
  author = {Simon Lyddon and Chris Holmes and Stephen G. Walker},
  title = {{General Bayesian updating and the loss-likelihood bootstrap}},
  booktitle = {Biometrika},
  year = {2019},
  doi = {10.1093/biomet/asz006},
}
Figure