Abstract
In this paper we revisit the weighted likelihood bootstrap, a method that generates samples from an approximate Bayesian posterior of a parametric model. We show that the same method can be derived, without approximation, under a Bayesian nonparametric model with the parameter of interest defined as minimising an expected negative log-likelihood under an unknown sampling distribution. This interpretation enables us to extend the weighted likelihood bootstrap to posterior sampling for parameters minimizing an expected loss. We call this method the loss-likelihood bootstrap. We make a connection between this and general Bayesian updating, which is a way of updating prior belief distributions without needing to construct a global probability model, yet requires the calibration of two forms of loss function. The loss-likelihood bootstrap is used to calibrate the general Bayesian posterior by matching asymptotic Fisher information. We demonstrate the methodology on a number of examples.
Introduction
Standard Bayesian inference operates under the assumption of a "well-specified" model—the belief that the true data-generating process resides within the chosen parametric family. However, in practical applications, models are often mere approximations. When a model is misspecified, standard Bayesian posteriors tend to become overconfident, concentrating on "pseudo-true" values that may not align with the researcher's actual goals.
To address this, researchers often turn to General Bayesian Updating (GBU). GBU replaces the traditional likelihood with a loss function (such as the hinge loss in Support Vector Machines), allowing for more robust inference. A persistent challenge in GBU, however, is the calibration of the "learning rate" or "loss scale" parameter ($w$). This parameter dictates the influence of the data relative to the prior. Historically, setting $w$ has relied on heuristics or computationally expensive cross-validation. This paper introduces a principled framework for this calibration by bridging GBU with a nonparametric interpretation of the bootstrap.
The Loss-Likelihood Bootstrap (LLB)
The authors first revisit the Weighted Likelihood Bootstrap (WLB). While typically viewed as a frequentist approximation to a parametric posterior, the authors demonstrate that the WLB is actually an exact posterior under a Bayesian nonparametric model. In this view, the parameter of interest is a functional of an unknown distribution.
By extending this logic, the authors propose the Loss-Likelihood Bootstrap (LLB). The LLB generates posterior samples for parameters defined as the minimizers of an expected loss. The algorithm is straightforward: for each bootstrap iteration, one draws Dirichlet weights for the observed data and solves a weighted optimization problem. This process yields a distribution of parameters that reflects the uncertainty inherent in the data without requiring a fully specified generative model.
Calibrating GBU via Information Matching
The core innovation of the paper lies in using the LLB to calibrate the learning rate $w$ in GBU. The authors prove that the LLB samples asymptotically follow a distribution characterized by the "sandwich covariance matrix" ($J^{-1}IJ^{-1}$), a staple of robust frequentist statistics.
To calibrate the GBU posterior, the authors propose matching its Fisher information number (the trace of the Fisher information matrix) to that of the LLB. This "Asymptotic Covariance Matching" provides a closed-form, consistent estimator for $w$. Crucially, this method ensures that the resulting Bayesian update remains "honest" about the information content of the data. If the model happens to be well-specified, the calibration naturally recovers the standard Bayesian update where $w=1$.
Experimental Evaluation
The methodology was tested across several scenarios to validate its theoretical consistency and practical utility:
- Synthetic Simulations: Using normal data and classification tasks, the authors demonstrated that as the sample size $n$ increases, the calibrated posterior correctly concentrates at the optimal decision boundaries.
- Bayesian SVM on German Credit Data: The authors applied the LLB to a "Bayesian SVM" using a smoothed hinge loss on the Statlog German Credit dataset. The results showed that the calibrated GBU achieved misclassification rates comparable to standard linear SVMs. However, unlike traditional SVMs, this approach provided posterior marginals, allowing for a nuanced visualization of uncertainty regarding the classification boundaries.
Discussion and Limitations
While the LLB provides a robust framework for generalized inference, it does introduce specific requirements. The calibration method relies on the first and second derivatives of the loss function, necessitating the use of smoothed versions of non-differentiable losses (like the hinge loss).
Furthermore, while the LLB is "prior-free," integrating subjective priors directly into the bootstrap remains a challenge due to the curse of dimensionality. Finally, although the method is highly parallelizable, the requirement to solve an optimization problem for every bootstrap sample may present a computational hurdle for extremely high-dimensional models.
Despite these constraints, the Loss-Likelihood Bootstrap represents a significant step forward in making General Bayesian Updating a practical and theoretically grounded tool for robust statistical modeling.
Summary updated