In machine learning and Bayesian statistics, calculating the model evidence (or marginal likelihood, denoted as $Z$) is a fundamental task. It represents the integral of the likelihood over the prior and is crucial for model selection and hyperparameter tuning. However, for complex models or massive datasets, evaluating the likelihood function is computationally expensive, making the resulting integral intractable.
Standard tools like Monte Carlo integration or Annealed Importance Sampling (AIS) require a massive number of samples to converge and often provide unreliable estimates of their own error. Existing "model-based" approaches, such as Bayesian Quadrature (BQ), treat integration as a statistical inference problem.
While BQ is promising, it traditionally struggles with two main issues:
- Non-negativity: Likelihoods are strictly non-negative, but standard Gaussian Processes (GPs) used in BQ can predict negative values.
- Sample Efficiency: Most methods use random or fixed sampling patterns rather than actively choosing points that provide the most information about the integral.
To address these shortcomings, this paper introduces Doubly Bayesian Quadrature (BBQ), a sample-efficient method that achieves more accurate integral estimates and better-calibrated uncertainty measures than traditional Monte Carlo methods.
The BBQ Approach: Methods and Innovations
The authors propose several key innovations to transform Bayesian Quadrature into a robust tool for evidence estimation:
1. Log-GP Modeling
Instead of placing a GP prior directly on the likelihood $\ell(x)$, BBQ places it on $\log \ell(x)$. This naturally enforces non-negativity and handles the high dynamic range typical of likelihood functions. A standard GP tries to smooth out peaks, whereas a log-GP captures the "spiky" nature of likelihoods much better.
Because integrating an exponentiated GP is mathematically difficult, the authors employ a linearization trick (a Taylor expansion) to keep the integral tractable.
2. Active Sampling
If each function evaluation takes minutes or hours, the computational overhead of deciding where to sample next is negligible compared to the cost of a wasted sample. BBQ uses Bayesian decision theory to select the next evaluation point $x$ that is expected to most reduce the uncertainty (variance) of the integral $Z$. This creates an optimal balance between exploration (sampling where the model is uncertain) and exploitation (sampling where the likelihood is high).
3. Hyperparameter Marginalization
GP performance depends heavily on hyperparameters, like the input scale. Usually, these are fixed to a single best guess. The authors introduce a way to approximately marginalize (average over) these hyperparameters using a Laplace approximation. This prevents the model from becoming overconfident in its estimates, leading to better-calibrated uncertainty.
Experimental Results
The authors evaluated BBQ against standard Monte Carlo (SMC), AIS, and basic Bayesian Monte Carlo (BMC) across two types of problems:
- Synthetic Benchmarks: Seven different integrands, including single Gaussians and mixtures of Gaussians, ranging from 1 to 20 dimensions.
- Real-World Astronomy: Detecting Damped Lyman-$\alpha$ (DLA) absorbers in quasar spectra using data from the massive Sloan Digital Sky Survey (SDSS), where evaluating the likelihood is particularly expensive.
Key Findings: * Accuracy: BBQ and its variant BBQ (which includes hyperparameter marginalization) consistently achieved the lowest Average Log Error (ALE). In a 20-dimensional synthetic test, BBQ significantly outperformed all other methods. * Uncertainty Calibration: Using a "calibration score" to verify if the model's predicted error bars contained the true value, BBQ proved to be much better calibrated (less overconfident) than standard BQ. * Real-World Performance:* On the SDSS astronomy data, BBQ was the clear winner. It provided the most accurate evidence estimates using only a very small number of samples (300 evaluations).
Limitations and Future Work
While highly effective for expensive integrands, BBQ does have limitations. The method relies on a Taylor expansion to keep the math tractable; while this approximation improves as more samples are added, it remains an approximation. Additionally, the active sampling process involves its own optimization step to choose the next point, which introduces computational overhead that might be overkill for "cheap" integrands.
Finally, in very high-dimensional spaces with many isolated peaks (multi-modality), active learning is inherently risky. If the initial samples miss a peak entirely, the active learner might never find it, leading to an underestimation of the evidence. Future work will likely focus on making the exploration phase more robust for these complex, multi-modal landscapes.
Summary updated