Bayesian Quadrature for Ratios

Michael A. Osborne, Roman Garnett, Stephen J. Roberts, Christopher Hart, Suzanne Aigrain, Neale GibsonAISTATS 2012 · OpenReview

Abstract

We describe a novel approach to quadrature for ratios of probabilistic integrals, such as are used to compute posterior probabilities. It offers performance superior to Monte Carlo methods by explicitly modeling the non-negativity of probabilities and the inherent correlations between the numerator and denominator.

Introduction

In Bayesian inference, marginalizing unknown hyperparameters requires evaluating definite, non-analytic integrals. The standard approach is Monte Carlo (MC) integration, but it is often sample-inefficient because it ignores the spatial distribution of samples. Bayesian Quadrature (BQ) treats integration as a machine learning problem by fitting a Gaussian Process (GP) to samples. However, standard BQ fails for posterior probabilities because it can assign mass to negative values and ignores the strong correlation between the numerator (joint distribution) and denominator (marginal likelihood).

Methods: Bayesian Quadrature for Ratios (BQR)

To address these issues, the authors propose Bayesian Quadrature for Ratios (BQR). First, to ensure non-negativity, BQR places GP priors on the logarithm of the integrands. While this log-transform makes the ratio of integrals non-analytic, the authors employ a linearization technique around the GP conditional means.

Log-GP modeling and nested BQ correction.

The linearization introduces correction factors that are themselves non-analytic integrals. BQR handles this through Nested Bayesian Quadrature, applying a second layer of BQ to estimate these error terms. Spatial partitioning algorithms like KD-trees are used to intelligently select sample points for these corrections.

Key Insights: The Power of Correlation

A central finding of this work is that explicitly modeling the correlation between the numerator and denominator is the primary driver of success. Standard "Naive" BQ treats them independently, leading to unstable estimates. By capturing the shared structure of these integrals, BQR extracts significantly more information from each sample.

Joint distributions showing the correlation between numerator and denominator.

Experimental Results

The method was evaluated on synthetic benchmarks and real-world data from the Kepler Space Telescope.

  • Synthetic 1D Test: BQR achieved an RMSE of 0.0042, significantly better than Monte Carlo (0.0217) and Naive BQ (0.0203).
  • Kepler Data: In regressing flux to infer star rotation rates—a task with complex, multi-modal likelihoods—BQR achieved a predictive log-likelihood of -10.262, outperforming MC (-12.563).
  • Efficiency: BQR typically converges with fewer than 100 samples, making it ideal for expensive integrands where each evaluation (e.g., a large matrix inversion) is costly.

Conclusion and Future Work

BQR provides a robust framework for Bayesian inference in computationally demanding scenarios. While it currently faces $O(N^3)$ scaling due to GP covariance matrices, future work involving sparse GP approximations could extend its reach to larger sample sizes.

Summary updated

Abstract

We describe a novel approach to quadrature for ratios of probabilistic integrals, such as are used to compute posterior probabilities. It offers performance superior to Monte Carlo methods by explo...
BibTeX
@inproceedings{osborne2012bayesian,
  author = {Michael A. Osborne and Roman Garnett and Stephen J. Roberts and Christopher Hart and Suzanne Aigrain and Neale Gibson},
  title = {{Bayesian Quadrature for Ratios}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2012},
  url = {https://openreview.net/forum?id=cCCh2jaZYQ5},
}
Figure