Abstract
We describe a novel approach to quadrature for ratios of probabilistic integrals, such as are used to compute posterior probabilities. It offers performance superior to Monte Carlo methods by explicitly modeling the non-negativity of probabilities and the inherent correlations between the numerator and denominator.
Introduction
In Bayesian inference, marginalizing unknown hyperparameters requires evaluating definite, non-analytic integrals. The standard approach is Monte Carlo (MC) integration, but it is often sample-inefficient because it ignores the spatial distribution of samples. Bayesian Quadrature (BQ) treats integration as a machine learning problem by fitting a Gaussian Process (GP) to samples. However, standard BQ fails for posterior probabilities because it can assign mass to negative values and ignores the strong correlation between the numerator (joint distribution) and denominator (marginal likelihood).
Methods: Bayesian Quadrature for Ratios (BQR)
To address these issues, the authors propose Bayesian Quadrature for Ratios (BQR). First, to ensure non-negativity, BQR places GP priors on the logarithm of the integrands. While this log-transform makes the ratio of integrals non-analytic, the authors employ a linearization technique around the GP conditional means.
The linearization introduces correction factors that are themselves non-analytic integrals. BQR handles this through Nested Bayesian Quadrature, applying a second layer of BQ to estimate these error terms. Spatial partitioning algorithms like KD-trees are used to intelligently select sample points for these corrections.
Key Insights: The Power of Correlation
A central finding of this work is that explicitly modeling the correlation between the numerator and denominator is the primary driver of success. Standard "Naive" BQ treats them independently, leading to unstable estimates. By capturing the shared structure of these integrals, BQR extracts significantly more information from each sample.
Experimental Results
The method was evaluated on synthetic benchmarks and real-world data from the Kepler Space Telescope.
- Synthetic 1D Test: BQR achieved an RMSE of 0.0042, significantly better than Monte Carlo (0.0217) and Naive BQ (0.0203).
- Kepler Data: In regressing flux to infer star rotation rates—a task with complex, multi-modal likelihoods—BQR achieved a predictive log-likelihood of -10.262, outperforming MC (-12.563).
- Efficiency: BQR typically converges with fewer than 100 samples, making it ideal for expensive integrands where each evaluation (e.g., a large matrix inversion) is costly.
Conclusion and Future Work
BQR provides a robust framework for Bayesian inference in computationally demanding scenarios. While it currently faces $O(N^3)$ scaling due to GP covariance matrices, future work involving sparse GP approximations could extend its reach to larger sample sizes.
Summary updated