Inferring Evidence from Nested Sampling Data via Information Field Theory

Margret Westerkamp, Jakob Roth, Philipp Frank, Will Handley, T. A. Enßlin2023 · DOI

TL;DR

This paper introduces a novel method to improve the accuracy of Bayesian evidence estimation, a crucial component for model comparison. By leveraging Information Field Theory (IFT) to reconstruct the smooth likelihood-prior-volume function from nested sampling data, they reduce the noise inherent in traditional methods, leading to more reliable evidence values.

What Problem Does This Paper Address?

In Bayesian inference, comparing different models often relies on calculating the "evidence" for each model. Nested sampling is a popular algorithm for computing this evidence. However, a key challenge in nested sampling is accurately determining the "prior volume" associated with different likelihood values. This uncertainty in prior volume introduces "probing noise," which can significantly degrade the accuracy of the evidence estimate. This paper aims to mitigate this noise by reconstructing the underlying likelihood-prior-volume function more precisely.

Methods

The authors propose an approach based on Information Field Theory (IFT), a framework for non-parametric function reconstruction. They treat the likelihood-prior-volume function as a smooth, monotonically decreasing field. Using IFT, they infer this function by:

  • Defining a Reconstruction Prior: They model the negative rate of change of the logarithmic prior volume as a log-normal process, assuming smoothness and monotonicity. This is parameterized by a function $a_L(\ln X)$.
  • Defining a Reconstruction Likelihood: They develop a likelihood model that relates the observed nested sampling data (likelihood contours) to the underlying function.
  • Joint Inference: They perform a joint inference of the $a_L(\ln X)$ function and the parameters describing the prior volume changes using the IFT framework and the geoVI algorithm for variational inference. This allows them to obtain posterior samples of the likelihood-prior-volume function, including estimates of its uncertainty.

Datasets

The paper uses a synthetic dataset generated from a simple Gaussian likelihood function for evaluation. This dataset consists of "dead point" likelihood contours and approximated prior volumes, as would be produced by standard nested sampling software (specifically, anesthetic is mentioned). No new benchmark datasets are introduced.

Key Insights and Ideas

  • Smoothness as a Powerful Prior: The core idea is that the likelihood-prior-volume function is generally smooth in real-world applications. Exploiting this prior knowledge through IFT significantly improves the reconstruction and, consequently, the evidence estimation.
  • Quantifying Uncertainty: The IFT approach naturally provides posterior samples of the reconstructed function, allowing for a direct quantification of the uncertainty in the evidence estimate. This is a significant advantage over methods that provide a single point estimate.
  • Bridging the Gap: The method effectively bridges the gap between the discrete, noisy data from nested sampling and the underlying continuous, smooth function that governs the evidence.

Key Results

The authors demonstrate their method on a simple Gaussian example. They show that their reconstructed likelihood-prior-volume function closely matches the ground truth. Crucially, when using these reconstructed functions to estimate the evidence, they achieve a significantly smaller standard deviation in the evidence values compared to classical nested sampling with the same number of samples.

Specifically, for the Gaussian example:

  • Ground truth log-evidence: $\ln Z_{\text{gauss}} = -37.798$
  • Classical nested sampling (200 samples): $\ln Z_d = -38.92 \pm 4.50$
  • Reconstructed evidence (200 samples): $\ln Z = -37.97 \pm 2.89$

This shows a substantial reduction in the uncertainty of the evidence estimate.

Limitations and Future Work

  • Assumption of Smoothness: The method's effectiveness relies heavily on the assumption of smoothness for the likelihood-prior-volume curve. While this is expected to hold in most real-world scenarios, its performance on highly non-smooth or discontinuous likelihoods is not yet tested.
  • Future Work: The authors plan to extend this work to:
  • Test the reconstruction algorithm on other likelihood functions where the ground truth is known.
  • Apply the method to actual nested sampling outputs from real-world problems.
  • Investigate the performance with non-Gaussian likelihoods.

Summary updated

Abstract

Nested sampling provides an estimate of the evidence of a Bayesian inference problem via probing the likelihood as a function of the enclosed prior volume. However, the lack of precise values of the enclosed prior mass of the samples introduces probing noise, which can hamper high-accuracy determinations of the evidence values as estimated from the likelihood-prior-volume function. We introduce an approach based on information field theory, a framework for non-parametric function reconstruction from data, that infers the likelihood-prior-volume function by exploiting its smoothness and thereby aims to improve the evidence calculation. Our method provides posterior samples of the likelihood-prior-volume function that translate into a quantification of the remaining sampling noise for the evidence estimate, or for any other quantity derived from the likelihood-prior-volume function.
BibTeX
@misc{westerkamp2023inferring,
  author = {Margret Westerkamp and Jakob Roth and Philipp Frank and Will Handley and T. A. Enßlin},
  title = {{Inferring Evidence from Nested Sampling Data via Information Field Theory}},
  year = {2023},
  doi = {10.3390/psf2023009019},
}
Figure