SuperNest: Accelerated Nested Sampling Applied to Astrophysics and Cosmology

Aleksandr Petrosyan, Will Handley

This paper introduces SuperNest, an open-source Python package designed to solve one of the most persistent bottlenecks in Bayesian inference: the slow "compression" phase of Nested Sampling.

TL;DR

SuperNest accelerates Nested Sampling by allowing researchers to provide a "proposal" distribution (an educated guess of the posterior) to the sampler. By using a novel technique called stochastic superpositional repartitioning, it reduces the computational work required to move from a broad prior to a narrow posterior, achieving up to an order-of-magnitude improvement in efficiency.

What Problem Does This Paper Address?

In Bayesian astrophysics and cosmology, researchers often use Nested Sampling (NS) to calculate the "evidence" (for model comparison) and the "posterior" (for parameter estimation). However, NS has a major weakness compared to other methods like MCMC: it is often much slower.

The primary reason for this slowness is the "compression" phase. NS starts by drawing samples from a broad, uninformative prior and gradually "squeezes" them into the tiny region of high likelihood where the posterior lives. The amount of work required is proportional to the Kullback-Leibler (KL) divergence—essentially a measure of how much volume the sampler must discard to find the truth. If your prior is a massive ocean and your posterior is a tiny island, NS spends a vast amount of time just finding the island.

Methods

The authors build on a concept called Posterior Repartitioning. The mathematical trick is that as long as the product of your prior ($\pi$) and likelihood ($\mathcal{L}$) remains the same, you can shift "weight" between them without changing the final result.

The paper’s core contribution is Stochastic Superpositional Repartitioning:

  1. The Proposal: The user provides one or more "proposal" distributions (e.g., a Gaussian centered on a previous best-fit result).
  2. The Selection Parameter: They introduce a discrete integer parameter, $\beta$. If $\beta=0$, the algorithm uses the original prior. If $\beta=1$, it uses the proposal distribution.
  3. The Modified Likelihood: To keep the math consistent, if the sampler chooses the proposal prior, it uses a modified likelihood that "corrects" for the change.
  4. Sampling: The algorithm samples over both the physical parameters and $\beta$.

Unlike previous "additive" methods that struggled in high dimensions because they were hard to map onto a unit hypercube, this stochastic approach is computationally "cheap" and scales easily to complex, many-parameter models.

Datasets

The authors demonstrate the method using a Cosmological Likelihood Model acting as a proxy for real-world data:

  • Source: Based on the Planck legacy archive chains (cosmic microwave background data).
  • Complexity: A 27-dimensional parameter space, consisting of 6 core cosmological parameters (like the Hubble constant $H_0$) and 21 "nuisance" parameters related to foregrounds and satellite calibration.

Key Insights and Ideas

  • The KL Shortcut: By providing a proposal that is closer to the final posterior than the original prior, you effectively "shorten the distance" the sampler has to travel. This directly reduces the KL divergence the sampler "sees."
  • Precision for Free: In Nested Sampling, faster compression doesn't just save time; it reduces the accumulated Poisson noise. This means you get a more accurate estimate of the evidence ($\mathcal{Z}$) with fewer samples.
  • Robustness to "Bad" Guesses: Because the method is stochastic, if you provide a poor proposal, the sampler will simply spend more time in the $\beta=0$ (original prior) state. It might not speed up, but it won't give you the wrong answer.

Key Results

Using the 27-dimensional Planck example, the authors compared standard Nested Sampling (via the PolyChord sampler) against SuperNest:

  • Speed: The algorithm terminated approximately 3 times faster.
  • Accuracy: The error bars on the evidence were 2 times smaller.
  • Overall Efficiency: When accounting for both speed and the increased precision (precision-normalized speed), SuperNest achieved a 12x improvement over the traditional run.
  • Flexibility: The authors showed that even when the proposal's shape (covariance) was intentionally misaligned with the true posterior, the method remained stable and accurate.

Limitations and Future Work

  • Proposal Dependency: The magnitude of the speedup is entirely dependent on the quality of the proposal distribution. If the proposal is as broad as the prior, there is no gain.
  • Future Research: The authors plan to investigate "hysteretic priors," where the choice of prior evolves as the sampler learns more about the posterior in real-time. They also aim to extend this "consistent partitioning" framework to functional spaces (where you are estimating a function rather than just a set of numbers).

Code Availability: The authors have released the code as an LGPL-licensed Python package called supernest.

Summary updated

Abstract

We present a method for improving the performance of nested sampling as well as its accuracy. Building on previous work we show that posterior repartitioning may be used to reduce the amount of time nested sampling spends in compressing from prior to posterior if a suitable “proposal” distribution is supplied. We showcase this on a cosmological example with a Gaussian posterior, and release the code as an LGPL licensed, extensible Python package supernest.
BibTeX
@misc{petrosyan2022supernest,
  author = {Aleksandr Petrosyan and Will Handley},
  title = {{SuperNest: Accelerated Nested Sampling Applied to Astrophysics and Cosmology}},
  year = {2022},
  eprint = {2212.01760},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.01760},
}
Figure