Monte Carlo Approximation of Bayes Factors via Mixing With Surrogate Distributions

Chenguang Dai, Jun S. Liu

TL;DR

Calculating Bayes factors—the "gold standard" for Bayesian model selection—is notoriously difficult because it requires computing intractable normalizing constants. This paper introduces a clever "mixture" approach that combines a target posterior with a tractable surrogate distribution, using an accelerated Wang-Landau algorithm to estimate the ratio between them. The result is a robust, high-performance method that handles high-dimensional models and non-overlapping distributions better than traditional techniques like bridge sampling.

What Problem Does This Paper Address?

In Bayesian statistics, comparing two models involves calculating the Bayes factor, which is the ratio of their marginal likelihoods. The marginal likelihood (or normalizing constant) is an integral over the entire parameter space. For complex, high-dimensional models, this integral has no analytical solution and is extremely difficult to approximate numerically.

Existing methods have significant drawbacks:

  • Importance Sampling and Bridge Sampling: These rely heavily on the "overlap" between the proposal and the target distribution. If they don't overlap well, the estimates become highly unstable.
  • Chib’s Method: Highly effective but requires that all conditional distributions are tractable (e.g., for Gibbs sampling), which isn't always the case.
  • Path Sampling/SMC: Powerful but often require delicate tuning of "tempering" schedules and can be computationally expensive.

The authors aim to create a more robust, "general-purpose" tool that works even when distributions are poorly aligned or when the sampler uses more complex kernels like Hamiltonian Monte Carlo (HMC).

Methods

The core innovation is the Wang-Landau (WL) Mixture Method. Here’s the "secret sauce":

  1. The Mixture: Instead of sampling the target $\gamma(\theta)$ directly, they create a mixture distribution $\pi(\theta) = \gamma(\theta) + q(\theta)$, where $q(\theta)$ is a "surrogate" distribution with a known normalizing constant.
  2. Wang-Landau Algorithm: They use the WL algorithm—originally from statistical physics—to adaptively estimate the relative weights of these two components. By iteratively "penalizing" the component the chain visits most often, the algorithm eventually forces the chain to spend equal time in both, revealing the ratio of their normalizing constants.
  3. Momentum Acceleration: The authors reframe the WL update as a stochastic gradient descent problem. By adding a "momentum" term (similar to those used in deep learning optimizers), they significantly speed up convergence.
  4. Surrogate Construction: They use Variational Inference (VI) to automatically build a surrogate distribution that roughly matches the target, ensuring the algorithm doesn't have to work too hard to jump between them.
  5. Multiple-Try Metropolis (MTM): To facilitate jumps between the target and surrogate (especially if they are far apart), they employ MTM, which proposes multiple candidates and picks the best one, ensuring the Markov chain doesn't get stuck.

Datasets

The authors evaluate their method on several classic and complex statistical benchmarks:

  • Log-Gaussian Cox Process: Used for spatial point patterns (specifically, locations of 126 pine saplings). This is a high-dimensional problem with up to 900 parameters.
  • Bayesian Lasso: Applied to simulated data to test hyperparameter selection.
  • Logistic Regression: Using the German Credit dataset (1,000 records, 301 parameters) to estimate marginal likelihoods.
  • g-Prior Variable Selection: Using the Pollution dataset (60 observations, 15 predictors) to test their trans-dimensional MTM-RJMCMC extension.

Key Insights and Ideas

  • Robustness to Separation: Unlike bridge sampling, which fails if the surrogate and target don't overlap, the WL mixture method remains stable. Because it uses an adaptive penalty, it can "push" the sampler across regions of low probability.
  • Optimization Perspective: Viewing the Wang-Landau algorithm through the lens of optimization (specifically projected gradient descent) allows for the application of acceleration tools like momentum, which the authors show provides a massive speedup in reaching the correct normalizing constant.
  • Automated Surrogates: By using Variational Inference (CAVI) to create the surrogate, the method becomes much more "plug-and-play" for researchers who don't want to manually tune proposal distributions.

Key Results

  • Efficiency: In the Log-Gaussian Cox process, the WL mixture method produced estimates with much lower variance than importance sampling for the same computational budget.
  • Convergence Speed: The "Accelerated" WL (AWL) reached stable estimates significantly faster than the standard WL algorithm. For example, in the Bayesian Lasso test, AWL converged in roughly 2,000 iterations, while standard WL was still oscillating at 10,000.
  • Accuracy in Model Selection: In the variable selection task, their MTM-based Reversible Jump MCMC (RJMCMC) correctly identified the true model predictors, whereas standard birth-and-death RJMCMC got stuck in local modes and selected incorrect variables.

Limitations and Future Work

  • Surrogate Quality: While the method is robust, a very poor surrogate (one that is extremely far from the target) will still slow down the MTM jumping mechanism.
  • Theoretical Framework: The authors note that while the empirical results are strong, a more rigorous theoretical framework is needed to fully understand the convergence properties of the accelerated mixture method.
  • Multi-component Mixtures: The current work focuses on a two-component mixture (target + one surrogate). Future work could explore using multiple surrogates to "bridge" even more complex, multi-modal landscapes.

Summary updated

Abstract

By mixing the target posterior distribution with a surrogate distribution, of which the normalizing constant is tractable, we propose a method for estimating the marginal likelihood using the Wang–Landau algorithm. We show that a faster convergence of the proposed method can be achieved via the momentum acceleration. Two implementation strategies are detailed: (i) facilitating global jumps between the posterior and surrogate distributions via the multiple-try Metropolis (MTM); (ii) constructing the surrogate via the variational approximation. When a surrogate is difficult to come by, we describe a new jumping mechanism for general reversible jump Markov chain Monte Carlo algorithms, which combines the MTM and a directional sampling algorithm. We illustrate the proposed methods on several statistical models, including the log-Gaussian Cox process, the Bayesian Lasso, the logistic regression, and the g-prior Bayesian variable selection. Supplementary materials for this article are available online.
BibTeX
@article{dai2020monte,
  author = {Chenguang Dai and Jun S. Liu},
  title = {{Monte Carlo Approximation of Bayes Factors via Mixing With Surrogate Distributions}},
  journal = {Journal of the American Statistical Association},
  year = {2020},
  doi = {10.1080/01621459.2020.1811100},
}

Loading related content...

Figure