Introduction
In Bayesian statistics, Markov chain Monte Carlo (MCMC) samplers are the essential tools for estimating complex probability distributions. However, evaluating the performance of a new sampler is notoriously difficult. Researchers typically rely on hand-crafted "toy" problems, which lack real-world complexity, or test set predictive performance, which often conflates the quality of the model with the quality of the sampler. This paper introduces a data-driven benchmarking system that bridges this gap by fitting flexible surrogate models to real posterior distributions, creating a "ground truth" for precise quantification of sampler performance.
The Surrogate Benchmarking Pipeline
The authors propose a six-phase pipeline designed to create a realistic, automated benchmark for MCMC methods. This process moves from raw data to a controlled environment where exact sampling is possible.
- Phase 0-1 (Data & Sampling): The system pulls over 2,200 diverse datasets from OpenML and applies them to a "Model Zoo" of Bayesian models (GLMs, Gaussian Processes, etc.). Initial samples are drawn using high-end samplers like NUTS.
- Phase 2 (Surrogate Strategy): Instead of treating the unknown real posterior as the target, the authors fit flexible surrogate models—such as Mixtures of Gaussians (MoG) or Neural Autoregressive Density Estimators—to the initial samples.
- Phase 3-5 (Ground Truth & Analysis): Because these surrogates are known mathematical entities, the authors can draw exact, independent samples ($p^*$) from them. This provides a "gold standard" to compare against MCMC samples ($\tilde{p}$), allowing for the calculation of the Real Effective Sample Size (RESS).
Key Results and the "Optimism Bias"
The benchmark reveals a significant discrepancy between standard MCMC diagnostics and actual performance. A striking finding is the "optimism bias" of the standard Effective Sample Size (ESS) diagnostic. Practitioners often use ESS to decide if they have gathered enough samples, but the study found that RESS was lower than the estimated ESS 55% of the time for mean estimation, and 83% of the time for complex metrics like Kolmogorov-Smirnov distance.
Sampler Performance Insights
- Bimodality of Success: Sampler performance is rarely mediocre; methods tend to either work very well (efficiency > 1%) or fail catastrophically (RESS < 1). The popular
emceesampler exhibited this bimodal behavior most strongly, showing high failure rates in high-dimensional problems despite its speed. - Top Performers: NUTS (No-U-Turn Sampler) and HMC (Hamiltonian Monte Carlo) remain the gold standards, providing the highest efficiency per sample. Slice sampling also proved highly competitive, particularly in lower-dimensional spaces.
- Diagnostic Utility: While Gelman-Rubin (GR) and ESS provide some predictive value, other common diagnostics like Geweke were found to be poor predictors of actual sampler error.
Conclusion
By creating a "COCO benchmark" for MCMC, this work provides a necessary reality check for the Bayesian community. The discovery that standard diagnostics are often dangerously optimistic suggests that practitioners should approach MCMC results with more caution. Future work will focus on expanding the "Model Zoo" and utilizing more sophisticated density estimators to ensure the benchmark keeps pace with modern deep learning challenges.
Summary updated