Improving the evaluation of samplers on multi-modal targets

Louis Grenioux, Maxence Noble, Marylou Gabrié

Abstract

Sampling from distributions with multiple modes is a persistent challenge in machine learning and physics. While much research focuses on mode discovery, correctly estimating the relative mass of these modes is often more difficult and less frequently evaluated. This paper introduces a systematic benchmarking framework that evaluates samplers on their ability to recover ground-truth mode weights across varying levels of dimensionality ($d$) and mode separation ($a$). We evaluate a broad spectrum of algorithms, ranging from traditional MCMC to modern diffusion-based samplers, revealing that diffusion methods like SLIPS offer superior robustness in high-dimensional, well-separated landscapes.

Introduction

In fields like Bayesian inference and statistical mechanics, researchers frequently encounter probability distributions with several high-probability regions (modes) separated by "valleys" of low probability. Navigating these landscapes presents two distinct hurdles: locating the modes and estimating their relative importance. While the former is a common focus, the latter—determining how much "mass" each peak holds—is arguably harder and often ignored in standard evaluations.

Standard metrics like Wasserstein distance or Maximum Mean Discrepancy (MMD) are computationally expensive and often fail to diagnose why a sampler fails. This paper advocates for mode weight estimation as a primary diagnostic tool, providing a clear, interpretable measure of a sampler's performance.

Methods and Experimental Setup

The authors propose a systematic evaluation framework rather than a new algorithm. They test a wide spectrum of sampling strategies, including Local MCMC (MALA), Importance Sampling (IS), Variational Inference (VI), Annealed methods (SMC and Replica Exchange), and modern Diffusion-based samplers (SLIPS and DDS).

The core of the benchmark is a synthetic bi-modal Gaussian mixture with unbalanced weights (2/3 vs 1/3) and mismatched covariances. Performance is evaluated across two difficulty axes: dimensionality ($d$) ranging from 4 to 256, and a separation parameter ($a$) ranging from 0.5 to 10.0.

Visualizing mode separation $a$ in the synthetic bi-modal Gaussian mixture.

Key Results

The evaluation reveals that mode collapse is a systematic failure for many traditional methods. Local samplers (MALA) and Variational Inference (Gaussian) fail almost immediately as the energy barrier between modes increases. While Normalizing Flows (NFs) improve Importance Sampling's ability to handle mode separation, they still struggle significantly as dimensionality increases.

Heatmaps showing averaged absolute error for various sampling methods across dimensions $d$ and separation $a$.

As shown in Figure 6, traditional "gold standard" methods like Sequential Monte Carlo (SMC) and Replica Exchange (RE) perform well in moderate dimensions but degrade as $d$ approaches 64 and 256. In contrast, diffusion-based methods like SLIPS (Stochastic Localization via Iterative Posterior Sampling) emerged as the most robust. The authors hypothesize that SLIPS succeeds because its intermediate diffusion densities are naturally multi-modal and preserve relative weights, unlike geometric annealing which can "wash out" smaller modes.

Discussion and Limitations

While the proposed benchmark provides clear insights, it currently focuses on a two-mode limit. Real-world problems may involve thousands of local minima or more complex local geometries like "donuts" or heavy-tailed distributions. Furthermore, the evaluation assumes mode boundaries are known, which is straightforward for Gaussians but challenging for complex, non-convex distributions. Future work will need to address these complexities and the sensitivity of modern samplers to hyperparameter tuning when ground truth is unavailable.

Summary updated

BibTeX
@inproceedings{grenioux2025improving,
  author = {Louis Grenioux and Maxence Noble and Marylou Gabrié},
  title = {{Improving the evaluation of samplers on multi-modal targets}},
  booktitle = {FPI @ ICLR},
  year = {2025},
  url = {https://openreview.net/forum?id=d91E9RhVFU},
}
Figure