Beyond ELBOs: A Large-Scale Evaluation of Variational Methods for Sampling

Denis Blessing, Xiaogang Jia, Johannes Esslinger, Francisco Vargas, Gerhard Neumann

Abstract

Monte Carlo methods, Variational Inference, and their combinations play a pivotal role in sampling from intractable probability distributions. However, current studies lack a unified evaluation framework, relying on disparate performance measures and limited method comparisons across diverse tasks, complicating the assessment of progress and hindering the decision-making of practitioners. In response to these challenges, our work introduces a benchmark that evaluates sampling methods using a standardized task suite and a broad range of performance criteria. Moreover, we study existing metrics for quantifying mode collapse and introduce novel metrics for this purpose. Our findings provide insights into strengths and weaknesses of existing sampling methods, serving as a valuable reference for future developments.

Introduction: The Evaluation Crisis in Sampling

Sampling from complex, high-dimensional probability distributions is a cornerstone of Bayesian statistics and the natural sciences. Traditionally, the field has been divided into two camps: Monte Carlo (MC) methods, which are asymptotically accurate but computationally expensive, and Variational Inference (VI), which is fast but often introduces significant bias. Recently, a wave of hybrid "Variational Monte Carlo" and diffusion-based samplers has emerged, promising to bridge this gap.

However, the field faces a significant evaluation hurdle. Most researchers rely on the Evidence Lower Bound (ELBO) to measure success. Because the ELBO is derived from the "reverse KL divergence," it is inherently mode-seeking. A model can achieve a high ELBO by perfectly capturing a single mode of a distribution while completely ignoring others—a phenomenon known as mode collapse. Without a standardized benchmark and metrics that penalize this behavior, it is impossible to discern which algorithms truly advance the state of the art.

A Unified Benchmarking Framework

To provide a rigorous "reality check," this study evaluates 15 different algorithms across a standardized suite of 12 diverse tasks. The methods are categorized into three primary groups:

  1. Tractable Density Models: Baselines such as Mean-Field VI (MFVI) and Gaussian Mixture Models (GMMVI).
  2. Sequential Importance Sampling (SIS): Methods that "anneal" a simple base distribution into the target, including Sequential Monte Carlo (SMC), Annealed Flow Transport (AFT), and Flow Annealed Importance Sampling Bootstrap (FAB).
  3. Diffusion-Based Methods: Modern samplers that treat sampling as a reverse-time Stochastic Differential Equation (SDE), such as Path Integral Samplers (PIS) and Denoising Diffusion Samplers (DDS).

New Metrics for Mode Coverage

To move beyond the limitations of the ELBO, the authors emphasize the Evidence Upper Bound (EUBO). Unlike the ELBO, the EUBO is based on the "forward KL divergence," which is mass-covering—it penalizes the model for missing parts of the target distribution.

Most importantly, the paper introduces Entropic Mode Coverage (EMC). This metric partitions the target space into regions (modes) and calculates the entropy of samples falling into those regions. An EMC score of 1.0 indicates that the sampler hits all modes with equal frequency, while a score of 0.0 indicates total collapse into a single mode.

Experimental Setup and Tasks

The benchmark spans 12 targets designed to test different failure modes:

  • Synthetic Challenges: The "Funnel" (testing high curvature) and Mixtures of Gaussians (MoG) or Student-t (MoS) distributions in dimensions up to $d=200$.
  • Real-world Inference: Bayesian Logistic Regression (using Credit, Cancer, Ionosphere, and Sonar datasets) and Random Effect Regression (Seeds).
  • High-Dimensional Statistics: Brownian motion time series and Spatial Statistics (Log Gaussian Cox Process, $d=1600$).
  • Image-based Targets: Generative models trained on MNIST and Fashion-MNIST.

Key Insights and Results

The ELBO Trap

The evaluation confirms that a high ELBO is often a misleading indicator of quality. In the "Digits" task, MFVI achieved a respectable ELBO but recorded an EMC of 0.0. This means the model completely collapsed to a single digit type, failing to represent the diversity of the target distribution.

The Curse of Dimensionality

Many methods that perform exceptionally well in 2D environments, such as standard SMC, suffer from severe mode collapse as dimensionality increases. By the time tasks reach 50 or 200 dimensions, many previously "robust" samplers fail to maintain coverage.

Efficiency and the Success of GMMVI and FAB

Despite the current research momentum behind diffusion-based methods, the study found that GMMVI and FAB often outperformed complex diffusion samplers. GMMVI, in particular, was found to be orders of magnitude more sample-efficient than Langevin-based diffusion methods, requiring significantly fewer evaluations of the target density to reach convergence.

Resampling Dynamics

In Sequential Monte Carlo, the "resampling" step is intended to eliminate low-weight particles. However, the benchmark reveals that in high-dimensional spaces, this step can actually accelerate mode collapse by prematurely killing off particles that might have eventually explored distant modes.

Conclusion and Future Directions

While diffusion methods demonstrate resilience to mode collapse—maintaining high EMC scores (approx. 0.98) on 200-dimensional Gaussian mixtures where SMC failed—they remain computationally slow and struggle to provide tight evidence bounds.

A primary limitation of the proposed metrics is the requirement for prior knowledge; calculating EMC requires "mode descriptors," and EUBO ideally requires ground-truth samples. Future research should focus on developing unsupervised versions of these metrics for black-box real-world problems. This benchmark serves as a foundational reference, highlighting that the path to better sampling lies in balancing mode coverage with computational efficiency.

Summary updated

Abstract

Monte Carlo methods, Variational Inference, and their combinations play a pivotal role in sampling from intractable probability distributions. However, current studies lack a unified evaluation framework, relying on disparate performance measures and limited method comparisons across diverse tasks, complicating the assessment of progress and hindering the decision-making of practitioners. In response to these challenges, our work introduces a benchmark that evaluates sampling methods using a standardized task suite and a broad range of performance criteria. Moreover, we study existing metrics for quantifying mode collapse and introduce novel metrics for this purpose. Our findings provide insights into strengths and weaknesses of existing sampling methods, serving as a valuable reference for future developments. The code is publicly available here.
BibTeX
@inproceedings{blessing2024beyond,
  author = {Denis Blessing and Xiaogang Jia and Johannes Esslinger and Francisco Vargas and Gerhard Neumann},
  title = {{Beyond ELBOs: A Large-Scale Evaluation of Variational Methods for Sampling}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=fVg9YrSllr},
}
Figure