Stein Boltzmann Sampling: A Variational Approach for Global Optimization

Gaëtan Serré, Argyris Kalogeratos, Nicolas VayatisAISTATS 2025

Introduction

Global optimization—the task of finding the absolute minimum of a non-convex function—is a fundamental challenge in machine learning and engineering. Current approaches typically fall into two categories: heuristic methods (e.g., CMA-ES), which are efficient but lack theoretical guarantees, and Bayesian Optimization, which is theoretically robust but scales poorly with dimensionality. This paper introduces Stein Boltzmann Sampling (SBS), a deterministic, particle-based method that bridges this gap by framing global optimization as a variational inference problem.

The SBS Framework

The core idea of SBS is to sample from a target Boltzmann Distribution (BD), defined by a density proportional to $e^{-\kappa f(x)}$, where $f(x)$ is the objective function and $\kappa$ is the inverse temperature. As $\kappa$ increases, the BD concentrates its mass on the global minimizers of $f(x)$.

To sample from this distribution without the computational overhead of Markov Chain Monte Carlo (MCMC), the authors employ Stein Variational Gradient Descent (SVGD). In SVGD, a set of particles (candidate solutions) is moved deterministically toward the target distribution. This movement is governed by two forces: an attraction toward high-density regions (the global minima) and a repulsion between particles. This repulsion serves as a natural diversity mechanism, preventing premature convergence to local minima and facilitating exploration.

Evolution of the particle mass distribution $m^{(s)}$ toward the global minimum.

Algorithmic Variants

The authors propose two significant enhancements to the basic SBS algorithm to improve practical performance:

  1. SBS-PF (Particle Filtering): To improve efficiency, this variant monitors particle trajectories and removes those that become stagnant or trapped in poor local regions. This filtering process focuses computational resources on the most promising areas of the search space.
  2. SBS-Hybrid: This approach uses a "warm-start" strategy, employing fast heuristic methods like CMA-ES for initial exploration before transitioning to the mathematically rigorous SBS for final refinement.
Comparison of standard SBS (a) and the filtered SBS-PF (b) on a complex 2D landscape.

Theoretical and Empirical Results

A key contribution of this work is the theoretical extension of SVGD. While standard SVGD theory often assumes smooth distributions over $\mathbb{R}^d$, the authors prove convergence for Sobolev functions on compact domains, making the method applicable to real-world bounded optimization problems.

Empirically, SBS and its variants were tested against state-of-the-art baselines (including BayesOpt, CMA-ES, and AdaLIPO) on standard benchmarks in both 2D and 50D.

  • Accuracy: SBS-Hybrid achieved the top rank in 2D benchmarks with the lowest Empirical Competitive Ratio (ECR) of 15.4. In 50D tests, SBS and SBS-PF dominated, demonstrating superior scalability in high-dimensional search spaces.
  • Efficiency: The SBS-PF variant achieved a 97% reduction in function evaluations compared to the standard version with negligible loss in accuracy. Unlike Bayesian methods, the computational cost of SBS remains lean as the number of evaluations increases.

Conclusion and Future Work

Stein Boltzmann Sampling provides a theoretically grounded yet computationally efficient framework for global optimization. By leveraging the diversity-promoting properties of Stein gradients, it effectively balances exploration and exploitation. Future research will focus on developing adaptive kernels to reduce hyperparameter sensitivity and exploring more advanced particle filtering strategies to further enhance efficiency.

Summary updated

BibTeX
@inproceedings{serre2025stein,
  author = {Gaëtan Serré and Argyris Kalogeratos and Nicolas Vayatis},
  title = {{Stein Boltzmann Sampling: A Variational Approach for Global Optimization}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
}
Figure