Abstract
We propose a novel deterministic sampling method to approximate a target distribution $\rho^*$ by minimizing the kernel discrepancy, also known as the Maximum Mean Discrepancy (MMD). By employing the general energetic variational inference framework (Wang et al., 2021), we convert the problem of minimizing MMD to solving a dynamic ODE system of the particles. We adopt the implicit Euler numerical scheme to solve the ODE systems. This leads to a proximal minimization problem in each iteration of updating the particles, which can be solved by optimization algorithms such as L-BFGS. The proposed method is named EVI-MMD. To overcome the long-existing issue of bandwidth selection of the Gaussian kernel, we propose a novel way to specify the bandwidth dynamically. Through comprehensive numerical studies, we have shown the proposed adaptive bandwidth significantly improves the EVI-MMD. We use the EVI-MMD algorithm to solve two types of sampling problems. In the first type, the target distribution is given by a fully specified density function. The second type is a "two-sample problem", where only training data are available. The EVI-MMD method is used as a generative learning model to generate new samples that follow the same distribution as the training data. With the recommended settings of the tuning parameters, we show that the proposed EVI-MMD method outperforms some existing methods for both types of problems.
Introduction
Sampling from complex probability distributions is a fundamental challenge in modern statistics and machine learning, essential for tasks ranging from Bayesian inference to training generative models. While traditional Markov Chain Monte Carlo (MCMC) methods are widely used, they are often stochastic and slow to converge. Variational Inference (VI) has emerged as a faster, deterministic alternative that treats sampling as an optimization problem, minimizing the distance between a set of particles and a target distribution.
However, standard VI methods typically rely on Kullback-Leibler (KL) divergence, which can struggle when target densities are near zero or when gradients are difficult to compute. Maximum Mean Discrepancy (MMD) offers a robust alternative metric, but its performance is highly sensitive to the kernel bandwidth. This paper introduces EVI-MMD, a framework that treats sampling as a physical system dissipating energy, utilizing an adaptive bandwidth strategy to overcome the limitations of fixed-kernel methods.
Methodology
Energetic Variational Inference (EVI)
The EVI-MMD method transforms the minimization of MMD into a system of Ordinary Differential Equations (ODEs). Under this framework, particles are viewed as a dynamic "flow" moving toward the target distribution along a path that minimizes energy. This approach allows the sampling process to be framed as a continuous-time dissipation of the MMD functional.
Numerical Scheme and Optimization
To solve the resulting ODE system, the authors employ an implicit Euler numerical scheme. Unlike explicit gradient descent steps, the implicit scheme leads to a proximal minimization problem at each iteration:
- Stability: The implicit approach ensures that the MMD is guaranteed to decrease at every iteration, providing significantly more stability than explicit methods.
- Optimization: To solve this proximal problem efficiently, the L-BFGS algorithm is utilized, allowing for larger step sizes and robust convergence in high-dimensional spaces.
Adaptive Bandwidth Selection
The core innovation of EVI-MMD is its dynamic bandwidth formula: $$h_n = \frac{a}{n^c} + b$$ where $n$ is the iteration count. This strategy addresses the "Goldilocks" problem of kernel selection:
- Exploration: At the start of the process, a large bandwidth allows particles to explore the global structure of the distribution and move toward distant modes.
- Exploitation: As iterations progress, the bandwidth shrinks, allowing particles to refine their positions and capture local density details.
Experimental Evaluation
The paper evaluates EVI-MMD across three distinct categories of problems:
Toy Distributions and High-Dimensional Scaling
In tests on complex toy distributions (e.g., star-shaped and 8-component Gaussian mixtures), EVI-MMD achieved lower MMD values faster than Stein Variational Gradient Descent (SVGD) and Langevin Monte Carlo (LMC). Furthermore, when scaled to high-dimensional Gaussians ($d=100$), EVI-MMD remained competitive with specialized methods like "Support-Points," which often struggle with convergence in high dimensions.
Generative Modeling (Two-Sample Problem)
EVI-MMD is also applied to cases where the target distribution is only available through raw data points. In this "two-sample" context, it functions as a non-parametric generative model. Evaluated on benchmark datasets—MNIST, Fashion-MNIST, and CIFAR-10—the method produced visually sharper images and achieved superior Fréchet Inception Distance (FID) scores compared to energy-distance-based alternatives.
Discussion and Conclusion
EVI-MMD provides a robust, deterministic framework for sampling that balances global exploration with local precision. By combining the stability of implicit numerical schemes with a novel adaptive bandwidth strategy, it outperforms traditional methods in both density-based sampling and generative modeling.
While the use of L-BFGS for proximal minimization introduces higher computational costs per iteration than simple gradient updates, the authors mitigate this through mini-batching. Future work may explore different dissipation functionals and the application of adaptive strategies to non-Gaussian kernels to further enhance the flexibility of the EVI framework.
Summary updated