Meta-Learning For Stochastic Gradient MCMC

Wenbo Gong, Yingzhen Li, José Miguel Hernández-Lobato

Introduction

Bayesian Deep Learning is a powerful paradigm because it provides robust uncertainty estimates alongside predictions. However, it requires sampling from the "posterior distribution" of neural network weights—a notoriously difficult task in high-dimensional spaces. The standard tool for this is Stochastic Gradient MCMC (SG-MCMC), which treats the network weights like particles moving through an energy landscape, using physics-inspired continuous dynamics (like friction and momentum) to explore the space.

Currently, designing these dynamics is something of a "black art." Researchers typically hand-design samplers (such as SGLD or SGHMC) based on physical intuition. Because these generic designs aren't tailored to the specific, highly non-convex energy landscapes of neural networks, they often struggle to explore efficiently. While meta-learning has successfully "learned" optimization algorithms (like Gradient Descent), plugging a neural network directly into an MCMC sampler is risky: if the dynamics aren't perfectly balanced, the sampler loses its mathematical guarantees and won't converge to the correct target distribution.

This paper introduces the first meta-learning framework for SG-MCMC that allows a neural network to "learn" optimal sampling physics while maintaining strict mathematical guarantees of correctness.

Methods: Learning the Physics of Sampling

The authors bridge the gap between meta-learning and MCMC by building on a complete mathematical framework for SG-MCMC (Ma et al., 2015). This framework proves that any valid sampler can be described by two key matrices: a diffusion matrix ($D$) representing friction and noise, and a curl matrix ($Q$) representing deterministic rotation.

Instead of hand-coding $D$ and $Q$, the authors introduce NNSGHMC, which uses small neural networks to compute these matrices on the fly based on the current energy level, momentum, and gradient.

Comparison of standard SGHMC and the proposed meta-learned NNSGHMC sampler architectures.

Because the parameterization stays strictly within the established mathematical template, the sampler is guaranteed to eventually reach the correct target distribution, regardless of what the neural networks learn. The meta-learner is trained using two novel objectives:

  • Cross-chain loss: Encourages multiple parallel samplers to reach the target distribution as quickly as possible.
  • In-chain loss: Ensures that samples within a single chain are diverse and unbiased.

Since the actual distribution of the sampler during training is unknown, the authors employ a "Stein gradient estimator" to approximate the gradients needed to update the meta-learner. To keep computational overhead practical for real-world deep learning, the learned $D$ and $Q$ matrices are constrained to be diagonal.

Generalization and Key Insights

A core strength of NNSGHMC is its ability to generalize. The authors trained the sampler on simple, small models (e.g., a tiny Multi-Layer Perceptron on MNIST) and tested it on much larger, more complex models (e.g., a deep Convolutional Neural Network on CIFAR-10, or Recurrent Neural Networks on polyphonic music datasets).

The successful transfer from simple to complex architectures suggests that the "geometry" of neural network loss surfaces shares common traits that a meta-learner can exploit. The neural network learns intuitive but complex strategies—for example, applying high friction when the momentum and the gradient point in opposite directions, effectively preventing the sampler from "overshooting" a high-probability area of the landscape.

Key Results

The meta-learned sampler demonstrated significant improvements over generic, hand-designed SG-MCMC algorithms across multiple benchmarks:

  • Faster Convergence & Lower Bias: On synthetic tasks, the meta-sampler showed much lower sampling bias (measured by KL divergence) than standard methods, meaning its samples were a more accurate reflection of the true distribution. It also reached these lower error rates significantly faster.
  • Robustness to Noise: The learned dynamics proved better at "filtering" the noise inherent in stochastic gradients, leading to more stable exploration of the weight space.
  • Strong CNN Performance: On CIFAR-10, NNSGHMC outperformed the popular Preconditioned SGLD (PSGLD), an impressive feat given that PSGLD relies on sophisticated moving-average heuristics.
KL Divergence over iterations showing NNSGHMC converging faster and to a lower bias than standard SGHMC.

Limitations and Future Work

While the learned sampler is fast during inference, the initial "meta-training" phase to learn the $D$ and $Q$ networks is computationally expensive. Furthermore, the Stein gradient estimator used during training scales cubically with the number of parallel chains ($O(K^3)$), limiting how many chains can be utilized during the meta-learning phase. Future work may explore incorporating "temperature" variables (annealing) and moving-average estimates within the meta-learning framework to further scale performance for massive models.

Summary updated

Abstract

Stochastic gradient Markov chain Monte Carlo (SG-MCMC) has become increasingly popular for simulating posterior samples in large-scale Bayesian modeling. However, existing SG-MCMC schemes are not tailored to any specific probabilistic model, even a simple modification of the underlying dynamical system requires significant physical intuition. This paper presents the first meta-learning algorithm that allows automated design for the underlying continuous dynamics of an SG-MCMC sampler. The learned sampler generalizes Hamiltonian dynamics with state-dependent drift and diffusion, enabling fast traversal and efficient exploration of energy landscapes. Experiments validate the proposed approach on Bayesian fully connected neural network, Bayesian convolutional neural network and Bayesian recurrent neural network tasks, showing that the learned sampler outperforms generic, hand-designed SG-MCMC algorithms, and generalizes to different datasets and larger architectures.
BibTeX
@inproceedings{gong2019meta,
  author = {Wenbo Gong and Yingzhen Li and José Miguel Hernández-Lobato},
  title = {{Meta-Learning For Stochastic Gradient MCMC}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=saIE17GGA1e},
}
Figure