Slice Sampling Reparameterization Gradients

David M. Zoltowski, Diana Cai, Ryan P. Adams

Introduction

In machine learning, optimizing objectives that take the form of an expectation over a probability distribution is a common challenge, particularly in Variational Inference. When the parameters being optimized define that distribution, computing the gradient of the expectation is non-trivial.

The "gold standard" approach is the reparameterization trick, which provides low-variance gradients. However, it typically requires the distribution to be normalized, meaning its scaling constant (the partition function) must be known. Many powerful models, such as Energy-Based Models (EBMs), are unnormalized because calculating this constant is computationally intractable. While Markov Chain Monte Carlo (MCMC) can sample from unnormalized distributions, standard MCMC algorithms (like Metropolis-Hastings) rely on discrete "accept/reject" steps. These jumps make the sampling path non-differentiable, precluding the use of standard reparameterization gradients.

This paper introduces a novel method to compute reparameterization gradients for unnormalized probability distributions by differentiating through the slice sampling MCMC algorithm.

Differentiating Through Slice Sampling

The authors leverage Slice Sampling, an auxiliary-variable MCMC method, because of one crucial property: it is rejection-free. If the random seeds are fixed, the sampling path becomes a deterministic and differentiable function of the model parameters.

Slice sampling and its differentiable computational graph

To adapt slice sampling for gradient computation, the authors introduce several key technical innovations:

  1. Direct Boundary Search: Instead of the traditional "stepping-out" procedure, which is difficult to differentiate, they employ numerical root-finding to locate the exact boundaries of the "slice" where the density exceeds a sampled threshold.
  2. Implicit Differentiation: Backpropagating through every iteration of a root-finding loop incurs massive memory overhead. To bypass this, the authors use the Implicit Function Theorem. This allows them to calculate how the slice endpoints change with respect to the parameters based solely on the final boundary values.
  3. Random-Direction Sampling: For multivariate distributions, the algorithm samples a random direction and performs the slice update along that 1D line. This ensures the root-finding problem remains one-dimensional regardless of the model's total dimensionality.

Key Insights

A major theoretical advantage of this approach is Log-Normalizer Cancellation. The authors demonstrate that when optimizing the KL divergence for unnormalized models, the gradient of the unknown log-normalizer mathematically cancels out. Consequently, these models can be optimized without ever needing to approximate the intractable partition function.

Furthermore, while all MCMC-based gradients suffer from some "initialization bias" due to simulating finite chains, the authors found that the variance of the slice sampling estimator is significantly lower than that of the "score function" (REINFORCE) estimator, which is the traditional fallback for unnormalized models.

Experimental Results

The method was rigorously evaluated across synthetic and real-world benchmarks:

  • Variance Reduction: In synthetic tests using 5D Gaussian and Laplace distributions, slice sampling gradients exhibited drastically lower variance than score function estimators, leading to much more stable optimization.
Gradient bias and variance comparisons across different distributions
  • Generative Modeling: When applied to Variational Contrastive Divergence (VCD), the reparameterized slice gradients achieved higher held-out log-likelihoods on MNIST and Fashion-MNIST compared to standard VCD approaches.
  • EBM Training: The method successfully trained EBMs to capture the geometry of complex, disjoint 2D datasets (like the "Swiss roll" and "spirals") using an adversarial Wasserstein GAN-style objective.

Limitations and Future Work

While powerful, the method has a few limitations. As dimensionality increases, sampling along a single random direction can become an inefficient way to explore the space. The authors suggest combining slice sampling with Normalizing Flows to "warp" the space into a more sampling-friendly geometry. Additionally, highly multimodal distributions pose a challenge; the root-finder might only identify the boundaries of the current local mode, potentially missing disconnected regions of high density. Finally, performing numerical root-finding at every MCMC step is computationally heavier than simpler HMC or Langevin steps, highlighting a need for faster boundary approximation techniques in future work.

Summary updated

Abstract

Many probabilistic modeling problems in machine learning use gradient-based optimization in which the objective takes the form of an expectation. These problems can be challenging when the parameters to be optimized determine the probability distribution under which the expectation is being taken, as the na\"ive Monte Carlo procedure is not differentiable. Reparameterization gradients make it possible to efficiently perform optimization of these Monte Carlo objectives by transforming the expectation to be differentiable, but the approach is typically limited to distributions with simple forms and tractable normalization constants. Here we describe how to differentiate samples from slice sampling to compute \textit{slice sampling reparameterization gradients}, enabling a richer class of Monte Carlo objective functions to be optimized. Slice sampling is a Markov chain Monte Carlo algorithm for simulating samples from probability distributions; it only requires a density function that can be evaluated point-wise up to a normalization constant, making it applicable to a variety of inference problems and unnormalized models. Our approach is based on the observation that when the slice endpoints are known, the sampling path is a deterministic and differentiable function of the pseudo-random variables, since the algorithm is rejection-free. We evaluate the method on synthetic examples and apply it to a variety of applications with reparameterization of unnormalized probability distributions.
BibTeX
@inproceedings{zoltowski2021slice,
  author = {David M. Zoltowski and Diana Cai and Ryan P. Adams},
  title = {{Slice Sampling Reparameterization Gradients}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=yBcvl8FI8H},
}
Figure