Stochastic Gradient Langevin dynamics that Exploit Neural Network Structure

Zachary Nado, Jasper Snoek, Roger B. Grosse, David Duvenaud, Bowen Xu, James MartensWorkshops @ ICLR 2018 · OpenReview

Introduction

Deep neural networks are notoriously "overconfident" in their predictions. In high-stakes fields like medical diagnosis or autonomous driving, it is critical to deploy models that know when they are uncertain. Bayesian inference provides a principled solution by treating model weights as probability distributions rather than single point estimates, but it is computationally expensive to scale to deep learning.

The "gold standard" for Bayesian inference is Hamiltonian Monte Carlo (HMC), but it requires processing the entire dataset at once, rendering it unusable for modern, large-scale neural networks. Stochastic Gradient Langevin Dynamics (SGLD) was developed as a mini-batch alternative. However, SGLD often suffers from "poor mixing"—it moves through the parameter space slowly and produces highly correlated samples. While "Riemannian" versions of SGLD attempt to fix this by using local geometry (the Fisher Information matrix) to guide the sampler, calculating the full Fisher matrix for a large network is mathematically intractable. Most researchers settle for a "diagonal" approximation, which is fast but discards most of the useful geometric information.

This paper improves SGLD by incorporating K-FAC (Kronecker-Factored Approximate Curvature) as a preconditioner. By exploiting the structural geometry of neural networks, the authors achieve better sampling efficiency and higher predictive accuracy than standard diagonal methods.

The KFAC-SGLD Method

The authors propose KFAC-SGLD, which uses a sophisticated yet tractable approximation of the Fisher Information matrix to precondition the sampler.

  1. Preconditioning SGLD: Standard SGLD is essentially Stochastic Gradient Descent (SGD) with a specific amount of Gaussian noise added to the updates. To make the sampler move more efficiently, the gradient can be multiplied by a "preconditioning" matrix. This warps the parameter space so that the sampler avoids getting stuck in narrow valleys or moving too slowly across flat plateaus.
  2. The K-FAC Innovation: K-FAC approximates the Fisher matrix by assuming that weights in different layers are independent and that activations are independent of backpropagated derivatives. This allows the massive Fisher matrix to be broken down into a block-diagonal structure where each block is a Kronecker product of two much smaller matrices (activations and gradients).
  3. Efficient Sampling: A key technical contribution of this work is noting that the Cholesky factorization of a Kronecker product is simply the Kronecker product of the individual Cholesky factorizations. This allows the authors to sample the required noise for SGLD very efficiently without ever constructing the full, giant covariance matrix.

Ultimately, KFAC-SGLD acknowledges that neural network parameters should not be treated as a flat vector. By respecting the layer-wise structure of the network, K-FAC captures the "natural gradient" of the loss surface much better than diagonal methods like RMSProp, providing a "fast and smart" middle ground between diagonal approximations and the full Fisher matrix.

Empirical Results

The paper evaluates KFAC-SGLD on both simulated 1D data and the Boston Housing regression benchmark.

Qualitative Superiority on 1D Data

To visualize how well the sampler captures the true distribution, the authors tested the method on a synthetic function sampled from a Gaussian Process.

Comparison of PSGLD and KFAC-SGLD on 1D simulated data and test log-likelihood.

As shown above, KFAC-SGLD produced a sample distribution that much more closely matched the ground truth compared to preconditioned SGLD (PSGLD) using RMSProp-style diagonal scaling. The test log-likelihood for KFAC-SGLD also converges faster and remains more stable.

Predictive Performance and Efficiency

On the Boston Housing dataset (a standard regression benchmark with 506 examples and 13 features), the authors trained a single hidden layer neural network. KFAC-SGLD achieved a test log-likelihood of $-2.226 \pm 0.069$, significantly outperforming previous state-of-the-art results on the same task ($-2.574 \pm 0.089$).

Effective Sample Size and Test Log Likelihood comparisons demonstrating KFAC-SGLD's efficiency.

Because K-FAC understands the curvature of the space, the sampler can take more "meaningful" steps. This leads to a higher Effective Sample Size (ESS), meaning the samples are less redundant and provide a better representation of the posterior distribution. The preconditioning helps the sampler explore the parameter space more effectively than standard SGLD.

Limitations and Future Work

  • Scale of Evaluation: The empirical results in this paper are limited to relatively small-scale problems (1D toy data and the Boston Housing regression task). Testing on larger classification tasks (like ImageNet) or complex Reinforcement Learning environments is a necessary next step to prove the method's general scalability.
  • Beyond SGLD: The authors suggest that the K-FAC preconditioning trick is not strictly limited to Langevin dynamics. It could potentially be applied to improve other MCMC methods, including Hamiltonian Monte Carlo.
  • Hyperparameter Sensitivity: While not explicitly detailed as a failure mode, preconditioning methods often introduce new hyperparameters—such as the frequency of updating the K-FAC factors—that may require careful tuning in practice.

Summary updated

Abstract

Tractable approximate Bayesian inference for deep neural networks remains challenging. Stochastic Gradient Langevin Dynamics (SGLD) offers a tractable approximation to the gold standard of Hamiltonian Monte Carlo. We improve on existing methods for SGLD by incorporating a recently-developed tractable approximation of the Fisher information, known as K-FAC, as a preconditioner.
BibTeX
@inproceedings{nado2018stochastic,
  author = {Zachary Nado and Jasper Snoek and Roger B. Grosse and David Duvenaud and Bowen Xu and James Martens},
  title = {{Stochastic Gradient Langevin dynamics that Exploit Neural Network Structure}},
  booktitle = {Workshops @ ICLR},
  year = {2018},
  url = {https://openreview.net/forum?id=rTMEcR0JE-5},
}
Figure