Evaluating Approximate Inference in Bayesian Deep Learning

Andrew Gordon Wilson, Pavel Izmailov, Matthew D. Hoffman, Yarin Gal, Yingzhen Li, Melanie F. Pradier, Sharad Vikram, Andrew Y. K. Foong, Sanae Lotfi, Sebastian FarquharNeurIPS 2021 · OpenReview

TL;DR

This paper summarizes the NeurIPS 2021 competition designed to measure the "fidelity" of approximate Bayesian inference in deep learning. By using massive-scale Hamiltonian Monte Carlo (HMC) as a "ground truth" reference, the authors show that the most successful methods are those that capture multiple modes of the posterior—effectively proving that deep ensembles are a powerful, "Bayesian" tool for uncertainty.

What Problem Does This Paper Address?

Deep learning models are notoriously overconfident. While Bayesian methods offer a principled way to represent epistemic uncertainty (the uncertainty stemming from a lack of data), the "true" Bayesian posterior is mathematically intractable for neural networks.

Researchers usually settle for approximate inference (like Variational Inference or Dropout). However, we’ve lacked a way to tell if these approximations are actually working as intended. Usually, we just measure test accuracy or log-likelihood, but those metrics conflate the quality of the model with the quality of the inference. This paper addresses the need for a "fidelity" benchmark: how closely does an approximation match the true Bayesian posterior predictive distribution?

Methods

To create a "gold standard," the authors ran Hamiltonian Monte Carlo (HMC), a gold-standard MCMC method that is guaranteed to converge to the true posterior but is prohibitively expensive. They parallelized these computations over hundreds of Google TPU devices, spending tens of thousands of training epochs to produce a single posterior sample.

Participants were then challenged to approximate these HMC results within a strict computational budget (no more than 1,000 SGD training epochs). The competition used three primary metrics to compare submissions against the HMC reference:

  • Top-1 Agreement: The fraction of test points where the approximation and HMC agree on the most likely class.
  • Total Variation (TV): A measure of how closely the full probability distributions (across all classes) match.
  • Wasserstein-2 Distance: Used specifically for regression tasks to measure the distance between predicted distributions.

Datasets

The competition utilized a mix of standard and specialized benchmarks across different architectures to thoroughly evaluate the submissions.

Summary of datasets, architectures, and evaluation metrics used in the competition.
  • Classification: CIFAR-10 and CIFAR-100 (using ResNet-20), and IMDB (using a CNN-LSTM).
  • Robustness: Corrupted versions of CIFAR-10 (CIFAR-10-C) to test performance under covariate shift.
  • Medical/Specialized: MedMNIST (specifically DermaMNIST for skin lesion classification) using LeNet-5.
  • Regression: The UCI Energy dataset using a fully connected neural network (FCNN).

Key Insights and Ideas

  • Ensembling is Bayesian: Every single one of the top-5 finishing teams used some form of ensembling to create a multimodal approximation. This suggests that capturing different "modes" (peaks) in the posterior landscape is more important than the fine-grained quality of the approximation within a single mode.
  • The "Non-Bayesian" Myth: The results suggest we should stop viewing deep ensembles as a "non-Bayesian" ad-hoc trick. Instead, they are a very strong baseline for matching the true HMC predictive distribution.
  • Hyperparameters Matter: Multiple participants noted that the quality of their posterior approximation was highly sensitive to hyperparameter tuning, highlighting a need for better automated tuning in Bayesian DL.
  • The "Infinite Width" Gap: Interestingly, the authors found that "infinite width" limits (Gaussian Processes) performed poorly in matching HMC results on the UCI benchmark, suggesting that finite-width neural networks have unique posterior characteristics that GPs don't capture.

Key Results

The winning team, moellenh, utilized an innovative method based on the Bayesian Learning Rule (BLR) and an "improved Variational Online-Newton" (iVON) optimizer. They constructed a Gaussian mixture posterior around multiple independent modes.

Competition leaderboard showing top team performances across CIFAR-10, MedMNIST, and UCI datasets.
  • CIFAR-10 Fidelity: As shown in the results, the winner achieved a Top-1 agreement of 0.787 and a Total Variation of 0.198 compared to HMC on CIFAR-10.
  • The Robustness Gap: All methods struggled significantly more on corrupted data (CIFAR-10-C). While the best agreement on clean data was ~91%, it dropped to 78.7% on corrupted data. This indicates that current approximate inference methods still fail to capture how the true Bayesian posterior behaves when faced with out-of-distribution inputs.

Limitations and Future Work

The primary limitation is that even HMC is technically an approximation (though the best one available). Furthermore, the computational cost of generating HMC "ground truth" limits the benchmark to relatively small architectures like ResNet-20.

The authors suggest that future work should focus on:

  1. Improving performance on covariate shift (corrupted data), where the gap between approximations and HMC remains large.
  2. Developing more efficient multimodal approximations that can scale to larger, modern architectures without the massive overhead of HMC.
  3. Using the released HMC samples as a permanent benchmark for the community to test new inference algorithms.

Summary updated

Abstract

Uncertainty representation is crucial to the safe and reliable deployment of deep learning. Bayesian methods provide a natural mechanism to represent epistemic uncertainty, leading to improved gene...
BibTeX
@inproceedings{wilson2021evaluating,
  author = {Andrew Gordon Wilson and Pavel Izmailov and Matthew D. Hoffman and Yarin Gal and Yingzhen Li and Melanie F. Pradier and Sharad Vikram and Andrew Y. K. Foong and Sanae Lotfi and Sebastian Farquhar},
  title = {{Evaluating Approximate Inference in Bayesian Deep Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=KS-pY0VqfOX},
}
Figure