Bayesian Layers: A Module for Neural Network Uncertainty

Dustin Tran, Mike Dusenberry, Mark van der Wilk, Danijar Hafner

TL;DR Researchers from Google Brain and Prowler.io have introduced Bayesian Layers, a software module providing drop-in, probabilistic replacements for standard neural network layers. By unifying Bayesian neural networks, Gaussian processes, and flow-based models under a single API, this work demonstrates that uncertainty estimation can finally scale to massive models, including a 5-billion parameter Transformer.

The Challenge of Overconfident Models

Standard deep learning models are notoriously overconfident. They provide a single "best guess" for their predictions but lack a principled way to express uncertainty. While Bayesian methods offer a solution by treating weights and activations as probability distributions rather than fixed numbers, they have historically been difficult to implement and even harder to scale.

Existing probabilistic programming languages (PPLs) often require researchers to rewrite their models from scratch or use complex, non-standard abstractions. This creates a "usability gap" where state-of-the-art deep learning architectures (like Transformers or LSTMs) are rarely paired with robust uncertainty quantification. The authors aim to bridge this gap by making "Bayesian-ness" a property of the layer itself, fitting seamlessly into the workflows of modern libraries.

Methodology: The "Drop-in" Philosophy

The core contribution is a library of Bayesian Layers that mirror the API of standard layers. Instead of a standard dense layer, a developer can simply use a variational equivalent. You shouldn't need a PhD in statistics to add uncertainty to a model; by maintaining the constructor signatures of standard layers, the barrier to entry for Bayesian deep learning is significantly lowered.

Key technical approaches include:

  • Variational Inference (VI): The layers use the "reparameterization trick" to approximate complex integrals. They optimize the Evidence Lower Bound (ELBO), balancing model fit against a complexity penalty (KL divergence).
  • Overloaded Initializers and Regularizers: To keep the API familiar, standard arguments are repurposed. For example, a kernel initializer in a Bayesian layer returns a distribution over weights rather than a single point estimate.
  • Unified Abstraction: The module treats diverse probabilistic concepts—Bayesian Neural Nets (uncertainty over weights), Gaussian Processes (uncertainty over functions), and Reversible Layers (for flow-based models)—as interchangeable components.
  • Edward2 Integration: The system leverages the Edward2 PPL to handle random variables, allowing these layers to be used within larger, more complex probabilistic programs.

Experiments and Key Results

The authors evaluated their module across a wide range of tasks, including machine translation, language modeling, image generation, and reinforcement learning, to prove its versatility.

Massive Scalability and Efficiency

By making uncertainty a local property of a layer rather than a global property of the whole model, the authors enabled model-parallel training. They successfully fit a 5-billion parameter Bayesian Transformer, utilizing up to 2500 TFLOPs. This is orders of magnitude larger than typical Bayesian neural networks. Training performance scaled linearly as the number of TPU shards increased, proving that Bayesian methods don't have to be the bottleneck in high-performance computing.

Linear scaling of training performance across TPUv2 shards

State-of-the-Art Performance

The Bayesian Transformer achieved a state-of-the-art BLEU score of 43.9 on the WMT14 EN-FR dataset and maintained a competitive perplexity of 23.1 on the One-Billion Word Benchmark.

Robust Reinforcement Learning

In reinforcement learning experiments using the DeepMind Control Suite's "Cheetah" task, the Bayesian PlaNet agent matched the performance of the original deterministic version. Furthermore, it provided a "non-trivial belief" (monitored via KL divergence) indicating it successfully learned a distribution over dynamics. The authors found that the Bayesian version captured more precise velocity predictions in its latent space, suggesting that probabilistic modeling helps agents build more accurate internal simulations of their environment.

Performance and Weight KL divergence for the Bayesian PlaNet agent on the Cheetah Run task

Limitations and Future Work

The primary limitation acknowledged is that the current framework is tightly coupled with Variational Inference (VI). Because the modularity is defined per-layer, it is difficult to implement inference algorithms that require global coordination across the entire parameter space, such as certain flavors of Markov chain Monte Carlo (MCMC).

The authors also noted that while they achieved high performance in text applications, scaling Bayesian neural nets for language remains a challenge that requires further research into how uncertainty interacts with discrete data. Future work will likely focus on expanding the library to include more stochastic processes and refining the integration with other probabilistic inference engines.

Summary updated

Abstract

We describe Bayesian Layers, a module designed for fast experimentation with neural network uncertainty. It extends neural network libraries with drop-in replacements for common layers. This enables composition via a unified abstraction over deterministic and stochastic functions and allows for scalability via the underlying system. These layers capture uncertainty over weights (Bayesian neural nets), pre-activation units (dropout), activations (<code>stochastic output layers''), or the function itself (Gaussian processes). They can also be reversible to propagate uncertainty from input to output. We include code examples for common architectures such as Bayesian LSTMs, deep GPs, and flow-based models. As demonstration, we fit a 5-billion parameter</code>Bayesian Transformer'' on 512 TPUv2 cores for uncertainty in machine translation and a Bayesian dynamics model for model-based planning. Finally, we show how Bayesian Layers can be used within the Edward2 language for probabilistic programming with stochastic processes.
BibTeX
@inproceedings{tran2019bayesian,
  author = {Dustin Tran and Mike Dusenberry and Mark van der Wilk and Danijar Hafner},
  title = {{Bayesian Layers: A Module for Neural Network Uncertainty}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=6tdMo4HTRm},
}

Loading related content...

Figure