Heavy Tailed Priors in Deep Bayesian Neural Networks

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 219 papers53 conceptsfrozen 2026-10-04

In Bayesian Neural Networks (BNNs), we typically place a simple Gaussian prior on the weights. This is mathematically equivalent to $L_2$ regularization (weight decay), a staple of classical deep learning. However, while the weights might be Gaussian, the signals they produce—the hidden unit activations—are anything but.

As information propagates through a deep network, the distribution of these activations undergoes a systematic transformation. Recent research reveals that depth acts as a "tail-fattener," turning light-tailed Gaussian inputs into heavy-tailed distributions. Understanding this transformation is key to moving beyond the "infinite-width" approximations that dominate theoretical BNN research.

The Sub-Weibull Transformation

When we initialize a network with Gaussian weights, we might expect the hidden units to remain roughly Gaussian. But as Mariia Vladimirova, Julyan Arbel, and colleagues show in Understanding Priors in Bayesian Neural Networks at the Unit Level, the marginal distribution of hidden units becomes increasingly heavy-tailed with depth.

They characterize these distributions using the sub-Weibull family. A sub-Weibull distribution is defined by a tail parameter $\theta$. When $\theta = 1/2$, the distribution is Gaussian (light-tailed). When $\theta = 1$, it is exponential. As $\theta$ increases, the tails become "heavier," meaning the network is significantly more likely to produce extreme outlier activations than a Gaussian model would predict.

The researchers found a striking regularity: in deep BNNs, this tail parameter $\theta$ scales linearly with the layer index $L$. Even if you start with Gaussian weights at the first layer, by the time the signal reaches the $L$-th layer, the units follow a sub-Weibull distribution with a tail heaviness proportional to $L$. This property persists across various architectures, including convolutional layers, and survives common operations like max-pooling and batch normalization.

The Tug of War Between Depth and Width

For years, the theoretical gold standard for BNNs has been the "infinite-width limit," which posits that as layers get infinitely wide, the network converges to a Gaussian Process (GP). In this limit, every unit in the network is Gaussian.

However, Jacob A Zavatone-Veth and Cengiz Pehlevan argue in Exact marginal prior distributions of finite Bayesian neural networks that this approximation is fundamentally misleading for the deep networks used in practice. They demonstrate a "tug-of-war" between two architectural forces:

  1. Depth increases non-Gaussianity: Each successive layer compounds the multiplicative interactions of weights, pushing the distribution toward heavy tails and extreme outliers.
  2. Width increases Gaussianity: Following the Central Limit Theorem, increasing the number of units in a layer sums up these signals, smoothing the distribution back toward a Gaussian shape.

Geoff Pleiss and John P. Cunningham take this further in The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective. They suggest that the heavy tails created by depth are actually a feature, not a bug. These non-Gaussian distributions allow the network to capture complex, sparse features. Excessive width acts as a regularizer that destroys these features, collapsing the hierarchical model into a simple, shallow GP.

Why Heavy Tails Matter for Optimization

This heavy-tailed behavior isn't just a curiosity of the prior; it also emerges during training. Liam Hodgkinson and Michael W. Mahoney show in Multiplicative Noise and Heavy Tails in Stochastic Optimization that the multiplicative noise inherent in Stochastic Gradient Descent (SGD) naturally drives parameters toward heavy-tailed distributions.

These heavy tails empower the optimizer to "hop" across high-energy barriers in the loss landscape, escaping local minima that would trap a Gaussian optimizer. This suggests that the success of deep learning may be tied to the model's ability to maintain these "fat-tailed" distributions, which provide the flexibility to represent complex data and the mobility to find good solutions.

Rethinking Regularization

If the natural state of a deep network is heavy-tailed, then our standard regularization techniques might be working against the architecture. David H. Wolpert noted as early as 1993 in Bayesian Backpropagation Over I-O Functions Rather Than Weights that uniform priors across all layers are theoretically suboptimal. Because the sensitivity of the output function to the weights changes at different depths, we should apply different priors to different layers.

Modern evidence from Bayes in action in deep learning and dictionary learning supports this, suggesting that we should use non-standard regularization penalties that account for the layer index. Instead of a one-size-fits-all Gaussian weight decay, we may need "depth-aware" priors that align with the natural sub-Weibull evolution of the network's internal signals.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 12 papers
  2. Version 1 · 2026-10-04 · 19 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_depth_induced_heavy_tailedness_33f757be,
  title = {depth-induced heavy-tailedness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-induced-heavy-tailedness/rdr_33f757be19f54cb190204f295f209e4c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_heavy_tailed_priors_in_deep_bayesian_neu_33f757be_v1,
  title = {Heavy Tailed Priors in Deep Bayesian Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heavy-tailed-priors-in-deep-bayesian-neural-networks/rdr_33f757be19f54cb190204f295f209e4c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}