Depth Induced Heavy Tailed Distributions in Neural Networks

24 concepts12 papers1 proposal

When initializing a Bayesian Neural Network (BNN), a standard practice is to assign independent Gaussian priors to the weights. Conventional wisdom—rooted in the Central Limit Theorem—suggests that if a layer is wide enough, the sum of these Gaussian inputs will ensure that the pre-activations (the signals entering each activation function) remain Gaussian as well. This assumption underpins many popular theoretical frameworks, including infinite-width approximations and Deep Gaussian Processes.

However, in realistic finite-width networks, this Gaussian assumption breaks down completely. As signals propagate deeper into a network, the marginal distributions of the hidden units undergo a dramatic transformation: they develop increasingly heavy tails. Understanding why this happens, and what it means for regularization and optimization, is a rapidly growing focus in deep learning theory.

The Mechanism of Tail Amplification

Why does depth breed heavy tails? The phenomenon stems from the compositional nature of deep architectures. Passing an input through successive layers is fundamentally a multiplicative process: inputs are multiplied by weights, transformed by non-linearities, and multiplied again. When random variables are repeatedly compounded in this manner, the probability of generating extreme outlier values increases layer by layer.

Vladimirova et al. (icml 2019) and Arbel et al. (ESAIM 2023) formalized this progression mathematically. They showed that while the first hidden layer remains Gaussian, subsequent layers quickly transition to sub-exponential and eventually to sub-Weibull distributions.

Crucially, the tail parameter characterizing these distributions scales linearly with the layer index. For a network utilizing Rectified Linear Unit (ReLU) activations, Vladimirova et al. (ACML 2021) demonstrated that the tail parameter decays inversely with the layer index (following a strict $2/l$ progression). This means that before any training occurs, the architecture itself naturally biases deeper layers to favor more extreme activation values than shallower layers.

Finite Width vs. Infinite Width Collapse

To understand why this heavy-tailed behavior is often overlooked, we have to look at the tug-of-war between network width and network depth.

In an infinitely wide network, the law of large numbers dominates. As Pleiss et al. (neurips 2021) demonstrate, increasing width acts as a regularizer that forces the output distribution back into a benign Gaussian form. However, this "Gaussian collapse" comes at a cost: excessive width strips away the model's non-Gaussian representation capacity, reducing a deep hierarchical model to a simple, shallow Gaussian Process.

Width Increases  ---> Pushes distribution toward Gaussian (Reduces representation capacity)
Depth Increases  ---> Pushes distribution toward Heavy-Tailed Sub-Weibull (Increases complexity)

In actual finite-width networks, depth wins this tug-of-war. Zavatone-Veth et al. (neurips 2021) and Noci et al. (neurips 2021) derived the exact marginal prior distributions for finite ReLU networks using Meijer-G functions. They proved that as depth increases, the prior predictive distribution deviates severely from a Gaussian, ultimately converging to a normal log-normal mixture. This structural divergence means that deep, narrow networks inherently suffer from signal drift and extreme output values, which can severely threaten learning stability.

The Flaw of Uniform Weight Priors

If the internal representations of a network naturally become heavy-tailed with depth, treating every layer identically during training is theoretically suboptimal. Standard regularization techniques, such as weight decay, apply a uniform Gaussian prior across all layers, assuming each weight contributes equally to the model's behavior.

This assumption ignores layer-specific sensitivity. Decades ago, Wolpert (neurips 1993) argued that the mapping from weights to the overall input-output function varies significantly across different depths, meaning effective Bayesian modeling requires distinct treatment for parameters at different layers.

When we look at modern deep networks, applying a single global prior fails to account for how structural properties shift with depth. To fix this, Fortuin et al. (iclr 2022) and Garriga-Alonso et al. (arxiv 2021) demonstrated that substituting global Gaussians with architecture- and depth-dependent priors—such as heavy-tailed Student-t or Cauchy distributions—significantly improves model calibration and helps resolve practical issues like the "cold posterior effect."

Taming vs. Exploiting the Tails

Faced with the reality of depth-induced heavy tails, researchers have split into two camps: those trying to eliminate the phenomenon to stabilize networks, and those trying to exploit it to improve optimization.

Taming the Tails

If your goal is to preserve the predictable theoretical guarantees of Gaussian pre-activations in finite networks, standard initializations will not work. To counter this, Wolinski et al. (Accepted by TMLR 2025) designed a novel family of activation functions paired with symmetric Weibull initialization distributions. This combination mathematically forces pre-activations to remain Gaussian regardless of the network's depth or width, preventing unwanted signal drift.

Exploiting the Tails

Conversely, heavy tails might not be a bug to be fixed, but rather the "secret sauce" behind why deep networks optimize so well. Hodgkinson et al. (icml 2021) proved that the multiplicative noise inherent in stochastic gradient descent (SGD) naturally induces heavy-tailed parameter distributions. This power-law behavior allows the optimizer to make large, discrete "hops" across complex loss landscapes, escaping suboptimal local minima that would trap standard Gaussian-bound optimizers.

Furthermore, Sommer et al. (icml 2024) discovered that the connectivity of high-probability regions in the posterior distribution increases with depth. This diffuse, interconnected structure suggests that the heavy-tailed overparameterization of deeper layers makes it significantly easier for sampling algorithms to explore the weight space efficiently.

Key Foundational Work

Understanding Priors in Bayesian Neural Networks at the Unit Level (2019) — contributed 4 concept elements to this direction.
Bayesian Backpropagation Over I-O Functions Rather Than Weights (1993) — contributed 3 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 12 papers
  2. Version 1 · 2026-10-04 · 19 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_depth_induced_heavy_tailedness_33f757be,
  title = {depth-induced heavy-tailedness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-induced-heavy-tailedness/rdr_33f757be19f54cb190204f295f209e4c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_depth_induced_heavy_tailedness_33f757be_v2,
  title = {depth-induced heavy-tailedness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-induced-heavy-tailedness/rdr_33f757be19f54cb190204f295f209e4c/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake