Stable Process Limits of Heavy-Tailed Neural Networks

23 concepts13 papers1 proposal

In deep learning theory, an infinitely wide neural network is traditionally treated as a highly predictable, smooth object. Under standard assumptions, as the width of a network's hidden layers approaches infinity, the network converges to a Gaussian Process (GP), and its training dynamics become governed by a deterministic Neural Tangent Kernel (NTK).

This elegant framework relies entirely on the classical Central Limit Theorem (CLT), which assumes that the network's weights possess finite variance and that no single hidden unit dominates the network's output. However, this Gaussian assumption often fails to reflect reality. Emerging research demonstrates that both the implicit dynamics of optimization and intentional architectural choices naturally push neural networks into a heavy-tailed, non-Gaussian regime.

The Breakdown of the Gaussian Assumption

Why do we expect weights to follow a neat, finite-variance bell curve? In practice, they rarely do. Early foundational work by Leen et al. in Weight Space Probability Densities in Stochastic Learning: I. Dynamics and Equilibria demonstrated that the equilibrium distribution of weights under stochastic gradient descent (SGD) inherently features heavier tails than a standard Gaussian. Because optimization noise increases as the model moves away from a local optimum, weights are frequently kicked far out into the tails of the distribution.

Furthermore, Mukherjee et al. proved in Asymptotics of Gradient-based Neural Network Training Algorithms that for any fixed, non-zero learning rate, the true limiting distribution of trained weights is fundamentally non-Gaussian. As training progresses, SGD actively drives parameters toward heavy-tailed distributions where variance can become poorly behaved or mathematically infinite, a phenomenon highlighted by Gurbuzbalaban et al. in Fractional moment-preserving initialization schemes for training deep neural networks.

Heavy tails are also prominent in biological systems. As noted by Xie et al. in Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networks, structural connectivity in the brain is dominated by a small number of "mega-synapses" that are orders of magnitude stronger than the average connection, a structure that cannot be captured by standard Gaussian modeling.

From Gaussian Processes to Stable Processes

When weight distributions exhibit infinite variance, the standard Central Limit Theorem breaks down. In its place, the Generalized Central Limit Theorem dictates that the sum of these variables converges not to a Gaussian distribution, but to a Lévy $\alpha$-stable distribution.

The mathematical consequences for infinite-width networks are profound. As Der et al. proved in Beyond Gaussian Processes: On the Distributions of Infinite Networks, initializing an infinite network with heavy-tailed, symmetric stable weights causes the network to converge to a Stable Process rather than a Gaussian Process.

Weight Variance     CLT Regime                  Infinite-Width Limit
────────────────────────────────────────────────────────────────────────
Finite Variance  ──► Standard CLT            ──► Gaussian Process (GP)
                                                 (Infinitesimal units)

Infinite Variance ──► Generalized CLT        ──► Stable Process
(Heavy Tails)                                    (Outlier units dominate)

To understand the intuition behind why this works, consider the infinitesimal requirement of standard Gaussian limits. For a network to become a GP, the scaling must ensure that the relative contribution of every individual hidden unit vanishes to zero as width approaches infinity; no single neuron can sway the output.

When weights are drawn from heavy-tailed stable distributions, this requirement is violated. As shown by Peluchetti et al. in Stable behaviour of infinitely wide deep neural networks, heavy tails introduce macroscopic "outlier" weights. These outliers retain significant, non-negligible influence over the network's activations even as the layer width approaches infinity. While standard Gaussian networks smooth out all individual variations, heavy-tailed networks preserve the distinct impact of prominent hidden features.

Unlocking Representation Learning and Discontinuities

This shift from Gaussian to stable process limits fundamentally alters what the network can compute and learn. Standard infinite-width Gaussian networks suffer from a major theoretical limitation: their NTK is completely deterministic and frozen during training, meaning the network acts as a fixed-feature linear model and cannot perform true representation learning.

Heavy-tailed initializations break this stagnation. Favaro et al. demonstrated in Large-width asymptotics and training dynamics of $\alpha$-Stable ReLU neural networks that infinite-width $\alpha$-stable networks require a unique logarithmic scaling and evolve under a random (stochastic) NTK. Because a few dominant hidden units retain macroscopic weight, the network's kernel remains dynamic and data-dependent during training, enabling the model to learn representations even in the infinite-width limit.

Additionally, standard Gaussian Processes are structurally limited by their smoothness, making them poor choices for modeling data with abrupt changes. In contrast, stable processes naturally accommodate sharp discontinuities. As Loria et al. showed in Deep Kernel Posterior Learning under Infinite Variance Prior Weights, infinite-width Bayesian neural networks with heavy-tailed priors yield stochastic kernels capable of naturally capturing non-smooth behavior and jump discontinuities in data.

Initialization     Infinite-Width Kernel     Function Space Properties
────────────────────────────────────────────────────────────────────────
Gaussian           Deterministic NTK         Smooth paths, 
                   (No representation learning) no representation learning

α-Stable           Random/Stochastic NTK     Discontinuous jumps, 
                   (Active representation learning) representation learning

While traditional asymptotic analyses, such as those reviewed by Bracale et al. in Large-width functional asymptotics for deep Gaussian neural networks and Panahi et al. in Precise Asymptotic Analysis of Deep Random Feature Models, rely on Gaussian bounds to simplify error equations, they miss these rich functional landscapes that non-Gaussian limits unlock.

Taming the Instability of Infinite Variance

Despite their theoretical advantages, heavy-tailed distributions are notoriously difficult to engineer. If activations propagate through a deep network unchecked, infinite-variance weights can cause signals to explode or diverge. For instance, Sedghi et al. in On the effect of the activation function on the distribution of hidden nodes in a deep network proved that certain activation functions inadvertently transform hidden node distributions into Cauchy distributions, causing network activations to destabilize as layers deepen.

To safely harness heavy tails, researchers have developed specialized initialization and inference frameworks:

By replacing the restrictive Gaussian assumption with the broader mathematics of stable processes, this research direction provides a more realistic theoretical foundation for deep learning—one that accounts for the heavy-tailed realities of optimization and unlocks the true expressive power of infinite-width networks.

Key Foundational Work

Weight Space Probability Densities in Stochastic Learning: I. Dynamics and Equilibria (1992) — contributed 2 concept elements to this direction.
Asymptotics of Gradient-based Neural Network Training Algorithms (1994) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_heavy_tailed_weight_initialization_5cf03846,
  title = {heavy-tailed weight initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heavy-tailed-weight-initialization/rdr_5cf0384602fb47acb610740b237b1ba7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_heavy_tailed_weight_initialization_5cf03846_v1,
  title = {heavy-tailed weight initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heavy-tailed-weight-initialization/rdr_5cf0384602fb47acb610740b237b1ba7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake