Heavy Tailed Dynamics in Stochastic Gradient Descent

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 217 papers26 conceptsfrozen 2026-10-04

Stochastic Gradient Descent (SGD) is often described as a noisy version of gradient descent. In the standard view, this noise is a byproduct of using mini-batches, acting as a simple regularizer. However, a growing body of research suggests that the noise in SGD is far more structured and fundamental than previously thought. Instead of following a standard Bell curve (Gaussian distribution), the parameters in SGD often follow heavy-tailed distributions, where extreme "jumps" are much more common than expected.

This heavy-tailed behavior is not an accident; it is the "secret sauce" that allows deep learning models to escape poor local minima and find solutions that generalize well to new data.

The Origin of Heavy Tails: Multiplicative Noise

The primary reason SGD exhibits heavy tails is that its noise is multiplicative, not just additive. In a simple system with additive noise, the uncertainty is independent of the current state. But in SGD, the noise scales with the magnitude of the parameters themselves.

Liam Hodgkinson and Michael W. Mahoney demonstrated in Multiplicative Noise and Heavy Tails in Stochastic Optimization that modeling optimizers as random recurrence relations reveals this multiplicative structure. Because the noise depends on the current weight values, the system can experience feedback loops that push parameters far from their mean, resulting in a power-law distribution rather than a Gaussian one. Even when the underlying data is "light-tailed," the optimization process itself generates these heavy tails.

The Hyperparameter Dial: $\eta/b$

If heavy tails are the engine of exploration, the ratio of the learning rate ($\eta$) to the batch size ($b$) is the dial that controls it.

Research by Mert Gürbüzbalaban, Umut Simsekli, and Lingjiong Zhu in The Heavy-Tail Phenomenon in SGD proved that the "heaviness" of the tails (quantified by the tail index $\alpha$) is directly determined by the ratio $\eta/b$ and the local curvature of the loss landscape.

  • Increasing the learning rate or decreasing the batch size makes the tails "heavier" (smaller $\alpha$).
  • Heavier tails increase the probability of large, sudden updates in parameter space.

This explains the long-standing empirical observation that smaller batch sizes often lead to better generalization: they induce heavier-tailed dynamics, allowing the optimizer to "hop" out of sharp, narrow basins and settle into flatter, more robust regions of the loss landscape.

Why Heavy Tails Improve Generalization

The benefit of heavy-tailed noise lies in how it explores the parameter space. In a Gaussian world, the probability of a massive jump is astronomically low. In a heavy-tailed world, these jumps are a feature.

  1. Basin Hopping: As shown by Liam Hodgkinson and Michael W. Mahoney, heavy tails empower the optimizer to perform "basin hopping," effectively jumping over high-loss barriers that would trap a standard optimizer.
  2. Clustering and Fractal Structure: Alexander Camuto and Murat A. Erdogdu found that the trajectories of SGD often settle into fractal-like distributions. In Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms, they argue that the Hausdorff dimension of these fractal sets is a predictor of generalization. Noisier (heavier-tailed) dynamics often lead to "simpler" fractal structures with lower dimensions, which correlate with better performance on unseen data.
  3. Tighter Generalization Bounds: Liam Hodgkinson and Rajiv Khanna introduced the concept of lower tail exponents in Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers. They proved that heavy-tailed dynamics cause iterates to "cluster" more tightly in specific regions of the parameter space, which mathematically leads to tighter generalization bounds.

Beyond Constant Learning Rates

The heavy-tail effect can be intentionally amplified through learning rate schedules. Mert Gürbüzbalaban and Yuanhan Hu demonstrated in Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD that cyclic or randomized learning rates induce even heavier tails than constant rates. By periodically increasing the learning rate, practitioners are essentially "heating up" the system to trigger large jumps, ensuring the model doesn't get stuck in suboptimal regions before cooling down to converge.

Even in "offline" settings with finite datasets—where the noise technically cannot be a perfect power law—Kruno Lehman and Alain Durmus showed in Approximate Heavy Tails in Offline Stochastic Gradient Descent that the system still behaves as if it were heavy-tailed, converging to the same beneficial dynamics as the number of training steps increases.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 11 papers
  2. Version 1 · 2026-10-04 · 17 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_learning_rate_to_batch_size_ratio_9451055c,
  title = {learning rate to batch size ratio},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-rate-to-batch-size-ratio/rdr_9451055ce5a4459a9bdef02b20fc4b53},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_heavy_tailed_dynamics_in_stochastic_grad_9451055c_v1,
  title = {Heavy Tailed Dynamics in Stochastic Gradient Descent},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heavy-tailed-dynamics-in-stochastic-gradient-descent/rdr_9451055ce5a4459a9bdef02b20fc4b53/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}