Heavy-Tailed Noise and Generalization in SGD

This direction is merged into another direction: heavy-tailed gradient noise. Its last version stays readable and citable here.

Version 1 of 112 papers18 conceptsfrozen 2026-10-04

In classical optimization theory, stochastic gradient descent (SGD) is often modeled as a physical system subject to Gaussian noise—similar to a particle vibrating with thermal energy. However, this "thermal" view fails to explain why SGD is so effective at finding solutions that generalize well to unseen data. A growing body of research suggests that the secret lies in the nature of the noise itself: it isn't Gaussian, but "heavy-tailed."

The Jump Mechanism vs. The Climb

In a traditional Gaussian noise model, escaping a local minimum (a "basin") requires the optimizer to gradually climb over the surrounding hills. This process is exponential; the time it takes to escape depends primarily on the depth of the basin. Deep basins become traps, even if they represent poor solutions.

Research by Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban in A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks demonstrates that stochastic gradient noise (SGN) follows an $\alpha$-stable distribution. Because these distributions have heavy tails, they produce occasional "shocks" or large jumps.

Under heavy-tailed noise, the optimizer doesn't climb; it jumps out of the basin. In this regime, the probability of escaping depends on the width of the basin rather than its depth. Sharp, narrow minima are small targets that are easily jumped over, while wide, flat minima are large targets where the optimizer is more likely to land and stay.

Why SGD Outperforms Adaptive Methods

A persistent mystery in deep learning is why simple SGD often generalizes better than adaptive methods like ADAM. Pan Zhou, Jiashi Feng, and colleagues address this in Towards Theoretically Understanding Why Sgd Generalizes Better Than Adam in Deep Learning.

They argue that ADAM’s adaptive scaling effectively "thins" the tails of the noise. By normalizing the gradients, ADAM makes the noise behave more like Gaussian noise. This forces the optimizer to spend more time in sharp minima. SGD, by preserving the heavy-tailed nature of the noise, perceives these sharp regions as having a smaller "measure," allowing it to quickly exit them in search of flatter landscapes that correlate with better generalization.

The Geometry of Escape

The dynamics of this escape are not just faster; they are qualitatively different. Takashi Mori and Ziyin Liu show in Power-Law Escape Rate of SGD that the escape rate follows a power-law rather than an exponential one. This suggests that the optimizer's movement is dictated by the ratio of loss values, effectively navigating a "logarithmic" landscape.

Furthermore, this noise is not uniform across the model. Barak Battash and Ofir Lindenbaum found in Revisiting the Noise Model of Stochastic Gradient Descent that specific "noisier" parameters drive the exploration. These parameters act as catalysts, determining the specific axes along which the model will jump between basins.

Controlling the Trajectory

Recent work has moved toward using these insights to steer training. In Global Dynamics of Heavy-Tailed SGDs in Nonconvex Loss Landscape: Characterization and Control, Chang-Han Rhee demonstrates that by combining heavy-tailed noise with techniques like gradient clipping, we can mathematically ensure the optimizer ignores narrow valleys. As the learning rate decreases, the process settles into a "Markov jump process" that naturally favors the most stable, wide regions of the loss surface, providing a theoretical guarantee for the "flat minima" folk wisdom of deep learning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_heavy_tailed_noise_and_generalization_in_39601172,
  title = {Heavy-Tailed Noise and Generalization in SGD},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heavy-tailed-noise-and-generalization-in-sgd/rdr_39601172013d4924924d82587f1e1fc7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_heavy_tailed_noise_and_generalization_in_39601172_v1,
  title = {Heavy-Tailed Noise and Generalization in SGD},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heavy-tailed-noise-and-generalization-in-sgd/rdr_39601172013d4924924d82587f1e1fc7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}