Heavy Tailed Dynamics from Stochastic Hyperparameter Ratios

18 concepts9 papers

The remarkable ability of deep neural networks to generalize well remains one of the central mysteries of machine learning. Standard optimization theory suggests that Stochastic Gradient Descent (SGD) converges to local minima, but it fails to fully explain why it consistently finds good local minima that generalize well to unseen data.

A growing body of research reveals that the secret lies in the nature of SGD's noise. Rather than acting as a mild, Gaussian hum that gently shakes parameters, SGD noise is inherently heavy-tailed. This means the optimizer frequently undergoes massive, discontinuous "hops" across the loss landscape. Crucially, the intensity of this behavior is governed by a single, elegant lever: the ratio of the learning rate to the batch size ($\eta/b$).

The Origin of Heavy Tails: Multiplicative Noise

Why does an optimization algorithm operating on ordinary data suddenly exhibit heavy-tailed behavior? The answer lies in how the noise scales with the model's parameters.

In classical optimization analysis, noise is often assumed to be additive and independent of the current state. However, in SGD, the noise arises from sampling a mini-batch of data to estimate the true gradient. Because the gradient depends directly on the current weights, the variance of the noise scales with the magnitude of the weights themselves. This is known as multiplicative noise.

When an optimizer experiences multiplicative noise, its updates can be modeled as a random recurrence relation. As proven by Hodgkinson et al. in Multiplicative Noise and Heavy Tails in Stochastic Optimization, random recurrence relations naturally produce power-law (heavy-tailed) stationary distributions. Even if the underlying training data is entirely light-tailed, the algebraic structure of the SGD update rule forces the parameter distributions to develop heavy tails.

The Control Knob: The $\eta/b$ Ratio

The "heaviness" of a probability distribution's tail is quantified by its tail index ($\alpha$). A smaller $\alpha$ signifies a heavier tail, meaning extreme, far-leaping updates are more common. If $\alpha \le 2$, the distribution enters an infinite-variance regime where standard central limit theorems break down.

The primary mechanism for controlling this tail index is the ratio of the learning rate to the batch size ($\eta/b$).

↑ Learning Rate (η) or ↓ Batch Size (b) 
                │
                ▼
    ↑ Ratio (η/b) ──> ↑ Noise Covariance
                │
                ▼
    ↓ Tail Index (α) ──> Heavier Tails (More Aggressive Exploration)

As established by Gurbuzbalaban et al. in The Heavy-Tail Phenomenon in SGD, the tail index is explicitly governed by the interaction between the $\eta/b$ ratio and the local curvature of the loss surface. This occurs because the ratio directly scales the noise covariance of the stochastic process—a relationship formalized by Yin et al. in Stochastic Gradient and Langevin Processes.

When you increase the learning rate or decrease the batch size, you amplify the noise covariance, driving down the tail index and inducing heavier-tailed dynamics.

Why Heavy Tails Improve Generalization

How do these violent, heavy-tailed jumps translate into better machine learning models? It comes down to how the optimizer explores complex loss landscapes.

Loss landscapes in deep learning are highly non-convex, filled with a mixture of sharp local minima and broad, flat valleys. Sharp minima are notoriously poor at generalizing because a tiny shift in the data distribution drastically changes the loss. Flat minima, by contrast, are robust and generalize well.

Loss Landscape
       _       _
      / \     / \     Sharp Minimum (Escaped by heavy tails)
     /   \___/   \
    /             \   Flat Minimum (Stable, high generalization)
___/               \___

In an infinite-variance regime induced by a high $\eta/b$ ratio, the optimizer behaves like a Lévy flight rather than a standard Brownian motion. The frequent, large jumps allow the optimizer to easily "hop" out of narrow, sharp local minima. However, once the optimizer lands in a sufficiently broad, flat valley, the local curvature decreases, shifting the local tail index and stabilizing the iterates.

To theoretically ground this intuition, Hodgkinson et al. analyzed the trajectory of optimizers using the Fernique–Talagrand functional in Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers. The authors proved that a higher $\eta/b$ ratio decreases the local tail exponent of the transition kernel, forcing tighter spatial clustering of iterates within favorable regions and directly shrinking the generalization gap.

Real-World Nuances and Extensions

While the core theory relies on ideal mathematical assumptions, several works have mapped these heavy-tailed dynamics to practical deep learning settings:

  • Finite Data Constraints: True power-law distributions require infinite data. In practical "offline" training, we reuse a finite dataset across multiple epochs. Lehman et al. addressed this gap in Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descent, proving that offline SGD asymptotically approaches true power-law behavior as the dataset size grows, with the $\eta/b$ ratio remaining the dominant modulator of tail thickness.
  • Non-Constant Learning Rates: Modern training rarely uses a fixed learning rate. In Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize, Gurbuzbalaban et al. demonstrated that cyclic or randomized stepsize schedules widen the range of learning rates, invoking significantly heavier tails than constant stepsizes and enabling more aggressive exploration.
  • Regularization and Homogenization: The geometry of the landscape can be altered by training choices. Jiao et al. showed in Emergence of heavy tails in homogenized stochastic gradient descent that decreasing regularization strength works in tandem with a high learning rate to further exaggerate parameter tail heaviness.
  • Memory Effects: Looking beyond independent noise steps, Tan et al. modeled SGD through Fractional Brownian Motion in Understanding Short-Range Memory Effects in Deep Neural Networks. They found that the $\eta/b$ ratio directly alters the short-range correlation (the Hurst parameter) of the noise, showing that memory effects also bias the optimizer toward flat minima.
  • Decentralized Optimization: This phenomenon is not restricted to centralized training. In Heavy-Tail Phenomenon in Decentralized SGD, Hu et al. derived explicit tail-index formulas for distributed systems, revealing that network topologies interact with local step-sizes and batch-sizes to generate even heavier tails than centralized SGD.

By viewing hyperparameters not merely as optimization speed dials, but as structural controls for the tail-index of multiplicative noise, this framework provides a unified, intuitive explanation for why specific learning rate and batch size configurations unlock superior generalization.

Key Foundational Work

The Heavy-Tail Phenomenon in SGD (2021) — contributed 3 concept elements to this direction.
Multiplicative Noise and Heavy Tails in Stochastic Optimization (2021) — contributed 2 concept elements to this direction.
Stochastic Gradient and Langevin Processes (2020) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 11 papers
  2. Version 1 · 2026-10-04 · 17 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_learning_rate_to_batch_size_ratio_9451055c,
  title = {learning rate to batch size ratio},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-rate-to-batch-size-ratio/rdr_9451055ce5a4459a9bdef02b20fc4b53},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_learning_rate_to_batch_size_ratio_9451055c_v2,
  title = {learning rate to batch size ratio},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-rate-to-batch-size-ratio/rdr_9451055ce5a4459a9bdef02b20fc4b53/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake