Abstract
Stochastic Gradient Descent (SGD) is a cornerstone of deep learning, yet the mechanisms behind its superior generalization capabilities remain partially understood. This research identifies multiplicative noise—noise that scales with the current parameter values—as a fundamental driver of "heavy-tailed" behavior in model weights. By analyzing SGD as a discrete random recurrence relation, the authors prove that multiplicative noise leads to power-law stationary distributions. These heavy tails facilitate "basin hopping," allowing the optimizer to escape sharp local minima and explore the loss landscape more effectively. This framework provides a theoretical explanation for why smaller batch sizes and larger learning rates often lead to better-performing models.
Introduction
A persistent mystery in deep learning is the "generalization gap," where increased stochasticity (e.g., smaller batch sizes) often results in models that generalize better to unseen data. Traditional analyses often simplify SGD noise as "additive," typically modeled as a random Gaussian jitter. However, such models predict "light-tailed" distributions (like a Bell curve) that fail to capture the "jumpy" trajectories observed in real-world training. This paper argues that the intrinsic variance in local convergence rates across different data samples manifests as multiplicative noise, which is essential for understanding SGD's dynamics.
Theoretical Framework: Modeling Stochastic Optimizers
The authors model stochastic optimizers as discrete random recurrence relations (Markov chains) of the form $W_{k+1} = \Psi(W_k, X_k)$. This approach avoids the simplifications of continuous-time approximations, such as Langevin dynamics, which can obscure the true nature of the noise.
The research identifies three primary regimes for tail behavior based on the interaction between the multiplicative and additive components of the update rule:
- Light-tailed: Characterized by bounded updates or rapidly decaying noise, leading to exponential tail decay.
- Heavy-tailed (multiplicative): Arises when the multiplicative component satisfies specific conditions, leading to power-law tails even with light-tailed additive noise.
- Heavy-tailed (additive): Driven by extrinsic factors, such as heavy-tailed data (Lèvy noise).
Key Insights: The Role of Multiplicative Noise
Multiplicative noise is intrinsic to the optimization process, representing how the convergence rate fluctuates depending on the sampled data point. Unlike additive noise, which "blurs" the loss landscape, multiplicative noise allows for large jumps to escape basins while maintaining high resolution near the bottom of a minimum.
The "heaviness" of the tails, quantified by the tail exponent $\alpha$, is directly influenced by training hyperparameters:
- Batch Size: Smaller batch sizes increase stochasticity, leading to heavier tails (lower $\alpha$) and better exploration.
- Step Size: Larger step sizes contribute to heavier tails.
- Regularization: Increased $L^2$ regularization tends to pull the distribution toward lighter tails.
Empirical Validation
The theory was validated across multiple scales, from simple 1D non-convex functions to complex architectures. Key findings include:
- Basin Hopping: PCA-projected trajectories showed that SGD frequently jumps between basins, a behavior that Gradient Descent with only additive noise failed to replicate.
- Parameter Sensitivity: In experiments using the Wine Quality dataset, decreasing batch size from 10 to 1 shifted the tail exponent $\hat{\alpha}$ from ~5.99 to ~2.97, confirming that lower batch sizes induce heavier tails.
- Real-world Evidence: Training MobileNetV2 on CIFAR-10 revealed strong evidence of heavy-tailed behavior in training steps, with a measured tail exponent of $\hat{\alpha} \approx 1.52$.
Conclusion and Future Work
This work provides a rigorous mathematical foundation for the observation that SGD's noise is not merely a nuisance but a critical feature for exploration. Future research aims to extend this theory to more complex optimizers like Adam and to explicitly link the tail exponent $\alpha$ to generalization error. Additionally, bridging the gap between temporal weight distributions and the spectral distributions of weight matrices remains a significant open challenge.
Summary updated