Abstract
What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention. In this paper, we study the distribution of the Stochastic Gradient Noise (SGN) vectors during the training. We observe that for batch sizes 256 and above, the distribution is best described as Gaussian at-least in the early phases of training. This holds across data-sets, architectures, and other choices.
Introduction
Stochastic Gradient Descent (SGD) is the engine behind modern deep learning. Unlike standard Gradient Descent, which utilizes the entire dataset to calculate weight updates, SGD uses a small "minibatch." This introduces "noise"—the difference between the true gradient and the minibatch estimate, known as Stochastic Gradient Noise (SGN).
For years, researchers have modeled this noise as Gaussian, invoking the Central Limit Theorem (CLT). This assumption allows them to treat SGD as a physical process like Langevin diffusion, which helps explain why SGD generalizes so well. However, recent high-profile research challenged this, arguing that SGN is actually "heavy-tailed" (following an $\alpha$-stable distribution), implying the noise is much more "jumpy" than previously thought. This paper seeks to settle the debate: Is the noise Gaussian, heavy-tailed, or something else entirely?
Methodology
The authors employ a rigorous statistical approach to test for "Gaussianity" in high-dimensional space. Since testing a high-dimensional vector for normality is difficult, they use a key property: if a random vector is multivariate Gaussian, then every one-dimensional projection of that vector must be a univariate Gaussian.
The experimental pipeline involves training various neural networks (ResNet18, VGG16, AlexNet, LeNet) on CIFAR-10 and MNIST. Every 100 iterations, 1,000 independent minibatch samples are taken to calculate SGN vectors, which are then projected onto 1,000 random unit directions. Gold-standard statistical tests—Shapiro-Wilk and Anderson-Darling—are applied to these projections.
To ensure reliability, the tests were first validated on synthetic "stable" (heavy-tailed) data. As shown in the validation plots, the tests correctly reject the Gaussian hypothesis for all $\alpha < 2$, only accepting it as the distribution approaches a true Gaussian.
Key Results
The study reveals that the distribution of SGN is highly dependent on batch size and training progress:
- Batch Size is the Deciding Factor: For batch sizes of 256 and 4096, the noise is indeed Gaussian-like at the start of training. However, for small batch sizes (e.g., 32), the noise is never Gaussian, with statistical tests confidently rejecting the hypothesis from the first iteration.
- Gaussianity Fades Over Time: Even with medium-to-large batches, the noise often starts Gaussian but becomes non-Gaussian as the model converges.
As illustrated in the Gaussianity testing results, the fraction of accepted projections (red line) drops significantly as training iterations increase. This suggests that the geometry of the loss landscape changes in a way that violates the "independent and identically distributed" (i.i.d.) requirements of the CLT as training nears a local minimum.
Discussion and Conclusion
The authors argue that previous research claiming SGN is heavy-tailed relied on flawed estimators that assumed coordinate independence—an assumption that rarely holds in complex neural networks. While this paper refutes the idea that SGN is always heavy-tailed, it highlights that the Gaussian assumption is also fragile, particularly in the later stages of training or with small minibatches.
Future theoretical work should move away from simple Gaussian Stochastic Differential Equation (SDE) models and account for the emerging non-Gaussianity as models converge. Characterizing these non-Gaussian distributions remains an open and vital question for understanding deep learning generalization.
Summary updated