Abstract
Adaptive gradient methods have achieved remarkable success in training deep neural networks on a wide variety of tasks. However, not much is known about the mathematical and statistical properties of these methods. This paper debunks the popular theory that the Adam optimizer requires a "warmup" phase due to divergent (infinite) variance at the start of training. By deriving the exact probability distribution of adaptive gradient updates, the authors prove that variance is always finite and actually increases over time, suggesting that the true cause for the necessity of warmup remains an open mystery.
Introduction
Adaptive gradient methods like Adam and RMSProp are the workhorses of modern deep learning, especially for large-scale models like Transformers. However, these optimizers are notoriously unstable at the very beginning of training, often requiring a "warmup" period where the learning rate is manually kept very low for the first few thousand steps.
A prominent previous theory (Liu et al., 2019) argued that this instability happens because the "preconditioner" (the moving average of squared gradients) has a divergent variance when estimated from only a few initial samples. This paper challenges that mathematical assumption, providing a rigorous statistical foundation for the Adam update $\Delta \theta$.
Theoretical Framework
The authors approach the problem by treating the gradient as a random variable. Assuming the underlying gradient follows a normal distribution, they derive the exact probability density function (PDF) for the update $\Delta \theta$.
Key theoretical insights include:
- Beta Distribution Mapping: The update follows a transformed Beta distribution.
- Correlation and Stability: The numerator (gradient) and denominator (root mean square of gradients) are not independent; they are calculated from the same samples. This correlation prevents the variance from exploding, keeping it finite even at initialization.
- Sub-gaussianity: The update is proven to be a sub-gaussian variable, meaning it is always bounded for any finite time step.
Empirical Validation and Results
The theory was validated using several benchmarks: CIFAR-10 (RegNetX-200MF), IWSLT14 (Transformer), and MNIST.
Distributional Transitions
The derived formula shows excellent agreement with empirical data. A notable prediction is the transition of the update distribution from bimodal (for $m < 3$) to unimodal as training progresses and more samples are averaged.
Finite Variance
The authors proved that the variance of the update $\Delta \theta$ is exactly 1 (when normalized) for all time steps, directly contradicting the "divergent variance" hypothesis. Empirical results across MobileNet, ShuffleNet, RegNet, and Transformers confirm that variance does not explode at initialization. Contrary to the idea that variance is huge at the start and settles down, the variance of the update magnitude $|\Delta \theta|$ is actually at its minimum at initialization and grows as more samples are collected.
Discussion
While CNNs follow the Gaussian theory almost perfectly, Transformers exhibit "pathological" distributions with multiple prominent peaks. This suggests that Transformer gradients are highly non-Gaussian, which might be the real reason they are harder to optimize and require warmup.
Since this paper proves that divergent variance is not the reason for warmup, it leaves the community with a significant unanswered question. The authors suggest that investigating the non-Gaussian, multi-peak nature of Transformer gradients is the most promising path forward to understanding optimization stability.
Conclusion
This work provides a solid statistical model for adaptive updates, proving that the "divergent variance" theory is mathematically flawed due to the neglect of correlation between the gradient and its preconditioner. While the theory holds well for CNNs, the unique behavior of Transformers highlights the need for more complex distributional assumptions in future research.
Summary updated