TL;DR
This paper provides a theoretical analysis of the distributional properties of adaptive gradient updates, challenging the common belief that their variance diverges at initialization. The authors demonstrate that under a normal gradient assumption, the update variance is actually bounded and increases with time, suggesting that the need for learning rate warm-up in optimizers like Adam might stem from other factors.
What Problem Does This Paper Address?
Training deep neural networks has become increasingly reliant on adaptive gradient methods like Adam, RMSProp, and AdaGrad. While these methods have proven highly effective, a deep theoretical understanding of their behavior, particularly their statistical properties, remains elusive. A key open question is why certain optimizers, especially Adam, require a "warm-up" period where the learning rate is gradually increased from a small value. One prominent hypothesis, proposed by Liu et al. (2019), suggests this warm-up is necessary due to a divergent variance of the adaptive gradient update at initialization. This paper aims to rigorously investigate this hypothesis and the underlying distributional properties of adaptive gradient updates.
Methods
The core of this work involves a theoretical analysis of the distribution of the adaptive gradient update, denoted as $\Delta\theta$. The authors make a crucial assumption that the underlying gradient $g_t$ follows a normal distribution. They then derive the exact distribution of $\Delta\theta$ under this assumption, considering the exponential averaging used in adaptive methods. A key methodological contribution is the proof that the variance of the adaptive gradient update is always finite (Proposition 1), directly contradicting the divergence hypothesis. They also derive the exact distribution of $\Delta\theta$ for time-independent, isotropic Gaussian gradients and a specific form of the preconditioner. The paper also explores the impact of exponential averaging and provides analytical derivations for various special cases and limiting behaviors.
Datasets
The paper uses the CIFAR-10 dataset for training and evaluating various neural network architectures, including RegNetX-200MF, VGG, ResNet-18, ShuffleNet-V2, ResNeXt-29, MobileNet, EfficientNet-B0, and DenseNet-121. For transformer models, they use the IWSLT14 Ger-Eng dataset.
Key Insights and Ideas
- Bounded Variance, Not Divergent: The most significant finding is that the variance of the adaptive gradient update, under a normal gradient assumption, is bounded and actually increases with time, rather than diverging at initialization. This directly challenges a leading explanation for the necessity of learning rate warm-up.
- Warm-up Mystery Persists: The paper suggests that the divergence of variance is not the cause of warm-up, leaving the exact reason for this phenomenon an open question.
- Transition from Bi-modal to Uni-modal: The theoretical analysis predicts, and experiments confirm, a transition in the update distribution from bi-modal to uni-modal as the time step increases. This transition occurs precisely at $m=2$ for a specific case.
- Gaussian Convergence: The update distribution converges to a Gaussian distribution as the number of time steps increases.
- Correlation Matters: The authors highlight that neglecting the correlation between the numerator ($g_t$) and denominator ($\sqrt{n_t/c_n}$) in the adaptive gradient update can lead to incorrect predictions, such as overestimating variance.
- Transformer Pathology: The paper notes that while many architectures show good agreement with their theory, transformer architectures exhibit a more pathological update distribution, with multiple prominent peaks, suggesting a more complex underlying gradient behavior.
Key Results
- Proposition 1: The variance of the adaptive gradient update is always finite, regardless of the gradient distribution.
- Theoretical Distribution: The paper derives the exact probability density function (PDF) for the update $\Delta\theta$ under specific assumptions (Gaussian gradient, time-independent, isotropic, specific preconditioner).
- Experimental Agreement: The derived theoretical distributions show excellent visual agreement with experimental histograms across various neural network architectures and datasets.
- Variance Trend: Experiments confirm that the variance of the update is an increasing function of time, aligning with theoretical predictions.
- Bi-modality Trigger: The bi-modal structure of the update distribution is shown to be related to the increase in gradient variance, not its divergence.
Limitations and Future Work
- Gaussian Gradient Assumption: The theoretical analysis relies on the assumption that the underlying gradient is a zero-mean Gaussian with constant variance. The paper acknowledges that real-world gradients may be skewed or exhibit correlations between parameters, which could lead to deviations from the predicted distributions.
- Transformer Behavior: The pathological update distribution observed in transformers suggests that the Gaussian assumption might not hold for all architectures or tasks. Future work could investigate the causes of these multiple peaks and the heavy-tailed nature of gradients in such cases.
- Warm-up Cause: The paper explicitly states that identifying the correct cause of Adam's warm-up requirement remains an open problem.
- Correlation of Gradients: While the theory assumes uncorrelated gradients, the paper notes that this is unlikely in practice for non-linear systems. Understanding the suppression of off-diagonal covariance in gradients is suggested as a crucial future research topic.
- Approximate Bayesian Inference: The authors suggest that knowing the posterior distribution of optimization iterations has applications in approximate Bayesian inference for model parameters.
Summary updated