Adaptive gradient optimizers like Adam and RMSProp are the undisputed workhorses of modern deep learning, enabling the convergence of architectures ranging from deep Transformers to computer vision models. However, these optimizers suffer from a notorious "cold-start" problem: they are highly volatile during the initial steps of training. Without empirical workarounds like learning rate warmup schedules, models frequently experience massive loss spikes or catastrophic divergence before training even gets underway.
To understand why adaptive optimizers fail early in training, we must look at how they construct their step sizes and how a lack of historical data warps the optimization landscape.
The Cold-Start Problem and Sign Descent
The core power of an adaptive optimizer lies in its ability to scale updates for each parameter inversely proportional to its historical gradient magnitude. Consider a simplified update rule for a parameter vector $\theta$:
$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$ $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$ $$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t} + \epsilon} m_t$$
Here, $v_t$ tracks the running second moment (the uncentered variance) of the gradients. At the very first step of training ($t=1$), the optimizer has no history, meaning $v_0$ is initialized to zero.
As Abuduweili et al. demonstrate, initializing second-order moments to zero causes the very first update step to collapse into sign-descent. Because $v_1$ is purely a function of $g_1^2$, the gradient magnitude terms in the numerator ($m_1$) and the denominator ($\sqrt{v_1}$) cancel out. The update step effectively becomes:
$$\theta_{1} \approx \theta_0 - \eta \cdot \text{sign}(g_1)$$
In this state, the optimizer completely ignores the actual magnitude of the gradients. Whether a parameter's gradient is massive or microscopic, it is updated by a fixed step size $\eta$. As Abuduweili et al. note, this behavior can immediately push the model into highly suboptimal, flat regions of the parameter space from which it may never recover. This lack of initial gradient statistics creates a severe cold-start problem (Prazdnichnykh et al.), manifesting as extreme numerical volatility and loss spikes (Chu et al.).
The Debate: Variance vs. Update Magnitude
Why exactly does this uninitialized state cause models to crash? The optimization community offers two primary explanations for this early training fragility.
The Variance Perspective
Jiang et al. argue that the instability stems from the extreme variance of the adaptive learning rate term. When $t$ is small, $v_t$ is an estimate of gradient variance built on only a handful of samples. The authors analytically model this early-stage second-moment estimate using a scaled inverse chi-square distribution. They show that because the degrees of freedom are low at the start of training, the variance of $\frac{1}{\sqrt{v_t}}$ is mathematically divergent or extremely high. This high variance leads to erratic, unpredictable weight updates that disrupt the learning process before a stable descent regime can be reached (Huang et al.).
The Magnitude Perspective
Conversely, Ma et al. argue that the issue is not the variance of the estimator, but rather the absolute magnitude of the updates. They show that even if a model is initialized near a local minimum where gradients are small, the lack of historical data causes the denominator $\sqrt{v_t}$ to scale updates up excessively. This results in massive, unstable steps during the first few iterations until the second-moment estimator stabilizes, leading to catastrophic divergence if the learning rate is not tightly controlled.
Adding nuance to this debate, Zhang et al. derived the exact probability distribution of adaptive updates, proving that the variance is technically finite at initialization and actually increases over time. This suggests that the early-stage vulnerability may stem from non-Gaussian distributional properties of the updates rather than simple variance divergence.
The Long Memory of Early Mistakes
The danger of early training instability is not just that a few steps are bad, but that adaptive optimizers remember their early mistakes for a long time.
Because $v_t$ is an exponential moving average (typically with a hyperparameter $\beta_2 = 0.999$), the gradients encountered in the first few hundred steps remain baked into the denominator for thousands of iterations. As Kalra et al. explain, if the model experiences massive gradient magnitudes during an initial unstable phase, these large values permanently inflate $v_t$. This memory effect results in an excessively small effective learning rate ($\frac{\eta}{\sqrt{v_t}}$) later in training, effectively trapping the model in high-loss regions.
[Step 0: v_0 = 0] ──> [Sign-Descent Step] ──> [Massive Gradient Spike]
│
▼
[Trapped in High Loss] <── [Permanently Small LR] <── [v_t Inflated Long-Term]
This phenomenon is particularly destructive in specialized training regimes:
- Fine-Tuning: Kumar et al. discovered that when fine-tuning pretrained vision models, massive gradient spikes occur exclusively in the initial embedding layers. These spikes cause the adaptive optimizer to update the first layer too rapidly, obliterating valuable pretrained features before the rest of the network can adapt.
- Dynamic Sparsity: In memory-efficient training where network connections are dynamically activated, Xiao et al. note that newly activated weights experience localized cold-start spikes. Because these new connections lack historical gradient data, the optimizer applies disproportionately large updates to them, triggering sudden loss spikes.
Destabilizing Interactions: Regularization and Sparsity
The fragile initial state of an adaptive optimizer can be easily worsened by standard training techniques, such as gradient regularization.
Gradient regularization techniques are designed to guide models toward flat minima by penalizing large gradients. However, Zhao et al. show that this creates a severe conflict with adaptive optimizers early in training. By biasing gradient norms toward zero before the optimizer has accumulated accurate moment statistics, regularization causes the denominator $\sqrt{v_t}$ to become excessively small. This triggers an explosion in the effective learning rate, leading to extreme variance and poor convergence.
A similar issue occurs in distributed environments like federated learning. As Lin observes, when client data distributions vary significantly, the variance estimate in the denominator can fluctuate wildly or become excessively small, causing sudden learning rate explosions and training divergence.
Taming the Instability
To prevent adaptive optimizers from crashing during their initial steps, researchers have developed several distinct strategies to stabilize the early training phase.
1. Learning Rate Warmup and Rectification
The traditional fix is a manual warmup schedule, which gradually ramps up the learning rate $\eta$ over the first few thousand steps. This keeps the update magnitudes small while the optimizer collects enough gradient samples to build a reliable second-moment estimate (Jiang et al.; Ma et al.).
To eliminate the need to manually tune this warmup period, Jiang et al. introduced Rectified Adam (RAdam). RAdam mathematically predicts the variance of the adaptive learning rate at each step using a scaled inverse chi-square framework. It then applies a dynamic rectification factor to dampen updates when the variance is dangerously high, providing the stability of warmup automatically.
2. Non-Zero Moment Initialization
Rather than trying to suppress the symptoms of the cold start, Abuduweili et al. propose fixing the root mathematical cause. By initializing the second-order moments ($v_0$) to a small non-zero state—either through random noise or a quick data-driven pass—the optimizer avoids the sign-descent collapse entirely. The initial updates preserve gradient magnitudes, stabilizing training and eliminating the need for complex warmup schedules.
3. Architectural Initialization (T-Fixup)
Instead of altering the optimizer, some approaches fix early training instability by modifying the model's initialization. Huang et al. introduced T-Fixup, a theoretical weight initialization scheme for Transformers. By carefully scaling the initial weights based on network depth, T-Fixup ensures that the variance of the blocks remains bounded. This prevents the early-layer gradient explosions that typically crash adaptive optimizers, enabling the stable training of ultra-deep models without requiring warmup or layer normalization.
Key Foundational Work
On the Variance of the Adaptive Learning Rate and Beyond (2020) — contributed 4 concept elements to this direction.
Improving Transformer Optimization Through Better Initialization (2020) — contributed 1 concept elements to this direction.
Nostalgic Adam: Weighting More of the Past Gradients When Designing the Adaptive Learning Rate (2019) — contributed 1 concept elements to this direction.