Initial Instability in Adaptive Optimization

This direction is split into other directions: adaptive learning rate variance. Its last version stays readable and citable here.

Version 1 of 193 papers120 conceptsfrozen 2026-10-04

Adaptive optimizers like Adam and RMSprop are the default choice for training deep neural networks because they automatically scale learning rates for each parameter. However, these methods suffer from a "cold-start" problem: they are notoriously unstable during the first few hundred iterations of training. To prevent models from diverging, practitioners almost universally rely on a "warmup" heuristic—starting with a near-zero learning rate and gradually ramping it up.

Recent research has moved beyond treating warmup as a black-box trick, identifying the mathematical mechanisms that cause this early-stage volatility and developing more robust alternatives.

The Variance of Limited History

The core of the instability lies in how adaptive optimizers estimate the "scale" of gradients. Algorithms like Adam maintain a moving average of squared gradients ($v_t$) to normalize updates. In the initial steps of training, this estimate is based on very few samples.

Liyuan Liu, Haoming Jiang, and colleagues in On the Variance of the Adaptive Learning Rate and Beyond demonstrated that this limited history leads to extreme variance in the adaptive learning rate. When the second-moment estimate $v_t$ is calculated from only a handful of steps, it can be undesirably small, causing the effective learning rate to explode. This variance can be modeled as a scaled inverse chi-square distribution; until enough samples are collected to stabilize this distribution, the optimizer is prone to making erratic updates that trap the model in suboptimal regions of the loss landscape.

The Sign Descent Trap

A second, more structural issue arises from how these optimizers are initialized. Standard practice involves initializing the second-order moments to zero. Abulikemu Abuduweili identifies in Revisiting the Initial Steps in Adaptive Gradient Descent Optimization that this causes the first update step to collapse into "sign descent."

Because the denominator in the update rule is nearly zero, the magnitude of the gradient is effectively canceled out. The optimizer only "sees" the direction (the sign) of the gradient, and the step size is determined entirely by the global learning rate. This behavior ignores the actual curvature of the loss landscape, potentially pushing the parameters into "flat" or "sharp" regions that are difficult to recover from. Abuduweili suggests that simply using a non-zero initialization or a data-driven "warmup pass" can eliminate the need for manual learning rate schedules.

Memory Poisoning and Loss Spikes

The instability isn't just a temporary hurdle; it can "poison" the rest of the training run. Dayal Singh Kalra and Maissam Barkeshli argue in Why Warmup the Learning Rate? Underlying Mechanisms and Improvements that adaptive optimizers incorporate these early, massive gradients into their long-term moving averages. This "memory effect" can result in an excessively small effective learning rate later in training, effectively stalling the model.

Furthermore, Zhiwei Bai and others have shown that "loss spikes"—sudden jumps in error—occur when the second-moment estimator fails to track rapidly increasing gradients. In Adaptive Preconditioners Trigger Loss Spikes in Adam, they describe a "decoupling" where the optimizer's internal math fails to react to local changes in the loss landscape, allowing the effective learning rate to surge past the stability threshold.

Automated Solutions and Curriculums

To solve these issues without manual tuning, researchers have proposed several architectural and algorithmic fixes:

By understanding that initial instability is a product of high variance and poor initialization rather than just "noisy gradients," the field is moving toward optimizers that are robust by design, reducing the need for the fragile hyperparameter tuning that has long characterized deep learning.

Papers in this version

Showing 60 of 93 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_initial_instability_in_adaptive_optimiza_66bab55b,
  title = {Initial Instability in Adaptive Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initial-instability-in-adaptive-optimization/rdr_66bab55b778b49e2bacca48dcdc00a86},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_initial_instability_in_adaptive_optimiza_66bab55b_v1,
  title = {Initial Instability in Adaptive Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initial-instability-in-adaptive-optimization/rdr_66bab55b778b49e2bacca48dcdc00a86/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}