Two-Phase and Budget-Aware Learning Rate Schedules

43 concepts32 papers1 proposal

In stochastic optimization, choosing the step size (or learning rate) involves a fundamental tension between exploration and precision. Early in training, a large learning rate is essential: it allows the algorithm to make rapid progress across the parameter space and escape poor local minima or initialization biases. However, stochastic gradients are inherently noisy. If the learning rate remains high, the parameters will eventually bounce around aimlessly within a "noise-limited neighborhood" of the optimum, unable to settle.

To resolve this, practitioners and theorists use learning rate schedules that decrease the step size over time. While early optimization theory favored continuous, any-time decay formulas (such as $1/t$ or $1/\sqrt{t}$), modern research demonstrates that the most effective way to navigate the speed-versus-precision trade-off is through structured, multi-phase schedules. These approaches broadly fall into two categories: two-phase switching schedules that drop the learning rate after reaching a noise floor, and budget-aware schedules that continuously cool the learning rate to zero based on a fixed training horizon.

Two-Phase Solutions and Switching Rules

The intuition behind a two-phase schedule is straightforward: maximize progress early on by maintaining a constant learning rate, and then switch to a decaying learning rate once the model's performance plateaus. The initial constant phase exploits the fast convergence rates of fixed-step methods, rapidly pushing the parameters into the neighborhood of the solution. The subsequent decaying phase suppresses gradient variance, allowing the model to overcome the noise floor and converge precisely to the exact optimum.

This "Search-Then-Converge" paradigm was formalized early on by Christian Darken et al., who introduced a diagnostic metric called "drift" to automatically detect when a trajectory transitions from broad exploration to local convergence. Rather than relying on manual heuristics, modern optimization theory provides explicit, closed-form rules to determine the optimal switching point. For example, Xun Qian et al. derived switching rules based on the expected smoothness and noise characteristics of the objective function, maximizing overall optimization efficiency.

This hybrid strategy has proven robust across a wide variety of optimization landscapes:

  • Stochastic Games and Adversarial Dynamics: In min-max optimization and smooth games (such as training GANs), Nicolas Loizou et al. showed that a constant-then-decreasing step-size schedule allows solvers to achieve exact convergence where constant steps would cycle indefinitely.
  • Variational Inequalities: For structured non-monotone problems, both Konstantinos Emmanouilidis et al. and Sayantan Choudhury et al. demonstrated that single-call extragradient methods achieve high-precision refinement by switching from constant to diminishing step sizes.
  • Constrained and Distributed Systems: In distributed networks with imperfect information, Angelia Nedic et al. utilized constant steplengths within discrete regimes to exploit fast initial convergence. For complex or nonsmooth constraints, Hideaki Iiduka and Nitesh Kumar Singh et al. independently showed that switching regimes successfully balances initial movement toward constraint sets with final global convergence.

Budget-Aware Scheduling and Last-Iterate Optimality

While switching rules are ideal when training can run indefinitely, real-world deep learning typically operates under a strict, pre-defined iteration budget. Furthermore, a long-standing gap existed between optimization theory and practice: classical theory often optimized for the average of all parameter states across training, whereas practitioners care exclusively about the performance of the final model state (the last iterate).

Prateek Jain et al. bridged this gap by proving that if an algorithm incorporates the total iteration budget directly into its step-size sequence, the final iterate of Stochastic Gradient Descent (SGD) can achieve information-theoretically optimal convergence rates without requiring any averaging.

When the training horizon is fixed, the most effective schedule is a Linear Decay to Zero (LD2Z) or a Warmup-Stable-Decay (WSD) schedule. Instead of waiting for an empirical trigger to drop the learning rate, the schedule continuously scales the step size based on the remaining budget, ensuring it hits zero precisely at the final iteration.

Mengtian Li et al. demonstrated that this budget-aware linear decay consistently outperforms complex adaptive optimizers and standard step-decay schedules, particularly in resource-constrained scenarios. Because the decay is calibrated to the entire duration of the budget, it prevents the model from stopping prematurely while the learning rate is still high. Eric Arazo et al. further confirmed that budget-calibrated linear decay manages the exploration-convergence trade-off far more effectively than algorithmic data-selection techniques like importance sampling.

Why It Works: The Mechanics of the Cooldown Phase

The empirical success of budget-aware linear decay—especially in large-scale foundation models—can be explained by the mathematics of its final "cooldown" phase.

Classical any-time schedules (like inverse-square-root decay) suffer from residual logarithmic error terms because they never decrease the step size quickly enough relative to the accumulated gradient noise. By contrast, Fabian Schaipp et al. proved that a dedicated terminal cooldown phase mathematically eliminates these logarithmic error terms. This explains a phenomenon frequently observed in large language model pretraining: when analyzing frameworks like OLMo, Dirk Groeneveld et al. documented a sharp, dramatic spike in model accuracy precisely during the final linear decay steps. During this phase, the model is rapidly suppressing accumulated stochastic noise and settling cleanly into the bottom of the local minimum.

Learning Rate
  ^
  |   /-----------\
  |  /             \
  | /               \
  |/                 \
  +--------------------> Time / Iterations
  Warmup   Stable    Cooldown (Noise Suppression)

Theoretically, Soham Bonnerjee et al. showed that linear decay to zero provides a hybrid benefit: it maintains a high step size long enough to match the rapid initial convergence of constant schedules (overcoming initialization bias), while guaranteeing long-term stability as the horizon approaches. This is supported by the work of Binghui Li et al., who derived optimal schedules by numerically minimizing functional scaling equations. Their framework explicitly balances the trade-off between rapid signal learning during the stable phase and final noise suppression during the terminal decay.

Boundary Conditions and Historical Roots

The foundational principles of balancing time-based decay with information accumulation trace back to early learning theory. N. Barkai et al. and Todd K. Leen et al. established that schedules decaying inversely with time ($1/t$) optimize generalization error scaling by balancing step sizes against accumulated data. When dealing with finite training sets, Peter Sollich et al. demonstrated that the optimal learning rate decay is fundamentally constrained by dataset size; if a training set can be fitted perfectly, higher learning rates can be sustained longer without inflating the final generalization error.

However, when noise or label imperfections are present, the timing of the transition becomes critical. Hui Kang et al. revealed that in noisy-label environments, decaying the learning rate too early results in severe underfitting, whereas decaying it too late allows the model to memorize incorrect labels.

To handle these sensitivities safely, modern schedules either bridge constant updates and time-based decay by continuously tracking the system's noise limits (Nicholas F. Marshall et al.), coordinate the optimization step size jointly with secondary system parameters (Aswin Kannan et al.), or utilize budget-aware refinement techniques (Aaron Defazio et al.) to tailor the decay trajectory to the specific task at hand.

Key Foundational Work

On-line Learning of Dichotomies (1994) — contributed 1 concept elements to this direction.
Towards Faster Stochastic Gradient Search (1991) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_learning_rate_decay_36cabde1,
  title = {linear learning rate decay},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-learning-rate-decay/rdr_36cabde19ff34a59b5edc72f75980bf2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_learning_rate_decay_36cabde1_v1,
  title = {linear learning rate decay},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-learning-rate-decay/rdr_36cabde19ff34a59b5edc72f75980bf2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake