In modern deep learning, training a model is often a commitment to a rigid timeline. Standard optimization recipes rely heavily on learning rate schedules—most notably cosine decay—that require practitioners to specify the exact total number of training steps ($T$) before the first gradient is even calculated.
This dependency creates a severe operational bottleneck. If a training run is interrupted early, the model performs poorly because it never underwent the crucial final decay phase. Conversely, if a team acquires more data or compute budget mid-run, they cannot easily extend the training horizon; doing so requires either restarting from scratch with a new schedule or accepting highly suboptimal convergence.
To break free from this constraint, a rapidly growing body of research is developing horizon-independent optimization methods. These techniques allow models to be trained indefinitely, stopped early, or extended on the fly, all while maintaining or exceeding the performance of heavily tuned, fixed-horizon schedules.
The Geometry of the Problem: Why Horizons Matter
To understand why traditional schedules are so rigid, we must look at how the learning rate interacts with the loss landscape.
During the early and middle stages of training, a high learning rate is essential. It provides the stochastic noise necessary to escape poor local minima and allows the optimizer to make rapid progress across flat regions of the loss landscape. However, this high energy causes the model's parameters to oscillate violently across the walls of the local minima, resulting in a high training loss. To settle into the true bottom of a well and achieve peak performance, the learning rate must be systematically reduced (or "decayed").
As described by Kaiyue Wen et al., this dynamic can be visualized as navigating a river valley. A high learning rate drives rapid progress along the length of the flat "river floor" despite high visible oscillations. The decay phase stops these oscillations, allowing the model to drop straight down to the true floor.
Traditional Cosine Decay:
Learning Rate: \___...___
Progress: Locked to a predetermined total step count T.
Horizon-Independent (e.g., Warmup-Stable-Decay):
Learning Rate: /---------\___
Progress: Can extend the "Stable" phase indefinitely;
trigger "Decay" (cooldown) whenever ready.
When using a fixed schedule like cosine decay, the rate of this cooling is tied directly to the total budget $T$. If training is terminated prematurely, the model is caught mid-oscillation, high up on the valley walls. As Rong Ge et al. demonstrate, standard polynomial decay schedules ($1/t$) that attempt to decay continuously without a fixed horizon result in highly suboptimal final iterates for least-squares problems. This sensitivity creates a fundamental incompatibility with modern hyperparameter optimization techniques that rely on early stopping to prune underperforming models.
Paradigm 1: Warmup-Stable-Decay and Cooldowns
The most direct way to decouple training from a fixed horizon is to replace continuous decay with a multi-stage schedule consisting of a warmup, a long stable (constant) learning rate phase, and a short, aggressive "cooldown" or decay phase.
Learning Rate
^
| /-----------\
| / \
| / \____
+------------------------> Steps
Warmup Stable Cooldown
This approach, known as Warmup-Stable-Decay (WSD), allows practitioners to extend the stable phase indefinitely based on data or compute availability. The critical insight, validated empirically by Alexander Hägele et al., is that a short cooldown phase applied at any arbitrary point yields performance identical to a cosine schedule tuned specifically for that total duration. This simple shift makes training runs reusable and slashes the compute cost of discovering scaling laws by over 50%.
The Theoretical Justification
Why does a sudden cooldown work just as well as a gradual cosine decay? Fabian Schaipp et al. bridged the gap between practice and classical convex optimization theory by proving that a dedicated cooldown phase mathematically eliminates the logarithmic error terms ($\log t$) that typically plague the last-iterate convergence of stochastic gradient methods.
Further analyzing this mechanism, Aleksandr Dremov et al. studied the precise training dynamics during this final stage, confirming that the model rapidly compresses its variance and locks into high-generalization states. For open-ended or streaming data environments where an endpoint is fundamentally unknown, Vaibhav Singh et al. proposed an Infinite Cosine Schedule that transitions into a stable, constant learning rate rather than decaying to zero, completely mitigating the catastrophic forgetting that occurs when traditional schedules are prematurely restarted for continual pre-training.
Paradigm 2: Schedule-Free Optimization via Weight Averaging
While WSD schedules offer flexibility, they still require the user to manually trigger a cooldown phase to achieve peak performance. A more radical alternative is Schedule-Free (SF) optimization, which eliminates the decay phase entirely.
Instead of lowering the learning rate to stop the model from oscillating across the river valley, schedule-free methods use a continuous mathematical shortcut: iterate averaging. By maintaining a running average of the weights visited during training, the optimizer naturally filters out the high-frequency oscillations up and down the valley walls, revealing the true floor of the landscape at any given moment.
Optimizer Path: /\/\/\/\/\/\/\/\ (Oscillates wildly at high LR)
Iterate Average: ---------------- (Smoothly tracks the true minimum)
Aaron Defazio et al. pioneered this approach by unifying iterate averaging directly with the optimizer's momentum equations. This design ensures that the model always performs at its theoretical peak without needing to know the training duration in advance.
The underlying mechanics are further illuminated by the authors of Through the River, who show that schedule-free methods implicitly keep the model on the high-performance floor of the loss landscape throughout the entire training run without incurring extra memory costs. To make this approach robust to large batch sizes and momentum sensitivity, David A. R. Robin et al. introduced Stab-SGD, which uses a local "stability ratio" to automatically scale the step size based on gradient noise, achieving true schedule-free adaptivity.
Paradigm 3: Post-Hoc Checkpoint Merging and Power Laws
Other researchers have approached the fixed-horizon problem by rethinking how we extract the final model or how we scale the learning rate relative to non-temporal metrics.
Checkpoint Merging (WSM)
Rather than running a physical cooldown phase or maintaining a running average during training, Changxin Tian et al. introduced the Warmup-Stable and Merge (WSM) framework. WSM completely eliminates the learning rate decay phase during optimization. Instead, it trains the model indefinitely at a stable learning rate and mathematically simulates the effect of a decay phase post-hoc by merging multiple checkpoints collected across the stable trajectory. This allows researchers to instantly generate a fully converged model at any point in time, facilitating seamless data integration and budget extensions.
Dimension-Agnostic Scheduling
Taking a different angle, Yikang Shen et al. developed the Power Scheduler. By identifying a strict power-law relationship between the optimal learning rate, the batch size, and the total number of tokens processed, the authors created a scheduler that automatically scales based on data throughput rather than step counts. This allows for seamless early stopping and continual training without sacrificing performance.
The Broader Push for Horizon-Free Algorithms
The operational friction caused by fixed horizons is not unique to deep learning; it is a long-standing challenge across optimization theory.
- Convex and Non-Convex Optimization: Algorithms like GDA and SC-PDA developed by Wei Tao et al. focus on removing logarithmic factors from last-iterate convergence without requiring prior knowledge of total iterations. Similarly, adaptive variance-reduction methods like Ada-STORM (Wei Jiang et al.) and ADASPIDER (Ali Kavis et al.) achieve optimal theoretical complexity without requiring predefined target accuracies or smoothness parameters.
- Shuffling Gradient Methods: For data-shuffling algorithms (like Random Reshuffle), Zijian Liu et al. provided the first last-iterate convergence proofs for time-varying step sizes that do not require prior knowledge of the total epoch count.
- Data Augmentation: The rigidity of fixed horizons also extends beyond learning rates. For instance, Daniel Ho et al. noted that learned data augmentation schedules are frequently bound to specific training durations, meaning any change in the compute budget invalidates the schedule and forces a costly re-optimization process.
Despite these advances, creating a truly automated, "anytime" optimizer remains an open challenge. As Priya Kasimbeg et al. point out, even modern "learning-rate-free" optimizers often still secretly rely on a predefined training horizon to calibrate their weight decay or internal step-size bounds. Furthermore, scaling automated scheduling methods (such as the Graph-Network-based Scheduler proposed by Yuanhao Xiong et al.) from moderate fine-tuning tasks to massive, multi-week pre-training runs remains an active area of development.
Conclusion
The shift away from fixed-horizon scheduling represents a maturation of deep learning engineering. By decoupling optimization performance from a predetermined step count—whether through constant-plus-cooldown schedules, schedule-free weight averaging, or post-hoc checkpoint merging—the field is moving toward a more flexible, efficient, and truly continuous paradigm of model training.
Key Foundational Work
The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares (2019) — contributed 1 concept elements to this direction.
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (2019) — contributed 1 concept elements to this direction.
Gradient Descent Averaging and Primal-dual Averaging for Strongly Convex Optimization (2021) — contributed 1 concept elements to this direction.