Schedule-Free Optimization for Flexible Training Horizons

This direction is split into other directions: training horizon dependency. Its last version stays readable and citable here.

Version 1 of 139 papers42 conceptsfrozen 2026-10-04

In modern deep learning, the "cosine decay" schedule has become a rigid dogma. To achieve peak performance, researchers must commit to a fixed compute budget—the total number of training steps or tokens—before the first gradient is even calculated. If you suddenly acquire more data or your compute allocation changes mid-run, you face a lose-lose choice: stop early with a suboptimal model or restart from scratch to fit a new schedule.

A growing body of research is moving toward schedule-free and budget-agnostic optimization. By decoupling the training process from a predetermined end date, these methods allow for "anytime" training, where models can be interrupted, extended, or merged without sacrificing convergence.

The Cosine Trap and the Need for Flexibility

Standard learning rate (LR) schedules, such as cosine decay or linear decay-to-zero, are globally dependent. As noted by David L. Macêdo and colleagues in their work on TASO, these schedulers require a predetermined total step count to function. This dependency creates a "temporal alignment" problem: if the training budget changes, the entire schedule is invalidated (Ho et al., 2019).

This rigidity is particularly problematic for Large Language Models (LLMs). DeepSeek-AI researchers have pointed out that fixed schedules prevent efficient incremental updates; you cannot easily add new data to a model that has already decayed its learning rate to near-zero (DeepSeek LLM). Even "learning-rate-free" optimizers often fail this test, as they still rely on a pre-defined training horizon to calculate their internal decay logic (Kasimbeg et al., 2025).

The River Valley Intuition

To understand why we can move away from fixed schedules, we must look at the loss landscape. Kaiyue Wen, Zhiyuan Li, and their collaborators propose the River Valley perspective (Understanding Warmup-Stable-Decay Learning Rates).

Imagine the loss landscape as a long, flat river valley. A high, constant learning rate allows the model to travel rapidly down the "river" (the low-loss path), even if it bounces turbulently between the valley walls. This turbulence makes the instantaneous loss look high, but the model is making significant progress along the valley floor. The "decay" or "cooldown" phase is simply the act of slowing down so the model can settle into the deepest part of the riverbed.

This intuition suggests a more flexible framework: Warmup-Stable-Decay (WSD). Instead of a continuous curve, the model maintains a high, stable learning rate for as long as needed, followed by a short, non-negotiable cooldown phase when the user decides to finish training (MiniCPM).

Mechanisms for Budget-Agnostic Training

Several distinct mechanisms have emerged to implement this flexible training style:

1. The Cooldown Phase

Research by Alexander Hägele and Atli Kosson demonstrates that a constant learning rate followed by a short cooldown (e.g., 10-20% of the total steps) achieves performance identical to cosine decay (Scaling Laws and Compute-Optimal Training). This makes training runs reusable: you can train at a constant rate indefinitely and only trigger the cooldown when you are satisfied with the progress. Fabian Schaipp and Francis Bach recently provided a theoretical foundation for this, proving that the cooldown mathematically eliminates the logarithmic error terms that plague constant-step-size optimization (The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling).

2. Checkpoint Merging (WSM)

A more radical approach, Warmup-Stable and Merge (WSM), eliminates the decay phase entirely. Instead of lowering the learning rate, researchers like Changxin Tian and Jiaxin Mao propose training at a constant rate and then mathematically simulating decay by averaging multiple checkpoints from the stable phase (WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging). This allows for fully autonomous, continuous training where the "decayed" model can be produced at any time without affecting the ongoing training process.

3. Schedule-Free (SF) Optimizers

Rather than managing the schedule externally, Schedule-Free methods build adaptivity into the optimizer itself. Minhak Song and Chulhee Yun show that SF optimization implicitly performs weight averaging during training, allowing the model to stay on the "river floor" without an explicit decay step (Through the River). Similarly, Stab-SGD by David A. R. Robin uses a "stability ratio"—a local measure of gradient noise—to automatically shrink step sizes, achieving schedule-free adaptivity without manual tuning.

4. Power-Law Scheduling

For those seeking a "set-it-and-forget-it" approach, the Power Scheduler discovered by Yikang Shen and Shawn Tan leverages a power-law relationship between optimal learning rate, batch size, and token count. This scheduler is agnostic to the total number of tokens, enabling seamless early stopping or training extension (Power Scheduler).

Why It Works: The Power of Averaging

The recurring theme across these papers is that averaging is a proxy for decay. Whether it is the explicit model merging of WSM, the implicit averaging in Schedule-Free optimizers, or the "cooldown" phase that allows a model to settle, the goal is the same: to move from the turbulent, high-LR exploration of the loss landscape to the stable, low-LR convergence point.

By moving away from schedules that require "impossible" prior knowledge of the training duration (Jiang et al., 2024), the field is shifting toward a more robust, "anytime" paradigm. This not only slashes the compute cost of hyperparameter tuning and scaling law research but also brings us closer to truly autonomous learning systems that can adapt to data and resources on the fly.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_schedule_free_optimization_for_flexible__c16a1c00,
  title = {Schedule-Free Optimization for Flexible Training Horizons},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/schedule-free-optimization-for-flexible-training-horizons/rdr_c16a1c00f031459789227a532b6e306f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_schedule_free_optimization_for_flexible__c16a1c00_v1,
  title = {Schedule-Free Optimization for Flexible Training Horizons},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/schedule-free-optimization-for-flexible-training-horizons/rdr_c16a1c00f031459789227a532b6e306f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}