Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features

Noé Colomban, Francis Bach, Fabian SchaipparXiv 2026 · arXiv

TL;DR

This paper provides the first exact asymptotic analysis of the "Warmup-Stable-Decay" (WSD) learning-rate schedule for the last iterate of SGD in least-squares regression. The authors prove that adding even a brief "cooldown" period (decaying the learning rate at the end of training) provides a polynomial improvement in risk over constant schedules, with pure linear decay emerging as the theoretically optimal choice.

What Problem Does This Paper Address?

In modern deep learning, practitioners rarely use the "iterate averaging" techniques favored by classical optimization theory. Instead, they rely on the final iterate of a model trained with specific learning-rate schedules. The Warmup-Stable-Decay (WSD) schedule—where the learning rate stays constant for most of training before a final "cooldown" decay—has become a standard in large-scale model training (e.g., Llama).

However, theoretical understanding of why these schedules work for the final iterate has lagged. Most existing bounds are "worst-case" and don't account for the specific difficulty of a problem, such as the noise level or the data's covariance spectrum. This paper seeks to quantify exactly how much a decay phase "buys" you in terms of reduced risk, specifically within the context of streaming least-squares regression.

Methods

To make the complex dynamics of SGD tractable, the authors employ several key techniques:

  • Slock’s Model: Instead of assuming standard Gaussian features, which couple all dimensions together, the authors use a discrete model where each data sample activates only a single eigendirection. This decouples the SGD dynamics, allowing for an exact coordinate-wise analysis of the risk.
  • Power-Law Assumptions: The authors assume the problem follows "neural scaling laws"—specifically, that the data covariance spectrum ($\lambda_i \sim i^{-\alpha}$) and the initial parameter error ($m_{0,i} \sim i^{-\beta}$) decay according to power laws.
  • Asymptotic Analysis: Using $z$-transforms and Tauberian theorems, they derive "asymptotic equivalents"—simplified formulas that describe how the bias and variance of the model behave as the training horizon $T$ goes to infinity.
  • Schedule Comparison: They compare the Constant schedule against the WSD family, which is parameterized by a relative cooldown length $c$ (where $c=1$ is pure linear decay and $c \to 0$ approaches a constant rate).

Datasets

As a theoretical work, the paper does not use standard real-world datasets. Instead, it validates its mathematical derivations using synthetic simulations based on Slock’s model. These simulations (typically with $d=100$ dimensions and varying power-law exponents $\alpha$ and $\beta$) demonstrate near-perfect agreement with the derived theoretical risk formulas.

Figure 1: Comparison of the empirical SGD excess risk against the theoretical final risk from Corollary F.4. Both experiments are averaged over n = 50 independent runs with parameters γ = 0.1, L = ∆= 1, α = 1.5, β = 2, σ = 0.1 and d = 100. For the linear schedule (b), the plot displays the entire risk trajectories over time, where each trajectory is computed with a schedule decaying to zero at its specific horizon T.

Key Insights and Ideas

  • The Cooldown Advantage: The most striking finding is that any cooldown period of fixed relative length $c > 0$, no matter how short, improves the risk by a polynomial factor compared to a constant schedule. The convergence rate exponent $q$ is strictly better for WSD than for the constant schedule.
  • Linear Decay is Optimal: Within the WSD family, the asymptotic risk is minimized when $c=1$. This suggests that a pure linear decay schedule is theoretically superior to schedules that maintain a long "stable" phase, provided the base learning rate is tuned correctly.
  • Learning-Rate Transfer: The authors show that the optimal base learning rate for a $20\%$ cooldown can be directly translated to the optimal rate for a $100\%$ cooldown (linear decay) via a constant ratio that is independent of the training horizon $T$. This explains why practitioners can often "reuse" tuned learning rates across different schedule lengths.
  • Critical Batch Size: The optimal risk depends on the batch size $b$ and horizon $T$ only through their product $bT$ (the total sample budget). This holds up to a "critical batch size," beyond which increasing parallelism no longer yields benefits without a statistical penalty.

Key Results

The authors provide a comprehensive summary of the risk equivalents for different regimes of spectral decay ($\alpha$).

Table 1: Equivalents of the bias and the variance for the studied schedules (constant: both regimes 1 < α < 2 and α > 2), with ω = (α + β −1)/α. The cooldown length is c ∈(0, 1]. Constants depending only on (α, β) are omitted. W(α, c) is defined in Eq. (16).
  • Risk Scaling: For a horizon $T$, the optimal risk scales as $\Theta(T^{-q})$. The exponent $q$ for WSD is $\frac{\alpha+\beta-1}{\alpha+\beta}$ (for $1 < \alpha < 2$), which is significantly faster than the constant schedule's rate.
  • Cooldown vs. Averaging: In certain regimes, the last iterate of SGD with a cooldown recovers the same optimal convergence rate as averaged SGD, which is known to be unimprovable.

Limitations and Future Work

  • Quadratic Objectives: The analysis is restricted to least-squares regression (smooth, convex, quadratic). While this is a standard starting point, it does not capture the non-convex complexities of deep neural networks.
  • Slock's Model vs. Reality: While the authors show Slock's model matches Gaussian features in the vanishing learning-rate limit, it remains a simplified abstraction of real-world data distributions.
  • Linear Cooldowns Only: The study focuses on linear decay. The authors note that for very fast spectral decay ($\alpha > 2$), polynomial cooldowns might be required to achieve truly optimal rates, marking a clear path for future research.
  • Asymptotic Horizon: The results require a very large $T$ to be accurate. In practice, for small $T$, finite-time effects can dominate, potentially making the "stable" phase of WSD more useful than the theory suggests.

Summary updated

Abstract

We study the last iterate of stochastic gradient descent (SGD) on streaming least squares with discretely distributed features (Slock's model), which decouples the dynamics across eigendirections. Under power-law source and capacity conditions, we derive the exact finite-time risk and asymptotic equivalents of constant and Warmup-Stable-Decay (WSD) schedules. With a tuned base learning-rate, the resulting scaling law decays as $Θ(T^{-q})$, where $T$ is the horizon, and $q$ is independent of the cooldown length $c \in (0,1]$ but is strictly smaller for the constant schedule: any cooldown of fixed relative length $c > 0$, however short, improves the risk by a polynomial factor, while its length only affects the constant, which is minimized by linear decay ($c = 1$). Finally, we show that the optimal risk depends on the batch size $b$ and horizon $T$ only through $bT$, up to a critical batch size.
BibTeX
@misc{colomban2026asymptotic,
  author = {Noé Colomban and Francis Bach and Fabian Schaipp},
  title = {{Asymptotic Risk of Learning-Rate Schedules for Least-Squares Regression with Discretely Distributed Features}},
  year = {2026},
  eprint = {2610.16518},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.16518},
}
Figure