Many forecasting models are trained not to be accurate but to be cheap to act on. A load forecaster whose errors trigger emergency power purchases, or an engine-life predictor whose overestimates risk an in-flight failure, is trained on an asymmetric loss: a penalty that charges more for erring in one direction than the other. Often that loss is not written by hand. It is fitted from a simulator. Researchers perturb the true labels, run a dispatch or scheduling simulation on each perturbed forecast, and fit a smooth surrogate to the resulting costs. That surrogate is only trustworthy near the labels it was fitted around. Early in training, though, the model's predictions sit far from those labels, in territory where the surrogate is unreliable. Early training is also the period when the learning rate is largest.
This note proposes that the regret of a cost-trained forecaster depends on how much of its learning-rate budget is spent while its predictions lie inside the region where the simulator-fitted loss is accurate. A standard decaying schedule can burn most of its large steps optimizing a misspecified loss. By the time predictions enter the trusted region, the step size is too small to move the model to the cost-optimal bias. We will measure this quantity, test whether it predicts downstream cost better than conventional schedule descriptors, and test a simple fix. The fix holds the learning rate high until most predictions are inside the trusted region, then decays. The asymmetric loss is the object of study, premature learning-rate decay is the mechanism, and simulator-based evaluation is both the source of the problem and the lens for judging fixes.
Core Claim
Three bodies of work each touch one side of this question, and none joins them. Decision-focused surrogates such as the locally optimized decision losses of Decision-Focused Learning without Decision-Making fit convex, direction-aware losses (their "DirectedQuadratic" family) to oracle regret evaluated on Gaussian perturbations around ground truth. That paper notes that early predictions fall outside this neighborhood but does not study optimizer dynamics. Cost-Oriented Load Forecasting and the ProEnFo package in Benchmarks and Custom Package for Electrical Load Forecasting build piecewise-linear dispatch-cost losses from simulated IEEE 30-bus dispatch, smooth them with a Huber-style norm, and train with out-of-the-box Adam. The premature-convergence literature shows that schedules matter, but only for symmetric losses. Darken and Moody's kinetic-energy view in Note on Learning Rate Schedules for Stochastic Optimization covers this, as does the minima-manifold analysis of Understanding the Generalization Benefits of Late Learning Rate Decay. So does the finding in How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining that the learning rate acts as an implicit importance weight, so that late, high-quality data is ignored under decay-to-zero. The research file records no paper studying learning-rate decay under asymmetric losses.
The last of these supplies our central analogy. In curriculum pretraining, decay wastes the best data. Here, decay wastes the period when the loss is most accurate. We make the analogy measurable. Let $c(t)$ be the fraction of training predictions at step $t$ whose residual falls inside the surrogate's validated region. That region is the set of residuals where held-out oracle cost and surrogate cost agree within a tolerance. Let $\eta_t$ be the learning rate and $T$ the total number of steps. We define the trusted-budget fraction
$$B = \frac{\sum_{t=1}^{T} \eta_t\, c(t)}{\sum_{t=1}^{T} \eta_t},$$
which is the share of total step size spent while the loss is trustworthy. The claim is falsifiable. Across schedules, optimizers, and surrogate envelope widths, final oracle-measured regret should track $B$ more closely than it tracks the decay floor, the decay start time, or the total step-size budget. Furthermore, a schedule that raises $B$ should lower regret only when the surrogate is misspecified away from the labels, and not when the exact cost is used.
A second, subsidiary mechanism concerns adaptive optimizers. Steep-side errors produce large gradients early, and these inflate the optimizer's running estimate of squared gradient size. When residuals later cross to the flat side of the loss, the effective step shrinks. This is a directional relative of the gradient-accumulation trap that Why ADAGRAD Fails for Online Topic Modeling documented for symmetry-breaking gradients, and that Follow the Moving Leader in Deep Learning attributed to unbounded historical summation. We treat it as a competing explanation to be separated from the trusted-budget effect, not as the headline.
Research Program
The work has three parts. The first is instrumentation. For each task we fit a surrogate from simulator queries and estimate its validated region by querying the oracle on held-out perturbations at increasing distances from the labels. During training we then log $c(t)$, the trusted-budget fraction $B$, and Darken–Moody drift from Towards Faster Stochastic Gradient Search, which tells us whether the weights are still moving purposefully. We also log the mean signed residual, since the cost-optimal forecaster carries a deliberate bias and its final value is the quantity most likely to freeze prematurely.
The second part is a correlational sweep. It covers cosine, step, and linear decay to zero, moderate decay to one third of peak (the floor suggested by the curriculum-pretraining paper), and late decay. Each schedule is crossed with SGD with momentum, Adam, AdaGrad, and the leaky-accumulator AMH optimizer of AMH: AdaGrad-Momentum Hybrid for Robust Training on Small Coresets. The sweep also varies the perturbation envelope width used to fit the surrogate, which shifts when $c(t)$ rises without touching the schedule. Varying the envelope is our main causal lever.
The third part is the intervention, a coverage-gated schedule. It holds peak learning rate until $c(t)$ exceeds a threshold $\tau$ and then applies cosine decay to a nonzero floor. Its only new hyperparameter is $\tau$, and $\tau$ is set using simulated validation data alone. The real-trace evaluation described below checks whether that choice transfers. The evaluation lens follows Synthetic Data as Validation, which showed that simulator-generated validation can select models more reliably than small real splits. We ask the same question of schedule selection. We also report the gap between surrogate-measured and oracle-measured regret, because a method that only lowers the former has learned to exploit the surrogate.
Baselines
The naive reference is the practice in Cost-Oriented Load Forecasting: the fitted asymmetric surrogate trained with Adam under a standard cosine-to-zero schedule. Every other method must beat it on oracle-measured regret.
The strongest schedule competitor is moderate decay with checkpoint averaging. This is the Curriculum Model Averaging recipe from How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining: decay to one third of peak, or hold a constant high rate, and average late checkpoints. It is strongest because it already defends against premature decay without any knowledge of the surrogate. Its reported gain, a 1.64% average improvement for moderate decay plus averaging, comes from language-model pretraining and is not comparable to our tasks. We will tune its floor and averaging window with the same search budget given to $\tau$. Coverage gating beats it only if it lowers oracle regret at matched total steps and matched tuning trials.
The strongest optimizer competitor is AMH. Its leaky accumulator forgets the large early gradients that the subsidiary mechanism blames. We run it with its published decay constant of 0.995 and the same learning-rate search as the others.
The strongest task-level reference on turbofan prognostics is the ACB model in Effective Latent Representation for Prediction of Remaining Useful Life. That paper reports a Score of 1055.32 and an RMSE of 15.62 on FD004. Because our point is a schedule effect rather than a new architecture, we reimplement its encoder and its penalized-RMSE training. Our comparison is ACB with its own schedule against ACB with coverage gating, with the published number as a sanity anchor for the reimplementation. For dispatch-cost load forecasting, no reported result was found on Lacuna for dispatch cost under learned schedules. That baseline is therefore entirely our own reproduction.
Two ablations isolate the key ingredient. The first gates on a fixed step count matched to when coverage was reached in a pilot run, which tests whether the coverage signal matters or only the delay. The second runs coverage gating with the exact oracle cost substituted for the surrogate, on problems small enough for the oracle to be affordable. Under exact cost, gating should give no advantage over equally delayed decay.
Why not just…?
Why not just widen the perturbation envelope so the surrogate covers early predictions? This is a real threat, and it is one of our sweep axes. Widening the envelope trades local accuracy for coverage, because a convex surrogate fitted over a wide region blurs the asymmetric kink that encodes the cost. If wide envelopes match gated schedules on oracle regret, the trusted-budget account is still confirmed, since $B$ rises in both cases. In that case the practical recommendation becomes "fit wider," and we will report it that way.
Why not just warm-start with squared error and switch to the asymmetric loss later? Warm-starting brings predictions near the labels before the cost-sensitive phase, so it also raises coverage. What it does not do is restore step size. Under a single decaying schedule, the asymmetric phase begins at a reduced learning rate. We include MSE-then-surrogate under both cosine and coverage-gated schedules. If warm-starting with a restarted high learning rate performs as well as gating, the two are the same idea, and we will say so.
Why not just tune the decay floor by grid search? Grid search over the floor finds a good schedule for one task and one envelope width. Our sweep tests whether that optimum moves when only the envelope width changes. If it moves in the direction $B$ predicts, a fixed tuned floor is fragile and the coverage signal carries information the grid lacks. If the tuned floor is stable across envelopes, the trusted-budget idea adds little, and the experiment will show that.
Why not just use AMH or another forgetting optimizer? If the damage comes from gradient-magnitude accumulation, the leaky accumulator should remove it regardless of schedule. The cross of optimizer with schedule separates the two mechanisms. A gating benefit that persists under SGD with momentum, which has no accumulator, cannot be explained by accumulation.
Proposed Experiments
We use three tasks. The first is dispatch-cost load forecasting with ProEnFo's piecewise dispatch penalties on the simulated 30-bus grid, using the true dispatch simulation as the oracle. The second is C-MAPSS FD001–FD004, with penalized RMSE as the training surrogate and the official asymmetric Score as the oracle. The third is neural runtime prediction on ATLAS (ATLAS: Alibaba Dataset and Benchmark for Learning-Augmented Scheduling), with a LODL-style directed surrogate fitted to scheduler regret and Max-Stretch and completion-time ratio as the oracle metrics. ATLAS reports a completion-time ratio of 1.106 for SPJF and a Spearman correlation of 0.912 for its LightGBM predictor, and we use these to check that our neural predictor is competitive. ATLAS also supplies the synthetic-to-real test. It shows that simple Pareto job-arrival simulations differ from production traces, and that Max-Stretch is highly sensitive to under-predicting large jobs. We will select $\tau$ and all schedule hyperparameters on simulated Pareto arrivals and then evaluate on the real Alibaba trace.
The decisive checks are fourfold. First, rank correlation between $B$ and oracle regret across the full sweep should exceed that of decay floor, decay start time, and total step-size budget, on held-out task seeds. Second, changing only the envelope width should move both $B$ and regret in the predicted direction while the schedule is held fixed. Third, gating should lose its advantage when the exact oracle replaces the surrogate. If gating still helps there, the effect is the generic late-decay benefit described by Ren et al. and not a surrogate effect. Fourth, the schedule ranking chosen on simulated arrivals should survive on the real trace. Failure there would mean that simulator-based schedule selection is itself an unreliable form of synthetic evaluation, and that result is worth reporting on its own. We run five seeds per configuration, report intervals, and test each prediction against the matched-budget baselines above.
References
- Decision-Focused Learning without Decision-Making: Learning Locally Optimized Decision Losses
- Cost-oriented load forecasting
- Benchmarks and Custom Package for Electrical Load Forecasting
- Note on Learning Rate Schedules for Stochastic Optimization
- Understanding the Generalization Benefits of Late Learning Rate Decay
- How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
- Why ADAGRAD Fails for Online Topic Modeling
- Follow the Moving Leader in Deep Learning
- Towards Faster Stochastic Gradient Search
- AMH: AdaGrad-Momentum Hybrid for Robust Training on Small Coresets
- Synthetic Data as Validation
- Effective Latent Representation for Prediction of Remaining Useful Life
- ATLAS: Alibaba Dataset and Benchmark for Learning-Augmented Scheduling