Benchmarks that compare training algorithms, such as AlgoPerf, rank methods by how fast they reach a target score. That score is read off whatever weights each method happens to evaluate. Some methods evaluate a running average of their past weights, while others evaluate the latest, noisy iterate. Weight averaging alone is known to shorten time-to-target by about 12%. Schedule-Free AdamW, which won a track, evaluates an averaged sequence by construction. It is therefore possible that part of what these leaderboards measure is simply "who averaged." Separately, a time-to-target score is set by the first noisy evaluation that crosses a threshold, so it is exposed to the same checkpoint lottery that recent LLM-pretraining papers document.
We propose to make weight averaging an equalized, mandatory part of the evaluation protocol rather than an optional part of the method. Every submission would be scored on an exponential moving average of its own weights, kept at a fixed horizon set before any results are seen. We would then measure two things. The first is how much published speedups and rankings change under this protocol. The second is how much it shrinks seed-to-seed variance, and therefore the number of seeds a benchmark needs to rank methods with confidence. That second effect speaks directly to the cost problem: AlgoPerf consumed nearly 50,000 V100 GPU-hours.
Core Claim
The claim has two parts, and both can be falsified. The first part concerns confounding. Methods differ in how noisy their late iterates are, so they gain unequally from averaging. A benchmark that lets some methods average and others not therefore mixes up "better optimizer" with "smoother evaluation point." Speedups recomputed with averaging applied to every method should differ measurably from the published ones, and the largest corrections should go to methods that do not already average internally.
The second part concerns cost. Averaging reduces how much a method's score varies across random seeds. To keep the uncertainty on a ranking at a fixed level, the number of seeds needed grows with the square of that seed-to-seed spread (the variance of a mean falls as one over the number of seeds). Even modest variance reduction therefore buys a disproportionate saving in benchmark compute. This saving is distinct from the 12% optimizer speedup reported by When, Where and Why to Average Weights?. We are not trying to make any one method faster. We are trying to make the comparison between methods cheaper and less biased.
Relation to Prior Work
Two recent papers motivate the variance claim but stop short of method comparison. Instability in Downstream Task Performance During LLM Pretraining shows that averaging 20 checkpoints smooths downstream scores of a single pretraining run. On the 13B model, standard deviation on Human Examination falls from 0.021 to 0.015. MaP couples checkpoint merging with Pass@k and cuts how often pretraining rankings flip after fine-tuning from 50% to 22.73%. Both papers average within one model's trajectory to read its learning curve. Neither asks whether averaging changes conclusions when different training algorithms are compared. Neither applies the idea to time-to-target scoring, where a noisy first crossing of the threshold decides the result.
When, Where and Why to Average Weights? supplies the method and the default horizon. It embeds averaging into AlgoPerf and finds that a window of about 1% of the training budget works best. It treats averaging as an add-on that speeds up a baseline. It does not ask whether the benchmark's verdicts change when every competitor gets the same add-on, and it does not study seed variance. The Road Less Scheduled is the method most implicated by our confound hypothesis. Its averaged evaluation sequence may account for part of its 8% self-tuning win, and we test that directly rather than assume it.
The benchmarking literature constrains how we design the study. Accelerating neural network training shows that hidden software defaults bias AlgoPerf results. How much progress have we made in neural network training? shows that uncontrolled tuning budgets manufacture progress. Both warn that our protocol must not hand any method an extra tuned knob. That is why the averaging horizon is fixed globally and chosen on held-out workloads, not tuned per method. Quantifying Variance in Evaluation Benchmarks shows that seed variance alone produces 1–2% score swings and that discrete metrics are far noisier than continuous ones. It also motivates one of our main controls, because it implies averaging might be redundant once a continuous metric is used.
Research Program
We would build a thin evaluation harness that wraps any optimizer submission. During training, the harness keeps exponential moving averages of the weights at four horizons: 0.5%, 1%, 2% and 5% of the step budget. They live in host memory, so they do not change the optimizer's own state. At every scheduled evaluation, the harness scores the raw iterate and each average on both the benchmark's target metric and validation loss.
The official averaged score uses a single horizon. That horizon is chosen once, on workloads excluded from the comparison, and then frozen. Our prior from Ajroldi et al. is about 1%. Methods that already average internally, such as Schedule-Free, receive the same external average on top of their evaluation sequence, so no method is exempt. Averaging normalization statistics, where present, is handled by recomputing them on a fixed small training subset before each averaged evaluation. That recomputation is identical across methods.
The analysis has three parts. The first decomposes each method's averaging gain into a shift in mean and a reduction in seed variance, workload by workload. The second recomputes speedups relative to the baseline, and the full ranking, under raw and averaged scoring, using bootstrap resampling over seeds. The third estimates, for each scoring rule, the number of seeds needed before the ranking of every pair of methods is stable in 95% of bootstrap resamples.
Baselines
The naive reference point is the current protocol. Each method is scored on whatever weights it natively evaluates, with time-to-target read from the first raw evaluation crossing the threshold. This is how AlgoPerf reports results. We would reproduce it using the same seeds and hardware as every other condition. Beating it means showing that averaged scoring changes at least one published speedup by more than its seed-bootstrap confidence interval, and that it lowers the seed count needed for a stable ranking.
The strongest existing competitor is the result we are re-scoring. Externally tuned Distributed Shampoo reported "an average 28% speedup over the NadamW baseline." Self-tuning Schedule-Free AdamW reported "an 8% speedup over the self-tuning baseline" (Accelerating neural network training). Each method is rerun with its published hyperparameters and no additional tuning, so averaging is the only change. The question is whether these margins survive equalization. Our hypothesis predicts the Shampoo margin persists, since both it and NadamW gain from averaging. It predicts the Schedule-Free margin shrinks, since its baseline gains more.
The strongest variance-reduction competitor is the uniform rolling-window checkpoint average used by Instability in Downstream Task Performance During LLM Pretraining and MaP. We run it with the same horizon expressed as a fraction of the budget. Neither paper reports a result on AlgoPerf workloads; no reported result found on Lacuna for checkpoint averaging applied to optimizer-comparison rankings. The comparison with exponential averaging is therefore new evidence rather than a target to beat.
The key ablation is a checkpoint prediction ensemble over the same window. It averages outputs rather than weights. This isolates whether the stabilization comes from smoothing the evaluation point in weight space or merely from pooling several noisy readings. If the ensemble matches the weight average on ranking stability, the protocol's value is "evaluate on a window," and averaging is just the cheap way to do that.
Why not just…?
Why not just run more seeds? More seeds reduce ranking uncertainty but do not remove a confound. If averaging shifts methods' means unequally, more seeds simply estimate the biased gap more precisely. More seeds also do nothing about the cost problem we target. Our seed-count analysis prices the comparison directly, as raw scoring needing a certain number of seeds against averaged scoring needing fewer for the same pairwise stability. If raw and averaged seed counts come out equal, the variance half of our claim fails.
Why not just score validation loss instead of accuracy-type targets? This is a real threat. Quantifying Variance in Evaluation Benchmarks reports that continuous metrics have signal-to-noise ratios "often 10 to 100 times higher" than discrete ones. MaP argues that parameter oscillation is a separate noise source that metric changes do not remove. We settle the question by running the full analysis under both validation loss and the official targets. If averaging gives no variance reduction on validation loss, we report that switching metrics suffices for variance. The confound question survives regardless, since averaging also shifts the mean.
Why not just require every method to decay its learning rate to zero? A fully cooled iterate may already be nearly as smooth as an average, so the averaging gain could vanish at the end of training. Ajroldi et al. find that averaging combined with decay still helps. The threat is sharper for time-to-target, however, because thresholds are often crossed mid-schedule, before cooling. We measure averaging gains as a function of where in the schedule the target is crossed. If gains concentrate there, the protocol matters most for exactly the anytime scoring benchmarks use.
Proposed Experiments
Compute rules out the full suite at many seeds. The primary study therefore uses four AlgoPerf workloads: OGBG molecular graphs, FastMRI reconstruction, WMT translation, and ImageNet ResNet-50. Each runs ten seeds for NadamW, Distributed Shampoo and Schedule-Free AdamW, with LibriSpeech Conformer held out to select the horizon.
A cheaper replication runs Xiong et al.'s CIFAR-10, CIFAR-100 and RoBERTa-on-MRPC tasks with SGD, Adam, LAMB and Lookahead at thirty seeds. These tasks are cheap enough to estimate seed-count curves tightly. Their protocol showed that "Adam achieved ~85% accuracy, while SGD lagged at ~71%" on MRPC, so we can test whether a large gap like that is robust while small gaps reorder.
As an out-of-domain check, PointNet++ and SimpleView are retrained on ModelNet40 under the Goyal et al. protocol. That protocol reports 93.3% versus 93.0%, a gap smaller than typical seed variance and therefore a natural candidate for reversal.
Three outcomes would be decisive. The confound claim holds if the recomputed Schedule-Free margin over its baseline moves by more than its bootstrap interval while the Shampoo–NadamW margin does not. The cost claim holds if averaged scoring reaches 95% pairwise ranking stability with materially fewer seeds than raw scoring, on both validation loss and the official targets. The weight-space claim holds if the prediction-ensemble ablation falls short of the weight average on stability at equal window. Failing all three would show that current optimizer leaderboards are already robust to averaging, which is also worth knowing before more benchmarks consider adopting it.
References
- When, Where and Why to Average Weights?
- Instability in Downstream Task Performance During LLM Pretraining
- MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics
- The Road Less Scheduled
- Accelerating neural network training: An analysis of the AlgoPerf competition
- How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
- Quantifying Variance in Evaluation Benchmarks