When many machines train one language model over slow network links, a common approach is to let each machine take hundreds of local optimizer steps and only occasionally send back the net change it made. The asynchronous version goes further: machines report whenever they finish, without waiting for each other. The cost is that an update from a slow machine arrives late. It was computed from an old copy of the model, and the shared model has since moved on. Existing fixes use one of three rules. They penalize a late update by how many steps it missed, or by how far the shared weights have moved, or they rely on hand-built buffering of the server's momentum. None of them measures how useful a late update actually still is.
This proposal borrows a tool from student-modeling research. In that field, how well a learner retains a skill is estimated from data as a forgetting curve: how recall falls as a function of time since last practice, with a separate decay rate for each skill. We propose to fit the same kind of curve for late updates. The target quantity is how well an update that arrives $\Delta$ server steps late still agrees with where training is currently heading. The curve would be estimated online, separately for each block of parameters, and each late update would be weighted by its predicted retention. The claim to test is that a pooled, per-block retention curve with a tolerance plateau beats three things. It should beat distance-based penalties, which the file reports can hurt when delays are small. It should beat heuristic momentum buffering. It should also beat the naive option of simply measuring agreement on each arriving update.
Core Claim
Several terms need defining first. In DiLoCo-style training (DiLoCo), each worker runs $H$ local AdamW steps. It then sends a pseudo-gradient, meaning the difference between the weights it started from and the weights it ended with. A server combines pseudo-gradients with an outer Nesterov momentum optimizer. Asynchronous Local-SGD Training for Language Modeling shows that the naive asynchronous version fails because late pseudo-gradients disrupt the outer momentum. That paper repairs this with two heuristics. Delayed Nesterov buffers pseudo-gradients and applies momentum only intermittently. Dynamic Local Updates gives slow workers fewer local steps. It provides no theory for either. Gap-Aware Mitigation of Gradient Staleness takes a different view in the per-step asynchronous setting. It argues that how far the parameters have drifted matters more than elapsed steps, and divides each late gradient by that normalized drift. It also reports that at low worker counts this penalty performs slightly worse than doing nothing, because networks tolerate small amounts of staleness. The paper leaves the shape of that tolerance undefined.
We treat a late update's usefulness as an empirical quantity. For parameter block $b$, define its retention at delay $\Delta$ as
$$ r_b(\Delta) = \mathbb{E}\left[\cos\left(u_b^{\text{late}},\; \bar{u}_b^{\text{fresh}}\right) \mid \Delta\right], $$
where the terms are as follows. $u_b^{\text{late}}$ is the block-$b$ slice of a pseudo-gradient that arrives $\Delta$ outer steps after its worker pulled the model. $\bar{u}_b^{\text{fresh}}$ is the average of the block-$b$ slices of the next few pseudo-gradients that arrive with near-zero delay. The expectation pools over many arrivals.
We fit $r_b$ with a forgetting-curve form that has a plateau:
$$ \hat r_b(\Delta) = \rho_b \exp\left(-\max(0, \Delta - \kappa_b)/S_b\right). $$
Here $\rho_b$ is baseline agreement, which is below 1 because workers see different data. $\kappa_b$ is a tolerance plateau during which late updates lose nothing. $S_b$ is a memory-strength constant, in the same role as $S$ in the Ebbinghaus decay used by KVFKT. The server then scales each arriving block by $\hat r_b(\Delta)/\rho_b$ before it enters the outer momentum.
The plateau parameter $\kappa_b$ is the specific answer to Gap-Aware's unexplained "staleness noise." The per-block parameters are the analogue of the student-modeling finding that forgetting rates differ by skill. KVFKT, for example, reports population forgetting cycles of about 55 hours on ASSISTments and about 110 hours on EdNet. We expect the same kind of heterogeneity across embeddings, attention layers and MLP layers.
Research Program
The program has three parts.
The first part is measurement, and it is useful even if the weighting scheme fails. On a 150M-parameter model trained on C4 with the OpenDiLoCo framework, we simulate workers of heterogeneous speed. We then log every arrival's delay in server steps, its wall-clock delay, its Gap-Aware parameter drift, and its per-block cosine agreement with fresh updates.
This settles a question the literature argues without data: does step delay, wall-clock time, or geometric drift best predict obsolescence? We fit each as a single-covariate retention model and compare held-out prediction error. If drift dominates and shows no plateau, that confirms Gap-Aware's view and undercuts the need for a temporal curve. We would report this outcome directly.
The second part is the estimator. The curve parameters are updated online by weighted least squares on a sliding window of logged arrivals. We shrink toward a shared global curve, which is how student-modeling systems handle sparse per-skill data. The estimator's cost is a few dot products per block per arrival. This is negligible next to local training, and it avoids the compute contention that Agarwal et al. show can erase the gains of heavier per-step communication tricks.
The third part is integration. Retention weighting replaces Delayed Nesterov's buffering. We also test the two in combination, since they may be complementary: one reweights late updates, the other controls when momentum is applied.
Baselines
The naive reference is asynchronous DiLoCo with no staleness handling. Liu et al. report that it fails, and we expect it to set the floor. The synchronous reference is standard DiLoCo at the same $H$. It gives the best loss per update, but it pays for idle time on heterogeneous hardware. The DiLoCo paper reports 500 times less communication than fully synchronous training at comparable perplexity on C4, and 8 times faster wall-clock time at 150M parameters.
The strongest competitor is Delayed Nesterov with Dynamic Local Updates from Asynchronous Local-SGD Training for Language Modeling. It is the only method in the file built for exactly this asynchronous, many-local-step regime. No reported numerical result for it was found on Lacuna. The file states only that it matches DiLoCo's perplexity per update and beats it in wall-clock time on C4, so we will reproduce it ourselves.
The second strong competitor is Gap-Aware division adapted to pseudo-gradients. The Gap-Aware paper reports 72.18% ImageNet test accuracy with 128 asynchronous workers, and "nearly two orders of magnitude better perplexity" than step-delay penalties on WikiText-103. We will also run a step-delay penalty, $1/(1+\Delta)$.
Fairness controls apply to all methods. Every method sees the same data order per worker, the same simulated speed distribution, the same token budget and the same number of workers. Each gets an equal grid over outer learning rate and outer momentum. Down-weighting late updates shrinks the effective step, so a method that wins only through smaller steps would be caught by the retuned learning rate.
Two ablations isolate the key ingredient. The first uses one global curve instead of per-block curves. The second removes the plateau by setting $\kappa_b = 0$.
Beating the strongest baseline means two things at 32 workers with a heavy-tailed speed distribution. The method must reach DN+DyLU's final C4 perplexity in at least 15% less wall-clock time, or reach lower perplexity at matched wall-clock time. The margin must exceed the spread across three seeds.
Why not just…?
Why not just use Gap-Aware's parameter drift? Gap-Aware's own low-worker results show that a drift penalty with no tolerance can hurt. The penalty also ignores how outer momentum interacts with arrival order, which is exactly what breaks naive asynchronous DiLoCo. Still, drift may simply be the better covariate. The measurement study tests this directly. If drift wins, we would build the curve over drift instead of delay and keep the plateau and the per-block structure.
Why not just measure each late update's cosine with the server momentum when it arrives, and skip curve fitting? This is the most serious alternative, because it uses fresher information than any fitted curve. Our concern is that a single cosine between two very high-dimensional, noisy vectors mixes up two things: real obsolescence, and ordinary disagreement between workers that saw different data. Down-weighting on that basis would throw away useful diversity. The pooled curve averages out that noise. We run instantaneous-cosine weighting as a full baseline. If it matches our method, the forgetting-curve structure adds nothing, and we would say so.
Why not just lower the outer momentum or increase weight decay? Both are uniform contractions. They cannot distinguish a fresh update from a late one, or an embedding block from an MLP block. The KVFKT ablation illustrates the general point in student modeling: removing interval-dependent decay lowers accuracy on all four benchmarks even when other parameters are tuned. The equal tuning grid over outer momentum tests this within our own setting.
Why not just go synchronous and compress harder, for example with DeMo? DeMo cuts per-step traffic by up to 85 times, but it assumes synchronous workers. It still leaves stragglers idle, and its download volume grows with the number of workers. Heterogeneous hardware is the regime where asynchrony is needed at all. We include synchronous DiLoCo to show what idle time costs in our simulated setup.
Proposed Experiments
The main experiment trains 150M and 400M models on C4 with 8, 32 and 64 workers. Worker speeds follow either a uniform or a heavy-tailed distribution. We report validation perplexity against wall-clock time and against tokens, total bytes communicated, and downstream zero-shot HellaSwag, PIQA and ARC-Easy for the 400M runs.
A transfer check applies the per-block curve weighting to per-step asynchronous SGD on ImageNet-1K with ResNet-50. This compares directly with Gap-Aware's reported 72.18% at 128 workers and tests whether the plateau recovers Gap-Aware's losses at 8 workers.
The measurement study has a decisive result of its own. It either shows a measurable plateau that varies by block in the retention of late updates, or it shows retention falling smoothly from the smallest delays. In the second case the premise fails.
A second decisive check is whether our method's advantage over DN+DyLU grows with worker count. Liu et al. report that asynchronous Local-SGD degrades as workers are added, so a gap that shrinks with scale would show the method is not addressing that failure. Finally, we test whether learned $\kappa_b$ and $S_b$ from the 150M run, applied frozen to the 400M run, keep most of the gain. That would indicate the curves reflect properties of the architecture rather than tuning to one run.
References
- DiLoCo: Distributed Low-Communication Training of Language Models
- Asynchronous Local-SGD Training for Language Modeling
- Gap-Aware Mitigation of Gradient Staleness
- KVFKT: A New Horizon in Knowledge Tracing with Attention-Based Embedding and Forgetting Curve Integration
- OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
- On the Utility of Gradient Compression in Distributed Training Systems
- DeMo: Decoupled Momentum Optimization