Staleness Corrections That Keep Cloned Experts Apart

A common way to grow a language model cheaply is to copy parts of a trained checkpoint. Experts in a mixture-of-experts model (a network where a small learned gate sends each token to a few of many "expert" sub-networks) are duplicated, and a little random noise is added so the copies can drift apart and specialize. Separately, large models are increasingly trained asynchronously: workers apply gradients computed on slightly outdated weights instead of waiting for each other. Tricks that damp those outdated ("stale") gradients are what make asynchronous training stable. This note argues that the two practices interfere. Staleness corrections shrink every update in proportion to how outdated it is. They cannot tell the update that moves two copies together from the small difference update that pushes them apart. So in asynchronous training they may slow or stall the specialization that copying depends on.

The proposal has two parts. The first is to measure how fast cloned experts separate under synchronous training, under raw asynchronous training, and under standard staleness corrections. The second is a cheap fix: split each cloned group's update into a shared part (the group mean) and a difference part (each copy's deviation from that mean), and apply the staleness correction only to the shared part. The claim can be stated simply. Staleness mostly hurts the large, shared update direction. Specialization lives in the tiny difference direction. Treating the two separately should keep asynchronous stability without giving up expert diversity.

Core Claim

Two facts from the literature point toward the tension.

First, copied units sit on symmetric saddle regions of the loss, where the copies cannot specialize without a perturbation. This was shown for two-layer ReLU networks in Symmetry-Breaking Convergence Analysis of Certain Two-layered Neural Networks with ReLU nonlinearity. Barriers for Learning in an Evolving World formalized it as "cloned-unit manifolds" that SGD and Adam never leave; there, cloning $R^2$ stays at 1.0 on CIFAR-10 unless noise or dropout is added. Escape therefore depends entirely on how fast the difference between copies grows.

Second, staleness mitigations work by shrinking updates. Gap-Aware Mitigation of Gradient Staleness divides each parameter's update by how far that parameter drifted while the gradient was being computed. Delay-adaptive rules scale the step roughly as $1/\tau$, where $\tau$ is the number of updates the gradient is out of date (Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delays).

A small calculation shows why these interact badly. Write $d_t$ for half the difference between two cloned experts along the direction in which the loss curves downward. Near the symmetric saddle this direction has curvature $-\lambda$ with $\lambda>0$. With learning rate $\eta$, staleness $\tau$ and a mitigation factor $c\le 1$, the difference evolves approximately as

$$d_{t+1} = d_t + c\,\eta\lambda\, d_{t-\tau},$$

which is the linearized gradient step applied with a delayed argument. Its growth rate per step is $r-1$, where $r$ solves $r^{\tau+1} = r^{\tau} + c\eta\lambda$. Without delay the rate is $c\eta\lambda$. Once $\tau c\eta\lambda$ is large, the characteristic equation gives a rate of roughly $\log(\tau c\eta\lambda)/\tau$.

This leads to three predictions. Delay alone slows separation. The mitigation factor $c$ slows it further, multiplicatively. Corrected asynchronous training should therefore separate clones most slowly of all, even though it gives the best stability on the shared loss.

A second observation limits a tempting idea from the file's candidate gaps, namely that staleness could replace injected noise as the symmetry breaker. A stale gradient computed at exactly symmetric weights on the same data is still symmetric. Staleness can only create asymmetry when the copies experience different delays or different data, for example when the copies are placed on different expert-parallel nodes. That makes "staleness as free noise" a question about how clones are placed across workers, not a general property.

The proposed fix follows from the calculation. For a group of $k$ clones with updates $\Delta_1,\dots,\Delta_k$ and group mean $\bar\Delta$, apply

$$\Delta_j' = c\,\bar\Delta + (\Delta_j - \bar\Delta).$$

Here $c$ is whatever penalty the base method computes (Gap-Aware, delay-adaptive, or the stage-delay schedule of basis rotation). The rule wraps any existing staleness method and costs one mean per clone group. It only requires knowing which parameters were cloned, and an upcycling procedure knows this by construction.

Research Program

The work proceeds at three scales. The first is the controlled unit-cloning experiment of Barriers for Learning in an Evolving World on CIFAR-10, run under simulated asynchronous SGD with 4 to 48 workers. At this scale the difference vector $d_t$ can be tracked exactly and checked against the delayed-growth prediction.

The second is the roughly 100M-parameter nanoMoE on OpenWebText from Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation, using its public code (LOG-postech/basis-rotation), which already implements asynchronous pipeline schedules. We first train a nanoMoE with half the experts and then double the expert count by noisy duplication with $\alpha=0.01$, following Beyond Sunk Costs. Training then continues under pipeline staleness at 8, 16 and 32 stages, and under simulated asynchronous data parallelism with Gap-Aware correction.

The third is a scaled-up check at the largest model the budget allows, and it is run only if the mid-scale effect is clear.

Across all three scales we measure four things: the norm of each clone pair's difference over time, cloning $R^2$ between the output activations of paired experts, how evenly the gate splits tokens within each clone pair, and validation loss or perplexity at a fixed token budget and at a fixed wall-clock budget.

A secondary study tests the placement question directly. Clones are put on the same worker (same delay) or on different workers (different delays), with the injected noise either kept or removed. The question is whether asymmetric placement alone breaks symmetry, and how much it adds on top of noise.

Baselines

The naive reference points are synchronous training after noisy duplication, which is the setting in which Beyond Sunk Costs reported a 10.6% average accuracy gain over training from scratch at 70B scale, and uncorrected asynchronous training. The synchronous run sets the separation speed and loss we hope to approach. The uncorrected asynchronous run shows how much staleness alone costs.

The strongest existing staleness methods are Gap-Aware correction and basis rotation. Gap-Aware Mitigation of Gradient Staleness reports 72.18% ImageNet top-1 accuracy with 128 asynchronous workers, and on WikiText-103 perplexity roughly two orders of magnitude better than staleness-aware Adam. Basis rotation reports 46.8% fewer iterations to target loss on nanoMoE, and a 1.27x slowdown at 32 stages compared with 5.81x for standard asynchronous pipelines. These are the methods that currently make asynchronous training work, so any split correction must be compared against them directly. Each base method runs with its published settings. It then runs again, with an identical tuning budget over learning rate and penalty strength, wrapped in the mean/difference split, on identical data order, identical duplication seeds and identical hardware schedules. We found no reported result on Lacuna for any staleness method applied after expert upcycling, so all post-growth numbers will be new.

Beating a base method means one of two outcomes. The first is equal or lower validation loss at a fixed token budget together with at least as fast clone separation. The second is matching the base method's iterations-to-target on nanoMoE while clone-pair $R^2$ falls measurably faster.

The key ablation isolates the difference direction. We apply the penalty only to the difference part, which is the inverse of our rule and should trap the clones. We also exempt all cloned parameters from the penalty entirely. If the split helps only because cloned parameters are penalized less overall, full exemption will match it.

Why not just…?

Why not just inject more noise at duplication time? A larger $\alpha$ starts the difference vector further from the saddle, so it could offset slower growth. The available evidence counts against large noise, though. $\mu$pscaling small models ($μ$pscaling small models: Principled warm starts and hyperparameter transfer) found that noise must be calibrated to width to preserve training behavior, and that upscaled models still overfit on CIFAR-100. Beyond Sunk Costs found that noise stops helping once the learning rate is being annealed, which means ongoing dynamics matter and a one-time kick does not settle the outcome. We sweep $\alpha$ over a decade under Gap-Aware correction. If some larger $\alpha$ recovers synchronous-level separation without a loss penalty, the split rule is unnecessary, and we will report that.

Why not just train synchronously during the growth phase? This is a real option. The cost is paying straggler time exactly when models are grown mid-run, which Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delays motivates avoiding. We report a hybrid schedule, synchronous for a warm period and then corrected asynchronous training, and compare on wall-clock-to-target rather than tokens.

Why not just switch off the staleness correction for a while after growth? This is the most serious threat. Gap-Aware already observed that early penalization can hurt when few workers are used. The split rule differs because it keeps damping the large shared update right after growth, when losses spike and staleness damage is greatest, while leaving only the difference unpenalized. The full-exemption ablation, run with a tuned exemption window, settles the question. If exemption matches the split on both loss stability and separation, the contribution shrinks to a scheduling recommendation, and we will say so.

Decisive Checks

The central experiment is nanoMoE upcycling at 32 asynchronous pipeline stages with basis rotation, run with and without the split. Three outcomes would support the claim. Clone-pair $R^2$ should fall more slowly under corrected asynchronous training than under synchronous training. The split should close most of that gap. It should do so without losing the stage-scaling benefit that basis rotation reports.

The CIFAR-10 cloning experiment tests the mechanism. Measured growth rates of the difference vector should follow the delayed-growth equation as $\tau$ and $c$ vary. If they do not, the linear explanation is wrong even if the fix happens to help.

The placement study checks the negative prediction. With injected noise removed, clones on a single worker should stay exactly identical under any staleness. Only different-worker placement should break the symmetry.