Low-communication training methods let each copy of a model (a "replica") take many optimizer steps on its own before replicas average their weights. The best known of these, DiLoCo, syncs every 30 steps no matter what the training dynamics are doing. That fixed interval is a guess. Syncing too often wastes bandwidth. Syncing too rarely lets replicas wander apart until averaging them hurts. This note proposes a statistically grounded trigger for that decision. At every step, each replica shares a tiny random sketch of its weights, a few dozen numbers. All replicas then run the same econometric test on these sketches. The test asks whether each replica's deviation from the group is still mean-reverting, meaning it is pulled back toward the others, or has started to behave like a random walk, meaning it drifts away with ever-growing variance. The group syncs when the evidence points to a random walk.
The econometric tool is a second-generation panel unit root test. A unit root test checks whether a series wanders like a random walk. A panel test does this jointly over many parallel series. Second-generation tests, CADF and CIPS, additionally strip out a shared common trend before testing. This fits the distributed setting unusually well. Every replica's weights are non-stationary, because all of them are moving downhill together. The quantity that matters is whether each replica stays tied to that shared trajectory. In econometric terms, the question is whether the replicas are cointegrated. Regressing each series on the cross-sectional average, which is exactly what CADF does, asks that question directly.
Core Claim
A small-sketch panel unit root test on replica deviations can set the synchronization cadence of local-update training. Two claims follow. First, at a matched number of syncs, it reaches lower validation loss than the best hand-tuned fixed interval. Second, because the test statistic has no units, a single significance level transfers across model sizes, learning rates, and data heterogeneity. A tuned divergence-norm threshold does not transfer this way. The second claim is what separates this proposal from simpler adaptive heuristics, and the experiments are built so that it can fail.
Why the Mechanism Should Work
Consider a locally quadratic loss with curvature $\lambda$ along some direction. Let $\eta$ be the learning rate. Let $d_{i,t}$ be replica $i$'s deviation from the replica mean along that direction at step $t$, with $\epsilon_{i,t}$ the replica's own gradient noise. Plain SGD gives approximately
$$d_{i,t} = (1-\eta\lambda)\, d_{i,t-1} + \eta\, \epsilon_{i,t},$$
which is a first-order autoregression with persistence $\rho = 1-\eta\lambda$.
In sharply curved directions, $\rho$ is well below 1. The deviation saturates at a bounded variance, so further local steps add little extra disagreement. In flat directions, or after the learning rate or curvature changes, $\rho \to 1$. The deviation then becomes a random walk whose variance grows linearly in time. That unbounded growth is the harmful drift behind the "drift penalty" in Is Local SGD Better than Minibatch SGD?, which proves that local iterates drift and that averaging drifted iterates can underperform minibatch SGD on non-quadratic objectives.
That paper motivates this work but gives worst-case bounds rather than a diagnostic. The proposal turns its mechanism into something that can be measured online. The relevant persistence changes during training as the learning rate decays and curvature evolves. So a trigger that estimates persistence directly should track the right cadence better than a schedule fixed in advance.
The trigger has a known blind spot. With heterogeneous data, deviations can settle around a nonzero offset. That offset is stationary bias, not a unit root. The test includes an intercept, so it will not flag this case. We treat the offset as outside the trigger's scope, and the heterogeneous-data experiments below measure how much this matters.
Research Program
The method has four parts: a sketch, a test, a decision rule, and a cost analysis.
The sketch works as follows. Each replica keeps $k$ random projections of its weights, using a projection matrix generated from a shared seed. This trick is borrowed from SEPARATE, which uses shared-seed Gaussian projections to compress gradients. We use it only to monitor the replicas, not to compress them. The sketch is updated incrementally from each step's weight change, and the $k$ numbers are exchanged by a single all-gather. At $k=64$ and $M=8$ replicas, this is a few kilobytes per step.
The test works as follows. Over a sliding window of $L$ steps, each replica computes a CADF regression for each of its sketch coordinates. The regression includes the cross-sectional mean of that coordinate and its lags, which absorb the common downhill trajectory. The individual statistics are then averaged into a CIPS statistic. Every replica holds identical gathered data, so every replica reaches the same decision with no further coordination.
The decision rule syncs when CIPS fails to reject a unit root at level $\alpha$, subject to minimum and maximum intervals. Otherwise training continues locally. The DiLoCo outer optimizer from Scaling Laws for DiLoCo is kept unchanged: it averages replicas every $H=30$ steps and applies Nesterov outer momentum to the averaged pseudo-gradient. Only $H$ becomes adaptive, which the DiLoCo authors list as an open challenge.
Two issues need care. The first is the testing itself. The tests are repeated every step, and windows are short, so the power of any single test is low. We therefore treat $\alpha$ as a single operating knob rather than a guaranteed error rate. We rely on pooling across $M\times k$ series to raise power, which is the reason panel tests are used at all. The second is cost. On the Utility of Gradient Compression in Distributed Training Systems found that encoding overhead of about 45 ms for PowerSGD erased the gains in nearly all of its more than 200 configurations. That result constrains this design. We will measure the sketch's per-step cost directly: roughly $k$ multiply-adds per parameter, or equivalently a coordinate subsample. We will also measure the test cost, which is tiny least-squares fits on scalars. Both will be reported against step time.
Baselines
The naive reference point is fully synchronous data parallelism, with gradients all-reduced every step and communication overlapped with the backward pass. It sets the loss ceiling and the bandwidth cost. The adaptive method only matters in bandwidth-limited settings. For context, SEPARATE reports communication consuming up to 92% of training time in such settings, and BitDP reports a 7.47x speedup on Ethernet over FP32 synchronous training.
The strongest existing method is DiLoCo with a fixed interval of $H=30$. It is the strongest competitor because it already reaches most of the achievable bandwidth savings. Charles et al. report lower held-out C4 evaluation loss than data parallelism at 2.4B, 4B, and 10B parameters with $M=2$ replicas. They also report compute utilization above 80% while using "10x to 100x less bandwidth than standard training" (Scaling Laws for DiLoCo).
For a fair comparison, we will not stop at $H=30$. We will sweep $H$ over a grid at each scale and report the best fixed $H$ found after the fact. This gives the fixed-interval approach an advantage the adaptive method does not get. The adaptive method uses a single $\alpha$ chosen at the smallest scale, and both methods get the same total tuning compute. Beating DiLoCo means lower C4 validation loss at an equal number of syncs, or equal loss with fewer syncs, against the best fixed $H$ at every scale.
The second adaptive competitor is STL-SGD from STL-SGD. It doubles the communication period each time the learning rate is halved. On non-IID CIFAR-10 with ResNet-18, it reports reaching 99% training accuracy in 434 rounds, versus 755 for Local SGD and 7644 for SyncSGD. We will reproduce this setting and count rounds to the same target.
The third competitor is a divergence-norm threshold. It syncs when the sketched distance between replicas exceeds a tuned value. It shares our sketch exactly, so it isolates the test statistic from the monitoring signal. No reported result was found on Lacuna for a norm-threshold sync trigger, so this baseline is built here.
Two ablations isolate the key ingredient. The first replaces CADF with plain ADF, which lacks the cross-sectional augmentation. We expect it to nearly always detect a unit root, because the common downhill trend dominates each series, and therefore to sync almost every step. If it does not, the cointegration framing adds nothing. The second varies the sketch size $k$ and the window $L$ to map the trade-off between test power and cost.
Why not just…?
Why not just tune $H$ once and keep it fixed? Fixed-interval DiLoCo is strong, and at a single scale and learning-rate schedule it may match the adaptive trigger. That is a real threat. The proposal's case rests on two things. The persistence $\rho = 1-\eta\lambda$ changes within a run as the learning rate decays, and the best $H$ shifts across scales and data mixes. The per-scale best-$H$ sweep settles this directly. If the best fixed $H$ matches the adaptive trigger everywhere, the adaptive trigger's only remaining value is avoiding the sweep, and we will say so.
Why not just sync when replicas are far apart, using a norm threshold? This is the most serious alternative, since it uses the same cheap signal. A norm threshold has units. The deviation it measures depends on learning rate, parameter scale, and noise level, so a value tuned on a 35M model has no reason to transfer to a 1B model. The unit root statistic has no units and asks a different question: is disagreement still growing, rather than how large is it? The decisive experiment tunes both methods at the smallest scale, freezes them, and transfers them across scales, learning rates, and non-IID splits. If the threshold transfers equally well, our central claim is false.
Why not just follow the learning-rate schedule, as STL-SGD does? STL-SGD captures the $\eta$ part of $\rho = 1-\eta\lambda$ but ignores curvature $\lambda$, which changes during training in ways the schedule cannot anticipate. If the persistence the test estimates is fully explained by the learning rate, STL-SGD will tie our method. We will check for this by regressing the trigger's chosen intervals on $1/\eta$.
Why not just compress each sync instead, as DeMo or low-bit quantization do? DeMo reports 55 MB per step versus AdamW's 2.4 GB on OLMo-1B (DeMo), and SDP4Bit reports a 4.08x speedup at 18B parameters. These methods shrink each message. The trigger decides how many messages to send. The two are complementary rather than competing, and we will run one combined configuration with DeMo-compressed outer syncs to confirm that the savings stack.
Proposed Experiments and Decisive Checks
The first check is a calibration test on known dynamics. On a quadratic toy problem where each direction's $\rho$ is known exactly, the trigger's sync times should match when the flat-direction deviation variance begins growing linearly. If the test cannot find a planted unit root here, nothing downstream matters.
The main experiment follows the DiLoCo C4 protocol with Chinchilla-style decoder transformers from 35M to about 1B parameters and $M\in\{2,4,8\}$ replicas, plus an overtrained run on Dolma. It reports validation loss against cumulative synced bytes and against wall-clock time under simulated 1 to 10 Gbit/s links. The heterogeneity experiment reruns the STL-SGD non-IID CIFAR-10 setting with ResNet-18, where drift is strongest and the fixed-offset blind spot is most likely to show.
Three outcomes are decisive. The first is whether the adaptive trigger, frozen at one $\alpha$, beats the per-scale best fixed $H$ at matched sync counts. The second is whether it transfers better than the frozen norm threshold. The third is whether the cost of the sketch and test stays below a few percent of step time, the budget that the Agarwal et al. result says any added overhead must respect.
References
- Is Local SGD Better than Minibatch SGD?
- SEPARATE: A Simple Low-rank Projection for Gradient Compression in Modern Large-scale Model Training Process
- Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
- On the Utility of Gradient Compression in Distributed Training Systems
- BitDP: Ultra-low-bit Communication for Data Parallelism in LLM Training
- STL-SGD: Speeding Up Local SGD with Stagewise Communication Period
- DeMo: Decoupled Momentum Optimization
- SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training