When many accelerators train one model, they spend much of their time waiting to exchange gradients. Reports in the literature range from up to 50% of wall-clock time on multi-GPU clusters to up to 92% for large language model pre-training (SEPARATE). Local SGD is a standard way to wait less. Each worker takes several optimizer steps on its own data, and the workers then average their weights. That leaves two knobs to set: the learning rate and the number of local steps between averages. Separately, there is a line of work on statistical learning-rate schedules. These methods watch the optimization trajectory and cut the learning rate when the run stops making directed progress and starts bouncing around a minimum. Some of them need two parallel copies of the model to do this, which doubles the cost.
The observation behind this note is that Local SGD already runs that experiment for free. Every communication round starts all workers from the same averaged weights and sends them off on different mini-batches. That is exactly the "split" which SplitSGD creates on purpose, here with $M$ workers instead of two threads. Whether the workers' displacements point the same way tells us whether training is still in the directed phase. Computing that agreement costs one extra scalar per worker per round. We propose using this signal to drive both the learning-rate cut and a matching lengthening of the communication period. The schedule that STL-SGD hard-codes then becomes a closed-loop, data-driven one.
Core Claim
Let $\Delta_m = \theta_m - \bar\theta$ be worker $m$'s displacement after a round of $H$ local steps from the shared start $\bar\theta$. The averaging step already computes $\sum_m \Delta_m$. If each worker also sends the scalar $\|\Delta_m\|^2$, every node can form the average pairwise inner product between workers, normalized by the average squared displacement:
$$ \rho = \frac{\|\sum_m \Delta_m\|^2 - \sum_m \|\Delta_m\|^2}{(M-1)\sum_m \|\Delta_m\|^2}, $$
where $M$ is the number of workers. By construction $\rho$ lies between $-1/(M-1)$ and $1$. Its behavior separates the two phases. While training makes directed progress, the workers move along a shared descent direction and $\rho$ stays near 1. Once the run is stationary, the displacements are dominated by independent mini-batch noise and $\rho$ scatters around 0.
The claim has three parts. First, a SplitSGD-style rule applied to $\rho$ should match the timing quality of the expensive single-machine diagnostics at essentially zero cost. The rule declares stationarity when a fraction $q$ of the last $w$ rounds have $\rho$ below a small threshold, then multiplies the learning rate $\eta$ by $\gamma$ and the period $H$ by $1/\gamma$, following the $k = O(1/\eta N)$ relation of STL-SGD. Second, because $\rho$ averages over $M(M-1)/2$ worker pairs and over $H$ steps, it should be markedly less prone to false triggers than consecutive-gradient tests. Pesme et al. showed that the high variance of those tests causes repeated premature decays, which they call "abusive restarts." Third, coupling the period to the detected phase should reduce communication rounds relative to a fixed schedule without losing accuracy.
The same quantity also says something about drift between workers. Woodworth et al. and Balles et al. treat that drift as uniformly harmful bias. We instead read it as a phase indicator: coherent drift means the run is still in its directed phase, and drift that looks like noise means it has stalled.
Relation to Prior Work
Three papers motivate and supply the method.
SplitSGD splits one model into two threads on separate batches and decays the learning rate when their gradients stop agreeing. It is a single-machine scheduler that pays for the extra thread. We reuse its trigger logic, but compute the statistic from replicas that Local SGD already maintains, over $M$ workers rather than two.
Coupling-based SGD runs two chains on identical batches so that sampling noise cancels. It explicitly costs double memory and FLOPs. We deliberately do not replicate its design across workers, because feeding workers identical batches throws away data parallelism. Our statistic exploits the opposite situation: workers see different data.
The distance diagnostic of Pesme et al. tracks the log-log slope of $\|\theta_n - \theta_0\|^2$. It is also free to compute on the averaged trajectory, and it serves as our main competing diagnostic.
Three papers constrain the design.
STL-SGD establishes that the communication period may grow as the learning rate shrinks. It does so on a fixed geometric schedule, so it cannot tell whether a stage ended too early or too late.
ADA-LUPA-SGD grows the period according to forms derived from worst-case bounds under the Polyak-Łojasiewicz condition, not from measurements taken during the run.
LocalAdaScale measures variance and drift online, but only to rescale the learning rate for a fixed period $H$. It also shows that Local SGD beats synchronous gradient accumulation in wall-clock time only when one communication round costs more than about four local compute steps. That bounds where any of this matters.
DiLoCo uses a static $H=30$ and names adaptive cadence as open. It is the large-scale setting we aim at last.
Research Program
The first component is analysis on a strongly convex quadratic with additive noise. We will derive the mean and variance of $\rho$ in both phases as functions of $M$, $H$ and $\eta$. We will then compare its false-trigger probability with Pflug's consecutive-gradient test and with the distance slope. The expected variance reduction comes from averaging $M(M-1)/2$ pairwise products, which are only weakly dependent at stationarity. That predicts false-trigger rates falling roughly as $1/M$. The analysis must state when this fails.
The second component is an implementation of the controller in a standard Local SGD loop with momentum SGD inside each worker. It has three variants: the diagnostic cuts $\eta$ only with $H$ fixed; the diagnostic lengthens $H$ only, under a fixed $\eta$ schedule; and the coupled rule that changes both.
The third component is a stress study of two known threats to the signal. The first is non-IID data. When workers have different local optima, their displacements disagree for systematic reasons, not because of noise. That disagreement can push $\rho$ toward zero early and trigger premature decays. We will measure this and test a correction that subtracts a heterogeneity floor estimated from the first round after each decay.
The second threat is compressed outer communication. If the averaged update is sent through DeMo-style sparsification (DeMo) or 4-bit quantization (SDP4Bit), the first term of $\rho$ is computed from a lossy sum. Each worker's own norm remains exact. We will quantify how compression error biases $\rho$.
The final component carries the method into a DiLoCo-style language-model setting, where the inner optimizer is AdamW. Combining such diagnostics with coordinate-wise adaptive optimizers is unformalized, as Chee et al. note for the simpler momentum case. This component is therefore exploratory rather than a central claim.
Baselines
The naive reference is fixed-period Local SGD with a hand-set step decay and constant $H$. This is the configuration practitioners actually run. On the communication-rounds metric it is matched by the STL-SGD paper's own comparison, in which standard Local SGD needed 755 rounds to reach 99% training accuracy with ResNet-18 on non-IID CIFAR-10. We also report synchronous SGD, which needed 7644 rounds in that comparison, to anchor the communication axis.
The strongest existing method on this exact question is STL-SGD. It is the only method in the file that couples learning-rate decay to period growth, and it reports 434 rounds to 99% training accuracy on non-IID CIFAR-10 with ResNet-18 (STL-SGD). We will reproduce it in our codebase with the same model, data split, worker count, initial $\eta$ and $H$, and the same total gradient-step budget. Its stage length will be tuned over a grid of equal size to the grid we spend on our $(w, q)$.
Beating STL-SGD means two things. We must reach the same training-accuracy target in fewer rounds, and we must be no worse on test accuracy. The decisive version of this comparison is robustness, not peak performance. When STL-SGD's stage length is mis-set by a factor of two in either direction, its rounds-to-target should degrade, while ours, having no stage length, should not.
LocalAdaScale is the strongest method that measures statistics online. We run it at its best fixed $H$ under the same budget; no reported result was found on Lacuna for its CIFAR-10 accuracy.
For distributed language modeling, the reference is DiLoCo with $H=30$. Its paper reports lower evaluation loss than data parallelism at $M=2$ for 2.4B parameters and above. We could not afford that scale and will compare at small scale on C4 under matched tokens.
Two ablations isolate the key ingredient. The first replaces $\rho$ with the Pesme distance slope computed on the averaged model under the same coupled $\eta$/$H$ rule. This tests whether looking across workers matters, or whether any free trajectory statistic would do. The second is the $\eta$-only variant, which tests whether lengthening $H$ contributes anything beyond better-timed decays.
Why not just…?
Why not just use cosine annealing with a fixed period? Cosine needs the training horizon in advance and never lengthens the communication period, so its communication cost is the same as fixed Local SGD. A cosine run with a period schedule tuned by hand could plausibly match us at a known horizon. We include tuned cosine with STL-style period growth as a baseline. Our case rests on two experiments: an untuned transfer to a new dataset, and early-stop and extended-horizon runs, where a fixed schedule is mistimed. The coupling paper's finding that a manual drop at epoch 75 was premature compared with a detected plateau at epoch 145 (Coupling-based SGD) suggests that hand timing is fragile.
Why not just reduce the learning rate when validation loss plateaus? This is the most serious threat. It is cheap, needs no theory, and already works with any communication scheme. Its weaknesses are that it reacts with a lag of several evaluation windows, that it depends on a held-out split and on a patience setting, and that it says nothing about when the period can safely grow. We will run it with tuned patience under the same coupled $\eta$/$H$ rule. If it matches $\rho$ on rounds-to-target and test accuracy, we will report that the cross-worker statistic adds little beyond convenience.
Why not just use synchronous gradient accumulation and avoid drift entirely? On fast interconnects that is the right answer. Balles et al. show Local SGD wins in wall-clock time only when a round costs more than about four compute steps (On the Choice of Learning Rate for Local SGD). We therefore report wall-clock time under simulated communication-to-compute ratios of 1×, 4× and 16×, and expect to lose at 1×.
Why not just run SplitSGD or the distance diagnostic unchanged? SplitSGD's extra thread costs compute, and it was never designed around communication. The distance slope is free, and it is exactly our first ablation. If it matches $\rho$, the contribution reduces to the coupled control rule.
Proposed Experiments
Experiments on synthetic data come first. We use least-squares and the logistic regression setup of Woodworth et al., with 50,000 points in $\mathbb{R}^{25}$, and $M \in \{2, 4, 8, 16\}$. The decisive check here is whether the false-trigger rate of $\rho$ falls with $M$ and sits below both Pflug's test and the distance slope at matched detection delay. Detection delay is measured against the round at which the optimality gap stops improving.
Vision comes next: ResNet-18 on CIFAR-10 and CIFAR-100 with $M=8$, under IID data and under the non-IID split used by STL-SGD. We measure rounds to 99% training accuracy, final test accuracy, total bytes communicated, simulated wall-clock time, and the decay epochs chosen, compared against the epoch-140 to epoch-145 plateaus reported by the single-machine diagnostics. The decisive check is robustness: across three seeds and a twofold mis-specification of every competitor's schedule, our rounds-to-target varies less and stays at or below STL-SGD's.
The non-IID run is the falsification test. If $\rho$ triggers decays well before the distance diagnostic does under heterogeneity, and the floor correction does not fix this, the method is restricted to near-IID settings, and we will say so.
The compression check reruns the CIFAR experiments with the averaged update sparsified with top-$k$ selection, sweeping the fraction of coefficients kept. It measures the drift in $\rho$ and in the decay epochs.
Finally, a small DiLoCo-style run on C4, at roughly 35M to 150M parameters with $M \in \{2, 4\}$, compares a fixed $H=30$ against the diagnostic-driven $H$ on evaluation loss per byte communicated. It is exploratory and serves as a feasibility test for adaptive inner optimizers.
References
- SEPARATE: A Simple Low-rank Projection for Gradient Compression in Modern Large-scale Model Training Process
- Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic
- STL-SGD: Speeding Up Local SGD with Stagewise Communication Period
- On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent
- Is Local SGD Better than Minibatch SGD?
- On the Choice of Learning Rate for Local SGD
- Coupling-based Convergence Diagnostic and Stepsize Scheme for Stochastic Gradient Descent
- Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization.
- Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
- DeMo: Decoupled Momentum Optimization
- SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
- Understanding and Detecting Convergence for Stochastic Gradient Descent with Momentum