Training one language model across several compute sites joined by thin links is now practical because of methods like DiLoCo. Each site trains its own copy for many steps and only occasionally sends a summary of how its weights moved. Over a capacity-limited backhaul, which is the shared transport link from distributed sites to a central point, even these occasional summaries are expensive. A full-precision copy of the weight change has to cross the link from every site at every synchronization. These summaries are highly redundant. All sites start from the same weights, train on samples from the same distribution, and use the same optimizer, so their weight changes largely point the same way. No single site can exploit that redundancy, because no site sees the others' updates. Only the central server sees them all.
Wireless engineers solved this information structure years ago for cloud radio access networks, where base stations forward compressed copies of correlated received signals to a central processor. In Wyner-Ziv coding, which compresses a signal for a decoder that holds correlated side information the encoder lacks, each base station sends only enough bits to resolve its signal given what the processor already knows. The processor decodes the stations one after another and uses earlier decodes as side information for later ones. This scheme comes within 1 bit per station of the sum capacity under a total backhaul budget (Uplink Multi-Cell Processing: Approximate Sum Capacity under a Sum Backhaul Constraint). We propose to carry this successive side-information decoding over to infrequent-synchronization training. The server decodes the first site's weight change, then uses the running partial average as side information to decode each later site with far fewer bits. Under this scheme, the bits each site must send should scale with how much that site disagrees with the others, not with the size of its update.
Core Claim
In DiLoCo-style training, the "pseudo-gradient" is the weight change a replica accumulates over $H$ local steps. We conjecture that across replicas these pseudo-gradients share a large common component. If so, a server-side successive decoder with a modulo (lattice) quantizer can cut uplink backhaul bits per synchronization substantially below independent quantization of each replica, at matched final loss. The cost of encoding, which is what makes gradient compression a net loss in datacenters, is paid once every $H$ steps and so becomes negligible.
The mechanism is concrete. Each replica $i$ rotates its pseudo-gradient $x_i \in \mathbb{R}^d$ with a randomized Walsh–Hadamard transform, quantizes with step $\epsilon_i$, and transmits only each coordinate modulo a lattice of width $2\Delta_i$. The server holds side information $y_i$, the average of the already-decoded replicas, and recovers $x_i$ as the lattice point nearest $y_i$. Decoding succeeds whenever $\|x_i - y_i\|_\infty < \Delta_i$ in the rotated basis. The cost per coordinate is roughly
$$b_i \approx \log_2\!\left(1 + \frac{2\Delta_i}{\epsilon_i}\right),$$
where $\Delta_i$ now bounds the cross-replica disagreement, not the update's own range. Independent quantization pays the same formula with $\Delta_i$ replaced by the range of $x_i$ itself, so the saving per coordinate is about $\log_2$ of the ratio between update range and disagreement range. The theory comes from Wyner-Ziv Estimators: Efficient Distributed Mean Estimation with Side-Information. That paper proves this modulo quantizer is near-optimal for mean estimation with server side information. It treats the side information as passively given, for example past iterates, and runs no training experiments. Our contribution is to construct the side information from the other replicas within the same round, to choose the decoding order and per-replica bit allocation under a sum backhaul budget, and to test the whole scheme on real transformer pretraining.
Research Program
The first piece of work is measurement, because the idea stands or falls on one number. That number is the disagreement ratio $\rho = \|x_i - \bar{x}_{-i}\| / \|x_i\|$, where $\bar{x}_{-i}$ is the mean of the other replicas' pseudo-gradients. It should be measured across model size, number of replicas $M$, local step count $H$, and data heterogeneity. If $\rho$ sits near 1, replicas are nearly independent and cross-replica coding cannot help. If $\rho$ is around 0.3 or lower, a saving of a bit or more per coordinate is available. The side information also improves as more replicas are decoded, since the partial average sharpens, so later replicas should cost progressively less.
Two predictors must be separated, because the encoder already knows one of them. The previous outer update and outer momentum are shared by all sites, so ordinary predictive coding, which subtracts a predictor both ends know, removes that part of the correlation without any Wyner-Ziv machinery. Our encoder therefore first subtracts this shared predictor. It then applies modulo coding against the cross-replica side information, which only the server has. The ablations attribute savings to each stage separately.
The second piece is bit allocation under a sum backhaul constraint $\sum_i b_i d \le B$. Here we borrow the proportionality rule from Uplink Multi-Cell Processing, which sets each station's quantization noise in proportion to its background noise. The analogue we test treats each replica's own disagreement with consensus, $s_i = \|x_i - \bar{x}\|$, as the "background noise" of its contribution. On this view, adding quantization noise far below $s_i$ wastes bits. The allocation rule is $\epsilon_i \propto s_i$ with decoding order chosen by increasing $s_i$, compared against equal-noise allocation. This is a hypothesis grounded in an analogy, not a derived optimum. The training objective is not sum rate, so we compare against equal allocation empirically and do not claim the radio-network result transfers.
The third piece is robustness. A modulo decoder fails badly when the disagreement exceeds $\Delta_i$, because a coordinate wraps to the wrong lattice point. Each replica therefore sends a small per-block norm and a short checksum. Blocks that fail the check are retransmitted at the independent-quantization rate. The overflow rate and its bit cost are reported as first-class metrics. Shared randomness for the rotation is set once at startup over the control channel. This addresses the concern in the Mayekar et al. analysis about maintaining shared seeds, which matters for raw radio links but not for packetized backhaul with retransmission.
Finally, we characterize wall-clock time. On the Utility of Gradient Compression in Distributed Training Systems found compression helped in only 6 of over 200 datacenter configurations, because encoding time competes with the backward pass. That paper constrains our design. We do not compress per-step gradients; we compress once per outer round. The rotation is $O(d \log d)$ and runs while the GPUs are otherwise waiting on the link. We will measure encode and decode time directly and report the link capacity below which the scheme yields a net speedup.
Baselines
The naive reference point is DiLoCo with pseudo-gradients sent uncompressed in bf16 (DiLoCo: Distributed Low-Communication Training of Language Models). It shows how much the link costs before any coding. That paper reports perplexity comparable to fully synchronous training on C4 while communicating 500 times less. Fully synchronous data-parallel training is included at small scale as the loss ceiling.
The strongest competitor is DiLoCo with independent per-replica quantization of the pseudo-gradient: the same rotation and the same scalar quantizer, but no server-side information. This isolates exactly our ingredient. I found no reported result on Lacuna for quantized DiLoCo pseudo-gradients, so we run it ourselves at 2, 4 and 8 bits. A second strong competitor is DeMo (DeMo: Decoupled Momentum Optimization). DeMo communicates top-$k$ discrete cosine transform components of the momentum every step and reports 7.5 MB versus 637 MB per step on OLMo 300M with no downstream loss. We compare against it at matched total uplink bytes over the run. A third is per-step 2-bit gradient quantization in the style of BitDP, which reports GPT-350M validation perplexity of 22.74 against 22.17 for full precision. Our own runs of these per-step methods will use our data and models, not borrowed numbers.
Fairness is enforced by fixing data order, model, token budget, inner optimizer and outer optimizer across arms. Each arm gets the same tuning budget of an equal number of sweeps over learning rate and outer momentum. Every arm sits under the same simulated star backhaul with the same sum capacity. The ablations remove one stage at a time: shared-predictor subtraction only, cross-replica modulo coding only, equal versus proportional allocation, and random versus sorted decoding order. Beating the strongest baseline means one of two outcomes. Either the method reaches the same C4 evaluation loss, within seed noise, with at least 25% fewer total uplink bits than independent quantization at its best bit width, or it reaches lower loss at equal bits. This must also hold in wall-clock time under links of 1 Gbps and below.
Why not just…?
Why not just increase $H$ and send less often? Raising $H$ is the cheapest lever, and Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo shows the approach scales to 10B parameters. It also cuts a 405B model's bandwidth need from 222 to 3 Gbit/s. Larger $H$ costs data efficiency, though, and drives replicas into different loss basins, a failure mode that Fournier et al. counter with weight shuffling. Our method acts on the orthogonal axis of bits per synchronization, so the fair test is a Pareto plot of loss against total bits over a sweep of $H$ for both methods. If tuning $H$ alone traces the same frontier as coding at fixed $H$, the proposal fails. One genuine risk is that disagreement grows with $H$ and shrinks the coding gain exactly where it would matter most. The $\rho$-versus-$H$ measurement settles this.
Why not just subtract the previous global update and quantize the residual, which needs no side-information decoder? This is the most serious threat, because a predictor known at both ends always beats one known only at the server. We build it in as the first stage and report it as its own baseline. Cross-replica coding earns its place only if the remaining residuals of different replicas still share structure. That would be the case if replicas within a round agree with each other more than with last round's update. The ablation measures this directly.
Why not just use 1-bit or sign compression, which is already cheap? Sign methods like BitDP ignore cross-worker correlation entirely and need all-gather traffic that grows with the number of workers. They also operate per step, so they do not combine naturally with DiLoCo's infrequent pseudo-gradients. Sign-quantizing the pseudo-gradient is a cheap variant, and we include it in the independent-quantization sweep. If 1-bit independent coding already matches our loss at fewer bits, coding gain is moot.
Why not just rely on averaging to cancel independent quantization noise? Independent noise does shrink by $1/M$ in the average. This argues for using coarse quantizers in every arm, which we do. Averaging shrinks the noise but not the bit cost, which still scales with each update's full range. That bit cost is the term Wyner-Ziv coding attacks.
Proposed Experiments
The main testbed is Chinchilla-style decoder-only transformers of 35M, 150M and about 400M parameters on C4. This follows the setup in the DiLoCo scaling-law study, with $M \in \{4, 8, 16\}$ and $H \in \{30, 100, 500\}$. A replication on OLMo 300M trained on Dolma v1.5 (Dolma) allows direct comparison with DeMo, with zero-shot HellaSwag, ARC-Easy and PIQA as downstream checks. To stress the method, a non-IID arm gives each replica a different Dolma source, which should raise $\rho$ and shrink the coding gain. The backhaul is emulated as a star with sum capacity from 20 Mbps to 10 Gbps. That range spans the wireless backhaul regime cited by FedSpar, whose MNIST-only evaluation (89.70% at 0.4 bits per entry) never reaches transformers, up to the datacenter regime where Agarwal et al. found compression unhelpful. Metrics are evaluation loss against cumulative uplink bits, wall-clock time to target loss, decoder overflow rate, and the measured encode and decode time.
The study is decisive in three places. First, measure $\rho$ before building anything; if it exceeds roughly 0.7 at our settings, report the negative result and stop. Second, the shared-predictor-only ablation must leave a measurable residual gain for cross-replica coding. Without it, the Wyner-Ziv component is unnecessary. Third, the wall-clock crossover capacity must lie above realistic edge backhaul rates. Otherwise the bit savings are real but practically irrelevant.
References
- Uplink multi-cell processing: Approximate sum capacity under a sum backhaul constraint
- Wyner-Ziv Estimators: Efficient Distributed Mean Estimation with Side-Information
- On the Utility of Gradient Compression in Distributed Training Systems
- DiLoCo: Distributed Low-Communication Training of Language Models
- DeMo: Decoupled Momentum Optimization
- BitDP: Ultra-low-bit Communication for Data Parallelism in LLM Training
- Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
- WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average
- Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
- Communication-Efficient Federated Learning Over Capacity-Limited Wireless Networks