Spending Compression Savings on Extra Gossip Rounds in Low-Communication LLM Training

Low-communication training methods for large language models, such as DiLoCo, cut network traffic by letting each worker take many local optimizer steps (typically 30) before the workers average their models. Even so, they still require every worker in the cluster to stop at the same moment and average with all the others. That global barrier is fragile when workers sit on slow, unreliable, or geographically scattered links. The alternative is gossip: each worker averages only with a few neighbors in a sparse network such as a ring or a mesh, with no central server. Gossip removes the barrier, but agreement then spreads slowly across the network, and after 30 unsynchronized steps the workers may already have drifted far apart.

This note argues that the large compression ratios recently shown for these outer updates should not be banked purely as bandwidth savings. They should be reinvested in more gossip rounds per synchronization. Sending four messages at 1% density, instead of one at 4%, costs the same number of bytes but contracts disagreement between workers roughly four times as fast in the exponent. The core claim is that, at matched bytes, splitting the communication budget into several heavily compressed gossip rounds with error feedback beats a single denser round. With this allocation, pure peer-to-peer training on sparse graphs can approach the loss of all-reduce methods like SparseLoCo while keeping gossip's tolerance to failures and stragglers.

Where this sits

Two literatures meet here without quite touching. The first is decentralized consensus optimization, where each node mixes its parameters with neighbors' parameters through a weight matrix $W$ and then takes a local gradient step. On the Convergence of Decentralized Gradient Descent analyzes this scheme and shows how disagreement is governed by the graph's mixing rate $\beta$, the second-largest eigenvalue magnitude of $W$. That paper motivates the problem: a ring of 16 nodes mixes poorly. Communication Compression for Decentralized Training shows that quantizing gossip naively makes errors cascade through the graph. It fixes this by compressing differences between successive iterates, but it still diverges at 4 bits and supports only unbiased compressors. That paper constrains the design: we must compress differences and carry error feedback. AdaGossip, DoCoM, and compressed decentralized momentum methods refine compressed gossip with adaptive consensus steps, gradient tracking, and momentum. All of them communicate at every single optimizer step, and all were tested on small CNNs or NanoGPT. DoCoM also needs $9d$ memory per node, where $d$ is the number of parameters. We do not communicate every step, and we add no tracking variables.

The second literature cuts synchronization cost in data-parallel training. DiLoCo scaling laws show that $H=30$ local steps followed by an outer Nesterov step on the averaged parameter change (the "pseudo-gradient") scales to 10B parameters with 10–100x less bandwidth. SparseLoCo compresses those pseudo-gradients to 1–3% density with top-$k$ selection plus 2-bit quantization. It keeps the dropped residual locally as error feedback and even improves loss over dense DiLoCo. DeMo and its low-communication variant compress momentum in the frequency domain instead. All of these assume all-to-all aggregation. Sparse top-$k$ messages with different indices on each worker also do not fit hardware ring all-reduce, so they fall back to all-gather, whose cost grows with worker count.

The closest existing work is Factored Gossip DiLoCo. It replaces DiLoCo's blocking all-reduce with a background global average plus a blocking neighbor gossip on a statically chosen 43% of parameters. On a simulated 200 Mbps network it raises compute utilization from 53% to 66–100%. It still relies on a global averaging protocol, it sends exact full-precision parameters, and it uses one gossip pass per outer round. Our proposal removes the global average entirely and gossips compressed messages. Its question is also different: how to allocate a fixed byte budget between message density and the number of mixing rounds.

Core claim

Take $N$ nodes on a graph with mixing matrix $W$. Each outer round, every node runs $H$ local AdamW steps, forms its pseudo-gradient, and applies its own local Nesterov outer step. It then runs $r$ rounds of compressed gossip on parameters at density $\rho/r$.

The gossip follows the difference-compression pattern of Tang et al. Each node keeps a public copy of each neighbor's parameters. It sends top-$k$ plus 2-bit compressed differences between its true parameters and its own public copy, and it keeps the untransmitted residual, as in SparseLoCo's error feedback. The per-round byte cost is fixed by $\rho$ alone, independent of $r$.

A standard consensus argument suggests the disagreement among nodes after one outer round behaves like

$$\Xi_{\text{after}} \approx \beta^{2r}\,\Xi_{\text{before}} + c\,(1-\rho/r)\,\Xi_{\text{before}},$$

where $\Xi$ is the mean squared distance of node parameters from their average. The first term is $r$ exact gossip contractions, and the second term is the compression error that error feedback eventually recovers but which adds noise. On well-connected graphs, $\beta$ is small and $r=1$ suffices, so gossip offers nothing over all-reduce. On rings and tori, $\beta$ is close to 1, and the first term dominates. Raising $r$ at fixed bytes should then win until the compression term catches up.

The predicted optimum $r^*$ grows roughly like $1/(1-\beta)$, the number of rounds needed for information to cross the graph. This scaling relation is the paper's main falsifiable prediction.

The design also sidesteps the overlap paradox. On the Utility of Gradient Compression found compression sped up only 6 of 200+ datacenter configurations, because per-step encoding cost (for example, 45 ms for PowerSGD on ResNet-50) ate the savings. Here, top-$k$ encoding runs once every 30 steps on chunks of 4096 elements, so its cost is amortized. We will measure that cost rather than assume it away.

Research Program

The work has three parts.

The first is an implementation of compressed multi-round gossip DiLoCo. It uses point-to-point sends between neighbors only and needs no collective all-reduce. Each node stores its parameters, its public copies of its neighbors, a residual buffer, and its outer momentum.

The second is a measured map of final validation loss as a function of graph mixing rate $\beta$, number of gossip rounds $r$, density $\rho$, and local step count $H$, at matched total bytes. This map tests whether $r^*$ tracks $1/(1-\beta)$. It also fills the gap the research file notes: there are no characterizations of $H\ge 30$ local training over multi-hop graphs without a global barrier.

The third is a robustness study on time-varying graphs with link failures. The gossip mixing matrix there is no longer symmetric each round, which tests whether the public-copy error feedback stays stable when the doubly-stochastic assumption behind most theory fails.

Baselines

The naive reference is per-step data-parallel AdamW with exact all-reduce, run on the same data and token budget. It gives the loss ceiling and the byte cost that every method is measured against.

The strongest loss competitor is SparseLoCo with all-reduce. We use its reported configuration: $H=30$, chunked top-$k$ with 2-bit quantization, and outer error feedback on DCLM. The paper reports validation loss of about 2.70 for the 512M model at about 3% density, versus 2.76 for dense DiLoCo, and about 2.74 versus 2.78 at about 6.2% density. We rerun it on our hardware so the comparison is fair. Beating it on loss is not required. Success means coming within 0.02 loss of it on a 16-node ring at matched bytes, while winning on wall-clock under a 200 Mbps link model with stragglers and 5% link failures. That failure rate matches the regime where Factored Gossip DiLoCo reports that all-reduce stalls.

The strongest decentralized competitor is Factored Gossip DiLoCo. We run it in its GlobalM1LocalM2-subset configuration, and also with its exact messages replaced by SparseLoCo's compressor so that it gets the same compression we use. Beating it means lower loss at equal bytes, or equal loss with no global averaging protocol at all.

Dense DiLoCo with all-reduce provides the scaling reference. All methods get the same tuning budget over outer learning rate and momentum, with an equal number of trials each.

Two ablations isolate the key ingredient. The first fixes bytes and varies $r\in\{1,2,4,8\}$ with density $\rho/r$; the $r=1$ arm is "compressed gossip DiLoCo" without our reallocation. The second removes the residual buffer, to test whether error feedback is what keeps multi-round sparse gossip from accumulating error as in Tang et al. An independent-training-then-average arm serves as the lower bound.

Why not just…?

Why not just run SparseLoCo with all-reduce, since 1% density is already cheap? In a well-provisioned datacenter this is a real threat, and we expect it to win there; our fully connected graph arm should reproduce that outcome. The case for gossip rests on two other facts. Sparse messages with different indices on each worker force all-gather, whose cost grows with $N$. And any global barrier stalls under failures, which Factored Gossip DiLoCo documents. The wall-clock experiments under a 200 Mbps link model with failures settle whether these effects matter at 16–32 nodes. If they do not, the contribution shrinks to a characterization of when gossip is not worth it.

Why not just compress Factored Gossip DiLoCo's messages? This is the most direct competitor, so it is run as a baseline. Its background global average still needs a protocol spanning all workers and a single exact gossip pass. Our claim is specifically that, on sparse graphs, the same bytes buy more agreement when spread over several rounds. If compressed Factored Gossip matches us, the reallocation idea is refuted.

Why not just raise $H$ to cut communication further? The DiLoCo scaling results tune around $H=30$, and on a sparse graph a longer local phase only increases drift before mixing. The $H$ sweep in our grid tests this directly, at matched bytes, against adding gossip rounds.

Why not gossip exact parameters once per outer round? This is what the $r=1$, uncompressed arm measures. At the same bytes it can afford far fewer exchanges, and the analysis predicts it leaves $\beta^2$-level disagreement on rings.

Proposed Experiments

We pretrain 178M and 512M LLaMA-style models on DCLM, following the SparseLoCo setup, with 8, 16, and 32 nodes. The graphs are a fully connected graph, a 2D torus, a ring, a random 4-regular expander, and a time-varying version of the ring that drops 5% of links per round.

Metrics are validation loss against bytes sent per node, simulated wall-clock under a 200 Mbps per-link model with heterogeneous node speeds, consensus distance $\Xi$ over training, measured encoding time as a fraction of step time, and ARC-Easy accuracy. SparseLoCo reports 45.24% on ARC-Easy, DiLoCo 44.28%, and DeMo 41.92%.

The decisive check has two parts. On the 16-node ring at fixed bytes, loss must decrease from $r=1$ to some $r^*>1$. And the fitted $r^*$ across ring, torus, and expander must increase with $1/(1-\beta)$. If $r=1$ is best on every graph, the central claim is wrong. If $r^*$ does not track the mixing rate, the mechanism is something other than faster consensus. A secondary check is that the residual-buffer ablation degrades sharply at $r\ge 4$, which would confirm that error feedback is what makes sparse multi-round gossip stable. Finally, a 1.5B run on FineWeb against the Factored Gossip DiLoCo configuration tests whether the result holds at larger scale.