Gauge-Fixed Averaging: Removing Function-Preserving Drift Before Workers Synchronize

A neural network has many parameter settings that compute exactly the same function. Rescaling the weights that feed a normalization layer changes nothing. Multiplying a head's query matrix by any invertible matrix $A$, and its key matrix by $A^{-\top}$, leaves the attention scores unchanged. Parsing researchers have a name for this pattern: spurious ambiguity, where many derivations yield one meaning. Their standard remedy is a normal form, which keeps exactly one canonical representative per equivalence class. We propose bringing that remedy to communication-efficient distributed training.

In methods such as DiLoCo, independent groups of workers train for many steps and then average their parameter changes. During those steps, workers can drift apart along these function-preserving directions. The drift costs bandwidth. It also does harm when averaged, because the average of two equivalent parameterizations is generally not equivalent to either of them. Our core proposal is to align each worker's parameters to the shared global model along these symmetry directions, locally and with no extra communication, before the weight differences are formed. If the drift is real, this alignment should let workers go longer between synchronizations, or compress their updates harder, at the same loss.

Core Claim

Exact symmetries cannot move under pure gradient flow. For a scale-invariant weight the gradient is orthogonal to the weight, and for the query–key symmetry, gradient flow conserves the quantity $W_Q^\top W_Q - W_K^\top W_K$. In practice, three things break this protection. Adam rescales each coordinate separately, and that rescaling does not respect the symmetry. Weight decay pulls along symmetry orbits. Finite step sizes add second-order motion.

Different workers see different data, so they accumulate different motion along these orbits over $H$ local steps. Our claim is that this motion forms a measurable share of the disagreement between workers' pseudo-gradients (the weight change each worker reports). Averaging this component injects error into the products that actually define the function, such as $W_Q W_K^\top$. Removing it before averaging is a normal form for optimizer trajectories.

The method works as follows. Let $\theta_m$ be worker $m$'s parameters after $H$ local steps, and let $\theta_g$ be the current global parameters, which every worker already holds. Let $G$ be the group of function-preserving transformations we handle: per-matrix scalings for weights before normalization, and per-head invertible maps for query/key and value/output pairs. Each worker computes

$$ g_m^\star = \arg\min_{g \in G} \lVert g\cdot\theta_m - \theta_g \rVert^2, \qquad \Delta_m = \theta_g - g_m^\star\cdot\theta_m , $$

where $g\cdot\theta$ applies the transformation and $\Delta_m$ is the aligned pseudo-gradient handed to the usual outer optimizer.

For scalings, the minimizer has a closed form. For the per-head maps, we first balance the pair and then solve an orthogonal Procrustes problem, which finds the rotation that best matches one matrix to another. This needs one $d_{\text{head}}\times d_{\text{head}}$ singular value decomposition per head, every $H$ steps. The cost is tiny next to methods that transform tensors every step, which matters given the compute-contention concern raised by Agarwal et al..

Relation to Prior Work

Two strands of prior work supply the method's lineage. From parsing, Eisner-style normal forms (Eisner, 1996) and proof nets (LCGbank; Bhargava & Penn) remove spurious ambiguity by fixing a canonical representative or an invariant structure. Shift-reduce parsing against a dependency model instead trains so that any path to the right meaning counts as correct. We borrow the canonical-representative idea, not their machinery. Our equivalence classes come from the network's algebraic symmetries rather than from grammar rules.

SEAG is the closest conceptual neighbor. It merges paraphrased reasoning steps using an entailment classifier and reports cutting inference calls to 31% of those used by the Reasoning-via-Planning search method. It motivates the general bet that collapsing equivalent branches saves compute. However, it operates at inference time and needs a learned equivalence detector, whereas our equivalences are exact and free to compute.

On the systems side, DiLoCo (Scaling Laws for DiLoCo) runs $H=30$ local steps and then applies outer Nesterov momentum to averaged weight differences. That is precisely the operation we modify; we keep its outer optimizer unchanged. WASH shuffles parameters between workers to keep them in one loss basin, and MERIT shows one-shot merging works only from a merge-ready flat basin. Both constrain us. They show that averaging fails when workers become functionally misaligned, but they treat misalignment as something to prevent by mixing or by careful initialization. We treat part of it as an exact, removable ambiguity.

Compression methods compress whatever update they are given. DeMo sends top-$k$ frequency coefficients of momentum, SEPARATE uses shared random low-rank projections, and BitDP sends 1–2 bit weights. None of them distinguishes function-changing from function-preserving content. Our alignment is complementary and can sit in front of any of them.

One honest caveat bounds the claim. The symmetry directions are low-dimensional: the per-head query–key map has $d_{\text{head}}^2$ degrees of freedom against $2\,d_{\text{model}}\,d_{\text{head}}$ parameters. Removing them cannot by itself shrink the payload much. Any gain must therefore come from better averaging and easier compression, not from deleting coordinates.

Research Program

The first stage is a diagnostic, and the project lives or dies on it. We train Chinchilla-style decoder transformers on C4 under DiLoCo with $M\in\{2,4,8\}$ workers and $H\in\{30,100,300\}$. At each synchronization we decompose every pseudo-gradient into its component tangent to the symmetry orbits and the remainder. We also measure what share of the variance across workers lies along the orbits. In addition, we measure the function-space error of naive averaging: the difference between the averaged $W_QW_K^\top$ and the average of the workers' $W_QW_K^\top$, along with the resulting change in held-out logits. We run this diagnostic with inner AdamW and with inner SGD with momentum. Our mechanism predicts the orbit share is substantially larger under Adam, and if that prediction fails, the story is wrong.

The second stage builds gauge-fixed DiLoCo as defined above and composes it with DeMo-style top-$k$ compression of the outer update. The third stage tests whether alignment shifts DiLoCo's trade-off between communication interval and loss. Specifically, we ask whether gauge-fixed DiLoCo at $H=100$ or $H=300$ matches standard DiLoCo at $H=30$.

Baselines

The naive reference point is standard data-parallel AdamW, synchronizing every step, at matched tokens and model size. This gives the loss ceiling and the bandwidth floor.

The strongest direct competitor is DiLoCo itself, with $H=30$ and outer Nesterov momentum (Scaling Laws for DiLoCo). It reports more than 80% compute utilization with 10x to 100x less bandwidth than data parallelism, and lower held-out C4 loss than data parallelism at 2.4B, 4B and 10B parameters with $M=2$. The file gives no exact loss values, so the absolute comparison number will come from our own reproduction. We run it on identical data order, worker count and token budget. The outer learning rate and momentum get the same sweep budget as our method's, and that sweep is retuned at every $H$. Beating DiLoCo means lower held-out loss at equal $H$, or equal loss at an $H$ at least three times larger.

The strongest compressed competitor is DeMo, which on OLMo-300M with Dolma v1.5 reports sending 7.5 MB versus 637 MB per step with no accuracy loss (DeMo). We compare DeMo against DeMo applied to aligned updates at equal $k$. Beating it means matching DeMo's HellaSwag, ARC-Easy and PIQA scores at a smaller $k$.

Two ablations isolate the key ingredient. The first removes a random subspace of the same dimension as the symmetry orbits. This shows whether any gain comes from the symmetry itself rather than from generic dimension reduction. The second applies the alignment after averaging rather than before. This tests our claim that the damage happens in the averaging step.

Why not just…?

Why not just synchronize more often or tune DiLoCo's outer optimizer? Smaller $H$ always reduces drift, but it spends exactly the bandwidth we are trying to save. Outer momentum can damp noisy directions, but it has no notion of which directions are function-preserving. The comparison of loss against $H$ under a matched tuning budget settles whether alignment buys anything that tuning cannot.

Why not just let DeMo's top-$k$ compression discard the drift? This is a real threat. If drift along the symmetry orbits is high-frequency and small, top-$k$ may already drop it. The stage-one diagnostic measures what fraction of the drift survives top-$k$. The DeMo-versus-aligned-DeMo comparison then decides the question; if the gain vanishes under compression, we report that.

Why not just use an architecture without these symmetries, such as normalizing queries and keys, or rely on weight decay to balance them? Query–key normalization removes only part of the symmetry. Weight decay balances the pairs only in the long run, while the misalignment we target builds up within $H$ steps. Both are cheap and plausible competitors. We therefore include a query–key-normalized model, trained with and without alignment, and report the remaining gain separately.

Why not just align permutations or shuffle weights as in WASH? Permutation mismatch arises when models train apart from different starting points or for very long stretches. Our workers share a start every $H$ steps, so continuous gauge drift is the more likely mismatch. A WASH arm at matched shuffle rate shows whether the two effects stack or overlap.

Decisive Checks

We run three experiments, in order of how decisively they test the idea. First, the diagnostic must show that orbit-tangent directions carry a non-trivial share of cross-worker variance under AdamW, more than under SGD, and growing with $H$. If the share is negligible, the project stops there. Second, on C4 at 150M and 1B parameters, gauge-fixed DiLoCo must beat tuned DiLoCo in held-out loss at $H=100$ and $H=300$, while the random-subspace ablation must not. Third, on OLMo-300M and 1B trained on Dolma v1.5, aligned DeMo must match DeMo's downstream HellaSwag, ARC-Easy and PIQA at smaller $k$. Throughout, we report alignment wall-clock as a fraction of total step time to confirm it stays negligible.