Emergent Outer Step Sizes for Low-Communication Training of Bias-Free Networks

Low-communication training methods let several machines train separate copies of a model for many steps and only occasionally average them. This removes most of the network traffic that ordinary data-parallel training spends synchronizing gradients. Averaging has a side effect, though. When copies have drifted in different directions, the average of their weights is shorter than any one copy. In a bias-free network, which has every additive constant removed and uses only ReLU-type activations, scaling the weights down shrinks the output by a predictable multiplicative factor, and that factor compounds with depth. We propose to treat the step taken after each averaging round, the "outer" step, as something to measure from the loss rather than a tuned constant. We split it into two parts. The first is an output-scale correction, solved in closed form from one forward pass and a few numbers exchanged between machines. The second is a one-dimensional curvature-sensing line search along the averaged update, also synchronized with a handful of numbers.

The practical bet is that the averaging artifact is a large part of what the hand-tuned outer learning rate has been compensating for. If so, removing it analytically should let machines run longer between synchronizations at the same final quality. That would reduce communication further and remove a hyperparameter sweep. Bias-free networks are where this bet is cleanest to test, because in them a weight-norm loss is close to a pure output rescaling. They are also where scale-heterogeneous data across machines is a natural and well-motivated setting.

Core Claim

DiLoCo (Distributed Low Communication), studied at scale in Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo, works as follows. Each of $W$ copies runs $H$ local steps, typically $H=30$. The copies are averaged, and an outer optimizer applies SGD with Nesterov momentum to the "pseudo-gradient", the difference between the starting weights and the averaged weights. That paper finds a constant outer learning rate transfers across model sizes. The research file does not report how that rate should depend on $H$ or on how differently the copies' data are distributed.

Our claim has two parts. First, in bias-free networks, much of the loss increase caused by averaging can be removed by rescaling the output with a single scalar $s$. Second, the remaining outer step can be chosen by a quadratic line search along the pseudo-gradient. Both quantities can be estimated with scalar all-reduces, meaning every machine contributes a few numbers and receives their sum. The extra compute is two or three forward passes per outer round. Spread over $H$ inner steps, that is a few percent.

The scale correction has a closed form for squared-error tasks such as denoising. Let $f(y_i)$ be the averaged model's output on worker-local input $y_i$, and let $x_i$ be the target. The correction is

$$s^\star = \frac{\sum_{w}\sum_{i \in \mathcal{B}_w} \langle f(y_i), x_i\rangle}{\sum_{w}\sum_{i\in\mathcal{B}_w} \|f(y_i)\|^2},$$

where $\mathcal{B}_w$ is worker $w$'s evaluation batch. Each worker therefore sends exactly two numbers.

For cross-entropy classification, the loss as a function of a logit multiplier $s$ is convex in one variable. A few Newton iterations solve it, each exchanging a first and a second derivative computed on cached logits. The tangential step then follows the generalized-Newton recipe of Gradient descent with generalized Newton's method. That method fits a parabola to losses evaluated at a few trial step lengths. Here we fit it along the pseudo-gradient $\Delta$, after the scale correction has been applied:

$$\eta^\star = \arg\min_\eta \; \hat{q}(\eta), \qquad \hat{q}\ \text{fit to}\ \ell(\bar\theta_s + \eta\Delta)\ \text{at}\ \eta \in \{0, \pm\eta_0\}.$$

Here $\bar\theta_s$ is the averaged model with $s^\star$ absorbed into its weights, and $\ell$ is the loss summed across workers through a three-number all-reduce.

The bias-free structure matters in where $s^\star$ is absorbed. In a plain bias-free chain, multiplying each of the $L$ layers by $(s^\star)^{1/L}$ reproduces the scaled output exactly while keeping layer norms balanced. Putting all of it into the last layer gives the same function with a different parameterization. In bias-free residual networks only the last-layer option is exact, so we use that there.

Why this, and not the obvious combination

The obvious idea at this intersection is to run an emergent step-size method inside each worker of a low-communication scheme. The research file flags this as hazardous. Each worker adapts its step to its own mini-batch curvature, so with non-identical data the copies travel different distances before averaging. Whether line searches should instead operate on the outer pseudo-gradient is listed as unaddressed.

The classic parallel line-search analysis in Incremental and Parallel Machine Learning Algorithms With Automated Learning Rate Adjustments lets workers pick admissible steps locally. It predates outer-optimizer schemes and does not consider averaging artifacts. DES-LOC, in DES-LOC: Desynced Low Communication Adaptive Optimizers for Foundation Models, cuts communication up to 170x relative to data-parallel training by synchronizing weights, first moments and second moments on separate fixed cadences. Its cadences are hand-picked and its models contain biases. DeMo: Decoupled Momentum Optimization compresses momentum with a cosine transform and top-$k$ selection. It is orthogonal to this proposal and still relies on a manual warmup-plus-cosine schedule.

None of these works estimates the outer step from the loss. None separates the scale component of averaging from its directional component. Inner-loop step-size methods (Locally Optimal Descent for Dynamic Stepsize Scheduling, GeN, and the parabolic search in Parabolic Approximation Line Search for DNNs) supply our tangential estimator. We apply that estimator once per round rather than every step, which also sidesteps their known overhead and small-batch noise problems.

Role of the bias-free literature

Two lines of work motivate the architectural regime. Robust And Interpretable Blind Image Denoising Via Bias-Free Convolutional Neural Networks shows that removing all additive constants makes a denoiser locally linear in its input. A bias-free denoiser trained on noise levels $\sigma \in [0,10]$ generalizes to $\sigma=90$, where the biased version collapses. This makes bias-free denoisers the natural model when different machines hold data at different noise levels, a realistic non-IID split that is hard for averaging. Fast Axiomatic Attribution for Neural Networks shows that bias removal on ImageNet ResNet-50 costs little: top-5 accuracy moves from 92.56% to 91.12%. So the regime is not a toy.

Two theoretical papers constrain the design. Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect proves for matrix factorization that large steps push factors toward balanced, flatter minima. Greedy step choices may therefore miss this effect. This is why we test balanced versus last-layer absorption of $s^\star$, a direct and cheap way to impose balance without relying on large steps. The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks shows the optimizer's geometry shapes the solution of homogeneous networks under decaying schedules. We therefore keep the inner optimizer fixed across all arms, so that only the outer rule varies.

Research Program

The first stage is diagnostic and could falsify the premise on its own. On bias-free and standard versions of the same architectures, we run DiLoCo with varying $H$ and varying data heterogeneity. At each round we measure how much of the loss gap between the averaged model and the mean of the copies is closed by the optimal scalar $s^\star$ alone. Two predictions follow. In plain bias-free chains, most of the gap should be scale. The compounding product of per-layer norm ratios should predict $s^\star$. In biased networks the scalar should explain less of the gap. If the scalar explains little even in bias-free networks, the decomposition is pointless and we stop.

The second stage builds the outer rule: scale correction, then the parabolic tangential step, then optional Nesterov momentum on the corrected step. We study its stability across $H$ and heterogeneity.

The third stage asks the communication question. At a fixed quality target, how large can $H$ be, and how many bytes must cross the network?

Baselines

The naive reference point is synchronous data-parallel training with full-precision all-reduce every step and a tuned warmup-plus-cosine schedule. It defines the quality ceiling and the communication cost we aim to undercut. For bias-free ResNet-50 on ImageNet, the single-machine reference is the 91.12% top-5 reported in Fast Axiomatic Attribution for Neural Networks.

The strongest existing competitor is DiLoCo as described in Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo, with a constant outer Nesterov learning rate. That paper reports that DiLoCo with $H=30$ matches data-parallel quality on language models up to 10B parameters. There is no reported result found on Lacuna for DiLoCo on CIFAR, ImageNet or BSDS denoising, so we establish it ourselves. Fairness requires giving it its strongest form. We grid-search its outer learning rate separately for every combination of $H$ and heterogeneity, with the same inner optimizer, data, worker count and total inner steps as our method. Our method receives no outer tuning. Beating DiLoCo means one of two outcomes at matched final accuracy or PSNR (peak signal-to-noise ratio, the standard denoising quality measure). Either our rule tolerates a larger $H$ and so communicates fewer bytes, or at matched $H$ it matches the per-setting-tuned DiLoCo without any sweep. Merely matching DiLoCo tuned once, at one setting, would be a weak result, and we would report it as such.

Inner-loop emergent stepping is represented by DiLoCo whose workers run GeN, in two variants. In the first, local step sizes are left free. In the second, the scalars entering GeN's quadratic fit are all-reduced so all workers share one step. The synchronized variant is cheap and architecture-agnostic, and it is the most natural competitor to our design. We also compare against DES-LOC with its published cadence ratios, measured by bytes to reach the target.

Three ablations isolate our ingredients. The first is the tangential line search without the scale correction, which isolates $s^\star$. The second is the scale correction absorbed into the last layer versus spread evenly across layers, which isolates the balancing question. The third runs the identical method on biased architectures, which isolates whether bias-freeness matters at all.

Why not just…?

Why not just tune the outer learning rate once, since DiLoCo found it transfers across model sizes? This is the most serious threat. The scaling-law result concerns model size, not the number of local steps or worker heterogeneity. Our hypothesis is that the optimal outer step drifts with both, because norm shrinkage under averaging grows as copies disagree. The per-setting sweep in our baseline tests this directly. If a single tuned constant is near-optimal across all $H$ and heterogeneity levels, the scale correction is not needed, and we would report that the contribution reduces to removing one sweep.

Why not just synchronize the inner step sizes of GeN or GLyDER with a scalar all-reduce? That fixes the divergence of local step sizes for any architecture at essentially no communication cost. It still leaves the averaged model's scale loss uncorrected, and it pays curvature-sensing forward passes every few steps rather than once per round. The synchronized-GeN baseline settles the question. If it matches our method at large $H$, then averaging artifacts are not the bottleneck, and our contribution shrinks to cheaper overhead.

Why not just add a learnable output temperature and let training fix the scale? A trainable scalar is updated by the inner optimizer at the same small rate as everything else, and averaging shrinks the weights feeding it on every round. The first-stage diagnostic shows whether the shrinkage is repaired by local training within a few steps. If it is, the closed-form correction adds nothing. We include the learnable-temperature arm alongside the scale-correction ablation.

Why not just use normalization layers and weight decay to make scale irrelevant? Normalization with learned offsets breaks the homogeneity that gives bias-free networks their robustness to input scale. For the scale-heterogeneous denoising setting, that robustness is the reason to use these networks. The biased-architecture ablation, which includes batch normalization, shows what the standard design gains or loses under the same outer rule.

Proposed Experiments

Classification uses CIFAR-10 and CIFAR-100 with a bias-free plain VGG-style chain, where balanced absorption is exact, and a bias-free Wide-ResNet. We then scale to ImageNet with bias-free ResNet-50. Each runs at 8 to 16 workers with $H \in \{30, 100, 300\}$ and label-skewed splits of increasing severity.

Denoising uses BSDS training and Set68 evaluation with bias-free DnCNN, following Robust And Interpretable Blind Image Denoising Via Bias-Free Convolutional Neural Networks. Each worker sees only its own narrow noise band. Evaluation covers PSNR on held-out noise levels, including levels outside every worker's band. No dB values for this setup are reported on Lacuna, so all numbers are measured.

Metrics are final accuracy or PSNR, total bytes communicated to reach the target, wall-clock time under simulated bandwidths from fast interconnect to commodity Ethernet, and the extra forward passes per round.

Two checks are decisive. The first is the fraction of the averaging gap explained by $s^\star$. It must be large in bias-free networks and visibly smaller in biased ones, or the architectural link fails. The second is the largest $H$ at which each method still matches synchronous quality. Our rule must exceed per-setting-tuned DiLoCo and synchronized-GeN DiLoCo on this measure, or the communication claim fails.

Language models are deliberately excluded. Bias-free homogeneity has not been extended to attention-based Transformers, as the denoising paper itself notes. Making that extension work is a separate project.