Trading Style Statistics for Synchronization Rounds in Domain-Split Training

When a vision model is trained across machines that each hold data from a different environment, such as different camera fleets, hospitals or cities, and that data cannot be pooled, the machines' copies of the model pull apart between synchronizations. The standard response is to synchronize more often, which means paying for the expensive network traffic of shipping full weights or gradients. This note proposes a much cheaper fix. Each machine sends its peers a few hundred kilobytes of per-channel feature means and variances. These are the numbers that, in the style-transfer literature, encode "what the image looks like" (lighting, texture, color) as opposed to "what is in it." Each machine then trains its local copy on its own images re-dressed in the other machines' styles. The bet is that much of the drift between machines comes from differences in appearance, not content. If so, removing that difference should let machines run many more local steps between full synchronizations at the same final accuracy, including on an environment no machine trained on.

Two pieces of prior work set this up. On the communication side, DiLoCo shows a training scheme that works well when shards look alike. Each copy runs $H=30$ local steps, the copies average their weights, and an outer momentum step follows. The scheme matches ordinary per-step synchronization with 10x to 100x less bandwidth, but it was validated on homogeneous text corpora. On the representation side, AdaIN shows that swapping the per-channel mean and standard deviation of a feature map transfers appearance while preserving spatial content. AdvStyle shows that perturbing these statistics during training raises synthetic-to-real segmentation mIoU from 27.42% to 37.39%. Nobody has asked whether the statistics themselves are the right thing to communicate when the bottleneck is drift caused by domain skew.

Core Claim

In domain-partitioned local training, three things should hold if the bet is right. First, the disagreement between workers' updates should track the distance between their channel-wise feature statistics in the shallow layers. Second, injecting real peer statistics into each worker's forward pass should shrink that disagreement. Third, as a result, the number of local steps $H$ that can run between full weight averages at fixed accuracy should grow substantially, while the extra payload stays below 1% of a weight synchronization.

The key word is peer. The claim is not that style augmentation helps robustness, which is already known. The claim is that the specific styles held by other workers are what close the gap, and that random or adversarially synthesized styles do not close it as well. That second part is falsifiable, and the experiments are built around it.

Research Program

The method sits on top of DiLoCo. At each outer synchronization, every worker passes a small random subset of its local images through the current model. It records the per-instance channel mean $\mu$ and standard deviation $\sigma$ at a few early layers, the stem and first two residual stages, where appearance information concentrates. It then broadcasts a bank of $K$ such vectors alongside its weights.

During the next $H$ local steps, with probability $p$ per sample, a worker applies an AdaIN-style substitution at those layers:

$$\tilde{x} = \sigma_{\text{peer}} \frac{x - \mu(x)}{\sigma(x)} + \mu_{\text{peer}},$$

Here $x$ is the local feature map, $\mu(x)$ and $\sigma(x)$ are its own channel statistics over height and width, and $(\mu_{\text{peer}}, \sigma_{\text{peer}})$ is drawn from another worker's bank, optionally interpolated with the local statistics.

The payload is small. With $K=64$ vectors over layers totalling about 768 channels in a ResNet-50, the bank is roughly 400 KB per worker, against about 100 MB for the weights. Banks ride on the existing synchronization, so no extra communication round is added. Banks are stale by up to $H$ steps, and the program measures directly how much that matters. After averaging, BatchNorm running statistics are recomputed on a few local batches. WASH explicitly flags BatchNorm statistics as a weak point of weight averaging that it handled only crudely, so this step is held fixed across all methods to keep that confound out of the comparison.

The analytical component tests the mechanism. At the end of each round, we compute the cosine similarity and norm gap of workers' local updates (their pseudo-gradients). We compute the same quantities for the post-average weight drift, and correlate both with the per-layer Wasserstein-2 distance between workers' Gaussian-approximated channel statistics. FedGTST proves that transfer error is bounded by cross-client variance of parameter gradients and communicates only scalar gradient norms to control it. Our measurement asks a structurally sharper question: whether that variance decomposes into a part explained by appearance statistics, which our exchange should remove, and a residual tied to content. A weak correlation would undercut the premise before any accuracy numbers are consulted.

The regime is deliberately narrow. Workers hold domain-disjoint data that cannot be moved, and links are slow. In a datacenter with shuffleable data, the problem disappears and this method should be useless; we test that too.

Baselines

The naive reference is DiLoCo itself on domain-split workers, swept over $H \in \{10, 30, 100, 300\}$, with the same backbone, data, outer Nesterov optimizer and tuning budget for inner and outer learning rates as our method. This gives the bytes-versus-accuracy curve the proposal must beat. For comparison, the paper reports a 10x to 100x bandwidth reduction for DiLoCo, but only on language modeling. There is no reported result on Lacuna for DiLoCo on PACS, Office-Home or GTAV-to-Cityscapes, so we produce it. Per-step synchronous data parallelism is the quality ceiling at maximal communication cost. Beating the naive baseline means reaching, at some $H$, accuracy within 0.5 points of synchronous training while DiLoCo at the same $H$ falls measurably short. Equivalently, it means matching DiLoCo's accuracy at three or more times its $H$.

The strongest robustness competitor is DiLoCo with local AdvStyle on each worker. AdvStyle takes gradient-ascent steps on the channel statistics to synthesize hard styles. The paper reports ERM+AdvStyle at 65.4% average on PACS against 58.9% for ERM, and 37.39% mIoU from GTAV to real targets with ResNet-50. These are centralized numbers and serve as context, not targets in our setting. AdvStyle roughly doubles forward and backward passes, so we compare both at equal step count and at equal wall-clock compute. Beating it means matching or exceeding its held-out-domain accuracy at roughly half its compute.

We also run WASH with $p=0.001$ weight shuffling as the strongest competitor that fights drift through parameters rather than features. On homogeneous CIFAR-100 it reaches about 82% with a ResNet-18. Comparing it here at matched bytes tells us whether exchanging statistics beats exchanging a sliver of weights.

The ablation that isolates the key ingredient keeps everything identical but replaces peer statistics with three alternatives. The first is the worker's own statistics shuffled within its batch. The second is Gaussian perturbations of the local statistics with variance matched to the peer bank. The third is peer statistics from a domain not in training.

Why not just…?

Why not just synchronize more often, or compress the weights instead? That is the curve we compare against. Compressors such as DeMo achieve 85x reduction on OLMo-300M (7.5 MB versus 637 MB per step), and they are complementary to our method rather than rivals. However, compression reduces the cost of each round without reducing the drift that makes rounds necessary. Agarwal et al. also show that compression kernels can contend with backpropagation and eat into the savings. We report results for DeMo-compressed DiLoCo with and without peer statistics to show the gains stack.

Why not just augment styles locally, with random perturbation or AdvStyle, and skip the exchange? This is the real threat. If Gaussian-perturbed local statistics match peer statistics at equal $H$, the honest conclusion is that drift is fixed by style diversity of any kind, and the communication component is unnecessary. The ablation above settles it. Our prediction rests on a specific argument: random perturbations cover a generic neighborhood of each worker's own style, whereas peers' statistics sample exactly the directions along which workers disagree. AdvStyle's own comparison found random style augmentation inferior to targeted statistics, which is mild supporting evidence, not proof.

Why not just keep normalization layers local, or recompute their statistics at merge time? That repairs the merged model's normalization but does nothing about convolution weights that adapted to different input distributions during the $H$ local steps. WASH notes this failure point but leaves it open. Every method in our study gets merge-time recalibration, so any gain we report is beyond what recalibration provides.

Why not just share features once, as FedCGS does? FedCGS aggregates class-wise feature moments in one shot. With them it fits a Bayesian head over a frozen ResNet-18, transmitting about 267k parameters against 11.1M for a full FedAvg round. It cannot train the backbone, which is precisely where domain-specific drift lives. We report it as a frozen-backbone reference.

Proposed Experiments

For classification we use PACS, Office-Home and DomainNet, with one source domain per worker and one domain held out in leave-one-out fashion, on ResNet-18 and ResNet-50. For dense prediction, GTAV and SYNTHIA are split across workers, possibly further split by scene type. Evaluation is on Cityscapes, BDD-100K and Mapillary in mIoU. The metrics are held-out-domain and in-domain accuracy or mIoU plotted against total bytes communicated and against wall-clock time under a simulated 1 to 3 Gbit/s link. We also report the pseudo-gradient cosine across workers.

Three checks decide the proposal. The first is that the drift-versus-statistics-distance correlation is substantial in shallow layers and weak in deep ones. The second is that peer statistics beat matched-variance random statistics at large $H$, with the gap growing as $H$ grows. The third is that, under an IID split of the same data, the method yields no gain, which confirms that it targets domain-driven drift rather than acting as a generic regularizer. We also vary bank staleness by refreshing every round versus every fourth round, and vary the layers where statistics are injected, to locate where the effect comes from. Finally, because the statistics reveal each site's visual style, we report a simple membership attack on the banks so that deployers know what is being shared.