In federated learning, many phones or hospitals train one shared model without pooling their data, and the cost that dominates is sending model updates over the network. The standard way to shrink an update is to keep only its largest entries ("Top-$k$") or to round them to a few bits. Both methods treat every weight as equally meaningful. That assumption fails in a specific way in networks with normalization layers. These are layers such as batch normalization (BN) and group normalization (GN), which rescale each channel's activations to a fixed mean and variance. A convolutional filter followed by such a layer produces the same output whether its weights are $w$ or $3w$. Part of every update therefore changes nothing the model computes. The part that does matter has an effect that grows as the filter's weight norm shrinks. A third, small group of parameters, the normalization layer's own scale and shift, controls entire feature maps at once. Corrupting even a handful of them can collapse accuracy, as Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption shows. A compressor that ignores this structure spends bits on movements the network erases. It also under-protects the movements the network amplifies.
We propose to compress federated updates in coordinates that respect this structure. Each client splits the update to every normalized filter into two parts. The first is a "radial" part that only changes the filter's length, which the normalization layer cancels. The second is a "tangential" part that changes the filter's direction. The client sends the radial part as one number per channel. It ranks the tangential entries by their actual effect on the output, which can be computed in closed form from the weight norm with no calibration data. It sends the normalization scale and shift parameters losslessly. The server then reassembles each filter on the sphere of directions instead of averaging raw weights. This also addresses the "loudest voice" problem, where clients with larger weight norms dominate the average. The claim to test is that this split recovers most of the accuracy lost to aggressive compression at a 1% budget. It should do so at no data cost, and it should explain why output-aware selection beats magnitude-based selection in normalized convolutional networks.
Core Claim
Let $w_j$ be the weight vector of output channel $j$ of a layer whose output feeds directly into a BN or GN layer. Let $\Delta_j$ be the client's accumulated local update for that channel. Because the normalized output is invariant to positive rescaling of $w_j$, we decompose
$$\Delta_j = \underbrace{\frac{\Delta_j^\top w_j}{\|w_j\|^2}\, w_j}_{\Delta_j^{\parallel}} + \Delta_j^{\perp},$$
where $\Delta_j^{\parallel}$ is the radial component and $\Delta_j^{\perp}$ is the component orthogonal to $w_j$. To first order, the change in the normalized output of channel $j$ is proportional to $\|\Delta_j^{\perp}\|/\|w_j\|$. This follows from differentiating $w/\|w\|$, whose Jacobian is the projection orthogonal to $w$, scaled by $1/\|w\|$. The radial component contributes nothing.
In a single gradient step the gradient is already orthogonal to $w_j$. Federated updates, however, accumulate many local steps with weight decay and momentum. Weight decay is itself a purely radial shrink. Accumulated local updates therefore carry a radial part that magnitude-based Top-$k$ pays for with no change in what the model computes. Our selection score for entry $i$ of channel $j$ is $|\Delta^{\perp}_{j,i}|/\|w_j\|$. This is a data-free estimate of the output discrepancy that Enhancing Communication Compression via Discrepancy-aware Calibration for Federated Learning measures empirically with 32–64 local samples.
The new element is not the geometry of scale-invariant weights. It is the use of that geometry to decide which bits a federated client sends and how the server aggregates them. None of the papers in this file does this.
Relation to Prior Work
Two lines of work motivate the proposal. First, Enhancing Communication Compression via Discrepancy-aware Calibration for Federated Learning shows that update magnitude decouples from output impact under multi-step local training. They fix this by measuring each unit's output discrepancy on local calibration data. The authors do not isolate the role of normalization in that decoupling. We conjecture that in normalized CNNs, much of the decoupling is exactly the radial-versus-tangential split and the $1/\|w_j\|$ factor. If so, it can be removed analytically, without data or extra forward passes.
Second, Understanding Normalization Layers for Sparse Training proves that BN and LN amplify gradient norms in proportion to neuron sparsity. Their analysis is centralized and concerns sparse weights, but it rests on the same inverse-norm factor. That makes it the natural theoretical anchor for why compression error should be weighted by $1/\|w_j\|$.
Parameter-corruption work constrains the design. Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption finds that well-chosen corruption of only 10 parameters drops a ResNet-34 from 72.5% to near 0%. ReaLM: Reliable and Efficient Large Language Model Inference with Statistical Algorithm-Based Fault Tolerance and Improving Deep Neural Network Reliability via Transient-Fault-Aware Design and Training show that one bad value entering a normalization statistic pollutes a whole vector or batch. Compression error is a structured perturbation of exactly this kind. These results say the scale/shift parameters and any transmitted running statistics must not be quantized carelessly. The results also suggest that a lossy compressor could cause the same collapse those papers induce on purpose.
The normalization-in-FL papers supply the setting and the parts we reuse. Making Batch Normalization Great in Federated Deep Learning (FIXBN: freeze global statistics midway through training) and Overcoming the Challenges of Batch Normalization in Federated Learning (FBN: normalize with shared global statistics and correct variance aggregation with the law of total variance) make BN viable under non-IID data. Both assume uncompressed communication, and we adopt FBN's statistic handling as our BN substrate. Rethinking Normalization Methods in Federated Learning identifies the loudest-voice effect of unequal weight norms in averaging. Our server-side reassembly on the sphere is a direct attempt to neutralize it.
The opposite philosophy is to remove normalization entirely. FedWon: Triumphing Multi-domain Federated Learning Without Normalization does this with weight standardization and adaptive gradient clipping but sends full-precision updates. FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantization pairs weight standardization with 1-bit non-uniform quantization. FedWSQ is the closest existing system: it also uses weight geometry to make compression safer. It differs from our proposal in three ways. It standardizes gradients rather than exploiting the invariance that a downstream normalization layer creates. It uses quantization rather than sparse selection. It does not single out the high-leverage normalization parameters.
Research Program
The work has three parts.
Part 1: diagnostic. Run FedAvg with a BN- or GN-equipped ResNet-18 on Dirichlet-split CIFAR-10. Log each client's accumulated update and decompose it per channel. For each round, measure what fraction of update energy is radial, how much of that comes from weight decay versus momentum, and how much output discrepancy each component causes on held-out client data. This tests the premise directly. If the radial fraction is negligible, the proposal reduces to a reweighting of Top-$k$ and should be judged as that.
Part 2: compressor. On the client, compute $\Delta^{\perp}$ and send the radial coefficient as one scalar per channel. Select tangential entries by $|\Delta^{\perp}_{j,i}|/\|w_j\|$ under a global budget, with error feedback for unsent residuals. Send BN/GN scale and shift parameters, and FBN statistics where used, at full precision; every byte counts against the budget. On the server, rebuild each client's channel as a direction plus a norm, average directions weighted by data size, and set the norm by averaging radial coefficients. Layers not followed by a normalization layer (the classifier head and residual-stream projections) fall back to plain error-feedback Top-$k$.
Part 3: boundary analysis. In pre-LN Vision Transformers, few weights feed directly into a normalization layer, so exact scale invariance mostly does not hold. Here the method should give little benefit over Top-$k$, and data-driven calibration should win. Showing this boundary is part of the claim, not a failure of it.
Communication will be accounted as bytes on the wire, including indices and per-channel scalars. Experiments run in the Flower simulation runtime described in Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment. That work reports that serialization and non-training overhead often exceed training time, so payload format matters.
Baselines
The naive reference points are uncompressed FedAvg and FBN on the same backbone, which give the accuracy ceiling. For FBN, Overcoming the Challenges of Batch Normalization in Federated Learning reports 80% on extreme-skew CIFAR-10, versus 75% for FixBN and about 17% for naive BN. The second reference is magnitude Top-$k$ with error feedback at the same byte budget, which gives the floor.
The strongest competitor on our exact question is discrepancy-aware calibration from Enhancing Communication Compression via Discrepancy-aware Calibration for Federated Learning. It already targets output impact rather than magnitude. Its reported result is 29.62% test accuracy versus 21.03% for magnitude Top-$k$, on CIFAR-10 with ViT-tiny at a 0.01 compression ratio. No reported result was found on Lacuna for this method on BN or GN ResNets. We will run it ourselves with the authors' 64-sample calibration on the same splits, backbone, round count and byte budget, and we will report its extra client forward-pass compute. Beating it means matching or exceeding its accuracy at equal bytes in normalized CNNs while using no calibration data. Equal accuracy at lower client compute would be a partial win; we will report it as such and not claim an accuracy gain.
For the quantization regime, the strongest competitor is FedWSQ. It reports 45.11% on Tiny-ImageNet at 1 bit with Dirichlet $\alpha=0.1$, more than 7 points above FedRCL and 5 above FedACG. We compare at matched bits per parameter, with our tangential entries quantized to 1 bit instead of sparsified.
We also run the normalizer-free alternative: FedWon's architecture compressed with Top-$k$ and with FedWSQ's quantizer. This combination has not been reported.
Every method gets the same learning-rate and weight-decay grid of fixed size, tuned on a validation split of the same clients. FedWon receives its own grid because normalizer-free nets are known to be learning-rate sensitive.
Ablations isolate each ingredient. The first uses only full-precision scale/shift on top of Top-$k$. The second uses only radial removal. The third uses only $1/\|w_j\|$ weighting. The fourth uses only server-side reassembly on the sphere. The fifth runs the full method with weight decay set to zero, which should shrink the radial share and therefore the gain.
Why not just…?
Why not just exempt the normalization parameters from compression? It is cheap and it addresses the fragility documented by Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption. We include it as the first ablation. It does nothing about bits wasted on radial movement, nor about the $1/\|w_j\|$ amplification of filter-direction error. If it alone recovers most of the gap to the full method, that is the honest finding. The proposal would then shrink to a recommendation and a diagnostic, and we will report it that way.
Why not just use discrepancy-aware calibration, which already measures output impact? This is the real threat. Measured discrepancy should implicitly capture scale invariance, so in normalized CNNs the two rankings may largely agree. The experiment that settles it is the rank correlation between our closed-form scores and Wan et al.'s measured discrepancies, alongside the accuracy comparison. If the correlation is high, our contribution is an explanation plus removal of the calibration data and compute. That matters for clients with tiny or batch-size-1 data, where a 32–64 sample batch is unrepresentative and its BN statistics are themselves biased. If the correlation is low and calibration wins on accuracy, the closed form is missing something, most likely cross-layer effects. We would then test a hybrid that uses the analytic score as a prior.
Why not just remove normalization, as FedWon does, and compress that? FedWon shows that removing normalization can beat FedBN on unseen domains (67.8% vs 61.0%). With no normalization there is also no fragile statistic to corrupt. But FedWon has never been run under compression, and its weight standardization makes each convolution invariant to scale in its own way. Our method may therefore transfer to it rather than compete with it, and we test this by applying the tangential split to standardized weights. If compressed FedWon matches our method on BN networks at equal bytes, the conclusion is that removing normalization dominates exploiting it, and we will say so.
Why not just swap BN for GN or LN? Making Batch Normalization Great in Federated Deep Learning reports that GN reaches 59.69% versus 76.56% for FIXBN on extreme non-IID CIFAR-10, so the swap costs accuracy even before compression. GN is still scale-invariant per group, so we run our method on GN networks too. The question there is whether compression narrows or widens the GN–BN gap.
Decisive Checks
The benchmarks are CIFAR-10 and CIFAR-100 with Dirichlet $\alpha \in \{0.1, 0.5\}$ over 100 clients, Tiny-ImageNet at $\alpha=0.1$ for the FedWSQ comparison, and Office-Caltech-10 and DomainNet for the FedWon comparison. Backbones are ResNet-18 with BN (FBN statistics) and with GN, FedWon's normalizer-free ResNet, and ViT-tiny as the boundary case. Compression budgets are 0.1, 0.01 and 1 bit per parameter. Metrics are final accuracy, rounds to a target accuracy, and total bytes transmitted.
Four outcomes would decide the question. First, Part 1 must show a non-trivial radial share of update energy with near-zero output effect. Without that, the mechanism is absent. Second, at a 0.01 budget on BN/GN ResNets, the full method must beat error-feedback Top-$k$ and match or beat discrepancy calibration at zero calibration data. The zero-weight-decay ablation must lose part of the gain, which ties the effect to the predicted source. Third, on ViT-tiny the gain over Top-$k$ should largely disappear, confirming that the effect comes from exact scale invariance and not from generic reweighting. Fourth, a fragility probe: we will inject random bit flips into transmitted payloads to model noisy uplinks. Lossless scale/shift transmission should prevent the abrupt collapses that the parameter-corruption and fault-propagation literature predicts, while flips in tangential entries should degrade accuracy gradually.
References
- Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption
- Enhancing Communication Compression via Discrepancy-aware Calibration for Federated Learning
- Understanding Normalization Layers for Sparse Training
- ReaLM: Reliable and Efficient Large Language Model Inference with Statistical Algorithm-Based Fault Tolerance
- Improving Deep Neural Network Reliability via Transient-Fault-Aware Design and Training
- Making Batch Normalization Great in Federated Deep Learning
- Overcoming the Challenges of Batch Normalization in Federated Learning
- Rethinking Normalization Methods in Federated Learning
- FedWon: Triumphing Multi-domain Federated Learning Without Normalization
- FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantization
- Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment