In federated learning, many phones or edge devices each train a copy of a shared model on their own data, and a server periodically averages the copies. Two facts about this setting have been studied separately. First, layers disagree across clients at very different rates. The early feature-extracting layers stay similar across clients, while the final classifier drifts quickly, so recent methods synchronize early layers less often. Second, on the device itself, the memory bottleneck is the cache of intermediate activations kept for backpropagation. Centralized training can compress that cache to about 2 bits per value with little accuracy loss. This proposal joins the two facts. When a layer is averaged across $N$ clients only every $K_l$ local steps, random quantization noise in its gradients partly cancels. The disagreement caused by data heterogeneity does not cancel. It accumulates. So the right activation bit-width for a layer depends on how that layer is synchronized, not only on how noisy its gradient is.
The concrete payoff comes from where memory and drift sit in the network. In convolutional networks the early layers hold the largest activation maps. They are also the layers that differentiated-aggregation schemes already leave unsynchronized for long stretches. We propose a per-layer bit-allocation rule for client activation caches whose noise budget is set by the server's observed per-layer drift and synchronization interval. We then test whether it reaches a lower client memory footprint than centralized allocation rules at equal federated accuracy.
Core Claim
Take unbiased stochastic activation quantization of the kind used by ActNN. The error it contributes to the server-averaged weights of layer $l$ grows linearly in the number of local steps, while heterogeneity drift grows quadratically. To first order, the averaged update of layer $l$ after $K_l$ local steps with learning rate $\eta$ deviates from the ideal update by a drift term of size about $\eta^2 K_l^2 \zeta_l^2$ and a quantization term of size about $\eta^2 K_l \sigma_l^2(b_l)/N$. Here $\zeta_l^2$ is the per-step disagreement in layer $l$'s gradients across clients, and $\sigma_l^2(b_l)$ is the gradient variance that $b_l$-bit activation quantization adds. The quadratic growth comes from summing a systematic per-step bias. The linear growth comes from summing independent zero-mean noise, which is then averaged over $N$ clients.
Quantization only matters when its term is a non-negligible fraction $\epsilon$ of the drift the schedule already accepts. Each client therefore solves
$$ \min_{\{b_l\}} \sum_l M_l(b_l) \quad \text{subject to} \quad \sigma_l^2(b_l) \le \epsilon \, N K_l \, \hat\zeta_l^2 \ \text{ for every layer } l, $$
where $M_l(b_l)$ is the activation memory of layer $l$ at $b_l$ bits and $\hat\zeta_l^2$ is the server's running estimate of layer disagreement.
ActNN's allocation minimizes the total per-step gradient variance of a single learner. It ignores $N$, $K_l$ and $\zeta_l$ entirely, so our rule makes a different and falsifiable prediction. It should push long-horizon, low-drift, memory-heavy early layers to fewer bits than ActNN would give them. It should also hold precision in layers where $K_l \zeta_l^2$ is small. Which layers those are is an empirical question the experiments answer. The rule could favor either the classifier head (large drift, short interval) or the extractor (small drift, long interval).
Research Program
The pieces we reuse come from prior work. The disagreement estimate $\hat\zeta_l^2$ comes from the per-layer discrepancy that FedLAMA already computes at the server to set its two synchronization intervals. The intervals $K_l$ come either from FedLAMA or from the static extractor/head split of FedALS, which averages the head every $\tau$ steps and the extractor every $\alpha\tau$ steps. The per-bit variance $\sigma_l^2(b)$ comes from ActNN's closed-form expression, computed locally from a few mini-batches.
Each round the server broadcasts, per layer, the budget $\epsilon N K_l \hat\zeta_l^2$. Each client then solves the constrained allocation above as a small knapsack over its own memory limit. Clients with different RAM therefore choose different bit-widths under the same noise budget.
The work has three parts. The first is a measurement study that answers the open question the file raises and nobody has tested. Does aggressive activation compression in early layers, run for $\alpha\tau$ local steps, speed up client drift? Or does their cross-client stability absorb the noise? We will measure per-layer drift and quantization variance under matched schedules to check the linear-versus-quadratic scaling directly. The second part is the allocation rule itself. The third part is a stress test on the case where the first-order argument is most likely to fail: very small $N$ per round and non-convex early training, where noise may not average out.
Baselines
Naive reference. Each client runs FedAvg with standard FP16/BF16 automatic mixed precision, following Mixed Precision Training, and no activation compression. Its accuracy sets the ceiling. Its activation memory sets the denominator for all savings.
Schedule-only reference. FedLAMA or FedALS with automatic mixed precision and full activation caches. This fixes what the schedule alone buys. On non-IID CIFAR-10 with ResNet-20, FedLAMA reports 80.6% accuracy using 44% of FedAvg's communication budget. On two-class-per-client SVHN, FedALS reports 81.17% against FedAvg's 70.10%. Any compressed variant must stay within a stated margin of these numbers under our reimplementation.
Strongest competitor. Plain composition: the same FedLAMA or FedALS schedule, with every client running unmodified ActNN allocation at the same average bit budget. This is the real threat. ActNN already adapts bit-widths per layer and per sample, and it reports 76.9% versus 77.1% full precision for ResNet-50 on ImageNet with 2-bit activations. I found no reported result on Lacuna for ActNN inside federated training. We will therefore run it ourselves with the same data partitions, schedules, rounds and hyperparameter search budget as our method. We also include DIVISION, which caches a low-frequency part of activations in BF16 and the remainder in 2 bits (75.9% versus 76.15% on ImageNet), as a second composed baseline.
What counts as beating these. Our method wins if it reaches equal final accuracy (within 0.5 points) at lower peak activation memory. Alternatively, it wins if it gives higher accuracy at a matched memory budget, holding across at least three seeds.
Ablations. Two ablations isolate the key ingredient. The first uses our budget without $K_l$ and $N$, which reduces it to drift-weighted ActNN. The second applies our allocation under uniform FedAvg synchronization, where all $K_l$ are equal. If the gain disappears when $K_l$ is removed, the coupling between synchronization interval and precision is what matters. We also run an inverted allocation, giving more bits where the rule gives fewer, to confirm that the direction of the effect is causal.
Why not just…?
Why not just run ActNN on each client alongside FedLAMA? This is the composed baseline above, and it may well be competitive. ActNN's objective is per-step variance for one learner. It cannot know that noise in a layer averaged every $\alpha\tau$ steps across many clients is diluted. It therefore over-spends bits on exactly the early layers that dominate activation memory. If the measurement study shows that quantization error does not scale as $K_l/N$ relative to drift, composition is sufficient. We would then report that as the finding and drop the method.
Why not just use uniform low precision, such as FP16 or a flat 4 bits? WRPN and the DIVISION ResNet-50 measurement (about 20.5GB of activations at large batch before compression) show that halving precision still leaves activations dominant. Flat low bit-widths ignore the unequal sensitivity across layers that ActNN documents. A flat-bit sweep matched on memory is included, so this claim is tested rather than assumed.
Why not just use gradient checkpointing, which adds no noise? Recomputation trades memory for extra forward passes. On battery-limited clients this slows the stragglers that already set round time. FedHybrid treats recomputation, compression and swapping as per-tensor options chosen by memory saved per second, under uniform aggregation. We will report wall-clock round time against checkpointing at equal memory. If checkpointing is not slower on our hardware, the case for compression rests on memory alone, and we will say so.
Relation to Prior Work
Prior work plays three roles here. FedLAMA, FedALS and FedImpro motivate the proposal: they show that drift concentrates in late layers and that early layers tolerate long horizons. None of them touches the client activation cache. FedLAMA's pairing with FedPAQ, and FedLP-Q, quantize the updates sent over the network, not the memory used during backpropagation. They are orthogonal and could be stacked. FedHybrid is the closest systems work: it compresses client activations channel-wise, but uses conventional round-based synchronization and no signal from layer drift. ActNN supplies the variance model and the quantizer that we reuse. DIVISION and the low-rank projections of INSTANT and LANCE are alternative compressors that the allocation rule could drive in a follow-up.
Proposed Experiments
The main settings follow the cited federated papers:
- CIFAR-10 under Dirichlet $\alpha=0.1$ with ResNet-20, as in FedLAMA.
- SVHN with two classes per client and five clients, as in FedALS. This is the small-$N$ stress case.
- CIFAR-100 under Dirichlet $\alpha=0.1$ with WideResNet28-10, where activation memory is large.
- FEMNIST partitioned by writer, for natural heterogeneity.
- ResNet-50 on non-IID CIFAR-10, to connect to INSTANT's activation-heavy setting.
Metrics are:
- final and time-to-target accuracy;
- peak client activation memory, measured rather than estimated;
- bits per activation, broken down by layer;
- communicated parameters;
- wall-clock time per round.
Three checks are decisive:
- Scaling check. Varying $K_l$ and $N$ while logging per-layer drift and quantization error should show the linear-versus-quadratic separation. If it does not, the method's premise fails.
- Head-to-head. At matched memory, the method should beat composed FedLAMA+ActNN on CIFAR-100 and CIFAR-10. The gain should come mainly from the early-layer bits.
- Ablation collapse. Removing $K_l$ from the budget should erase most of the gain. The inverted allocation should hurt accuracy.
References
- ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
- Layer-Wise Adaptive Model Aggregation for Scalable Federated Learning
- Differentiated Aggregation to Improve Generalization in Federated Learning
- Mixed Precision Training
- WRPN: Wide Reduced-Precision Networks
- FedHybrid: Breaking the Memory Wall of Federated Learning via Hybrid Tensor Management
- FedImpro: Measuring and Improving Client Update in Federated Learning
- Towards Efficient Federated Learning: Layer-Wise Pruning-Quantization Scheme and Coding Design
- INSTANT: Compressing Gradients and Activations for Resource-Efficient Training
- LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learning