Choosing the Compression Level for Free in Private Federated Learning

Federated learning trains one model across many devices without collecting their data. When it is combined with differential privacy, a formal guarantee that the trained model reveals little about any single participant, three settings end up governing accuracy: how hard each update is compressed, the norm at which updates are clipped, and the learning rates. Each of these is usually found by grid search. In a private system, that search either spends privacy budget or is quietly left out of the privacy accounting, and the best setting moves with the privacy budget, with how different the clients' data are, and with the training round.

This note proposes removing the compression knob by computing it from outputs that are already public. The server receives noisy aggregates whose noise variance it knows exactly, so it can estimate, band by band, how much real signal sits above the noise. It then tells clients to transmit only the bands that are worth their noise. Because this rule only post-processes released outputs, it costs no extra privacy budget. Combined with private adaptive clipping applied after truncation, it should turn "compression boosts private federated learning" from a finding that depends on careful tuning into a default that works without it.

Core Claim

Prior work shows that shrinking updates before adding privacy noise can raise accuracy a lot. Compression Boosts Differentially Private Federated Learning (FL-CS-DP) projects updates onto low-frequency cosine-transform coefficients and reports 78% versus 61% accuracy on Fashion-MNIST at a 5% compression ratio. Constrained differentially private federated learning for low-bandwidth devices (FL-TOP) freezes all but a top-$K$ set chosen on public proxy data and reports 0.81 versus 0.56 at $r=0.5\%$. Both papers leave the compression ratio as a hand-set constant. FL-CS-DP explicitly names sensitivity to the compression ratio and clipping threshold as unsolved.

We claim three things. First, for orthonormal, structured truncation, the retained dimension can be chosen each round with zero added privacy cost. Second, this online choice matches an oracle-tuned fixed ratio across privacy budgets and heterogeneity levels, and beats a ratio tuned on proxy data once the private data drifts away from the proxy. Third, most of the "compression boost" comes from clipping the truncated vector rather than the full one, not from the reduced noise dimension alone. The third point matters because noise-dimension savings by themselves could be reproduced by denoising at the server with no compression at all.

The mechanism works as follows. Partition the transformed update coordinates into bands $j=1,\dots,J$ of sizes $n_j$, for example blocks of cosine-transform frequencies or, more coarsely, whole layers. At round $t$, with $m$ sampled clients, clipping norm $C_t$ and noise multiplier $\sigma$, the server holds the released noisy mean $\tilde u_j$ for every transmitted band. Each noisy band has expected squared norm equal to its signal energy plus $n_j\sigma^2C_t^2/m^2$, so the server keeps band $j$ in the next round if and only if

$$\hat s_j = \|\tilde u_j\|^2 - n_j \frac{\sigma^2 C_t^2}{m^2} > n_j \frac{\sigma^2 C_t^2}{m^2}.$$

This is the break-even point at which including a band lowers the expected squared error of the aggregate: its signal energy must exceed the noise energy it brings in. The threshold therefore comes from the error decomposition and is not tuned.

To detect when a band should be added back, clients also transmit one "guard band" just beyond the current cutoff. The server smooths $\hat s_j$ over rounds with a fixed exponential average, which is the only new constant.

The privacy argument is simple. The round-$t$ cutoff depends only on outputs from earlier rounds. Truncation onto orthonormal coordinates never increases an update's norm. Each round is therefore an ordinary subsampled Gaussian mechanism with sensitivity $C_t$, and the accounting is identical to DP-FedAvg at the same $\sigma$ and sampling rate. The clipping norm $C_t$ is adapted with the private quantile tracker of Differentially Private Learning with Adaptive Clipping, which tracks the norm of the truncated update and so shrinks automatically as fewer bands are kept. This is the data-driven analogue of the $\sqrt{s}C$ rescaling that DP-SparFL derives for random sparsification.

Relation to Prior Work

Two lines of work motivate the proposal by exposing the gap.

The first automates privacy hyperparameters but ignores compression. Towards hyperparameter-free optimization with differential privacy (HyFreeDP) removes the clipping norm and learning rate in centralized DP training. It spends a small slice of budget on privatized loss probes and does not treat federated drift or compression. DP-FedEXP adapts the server step size from update-diversity ratios, but it assumes dense updates. Fine-Tuning with Differential Privacy Necessitates an Additional Hyperparameter Search shows that which layers to train is itself a sensitive privacy hyperparameter: first-and-last-layer tuning reaches 77.9% versus 74.7% whole-model on CIFAR-100 at $\epsilon=2$. That paper also notes that the search is not charged to the budget. Our rule with layers as bands is an online, zero-cost replacement for that search.

The second line makes compression and privacy compatible but hard-codes the dimension. That covers FL-CS-DP and FL-TOP above, and Sketched Gaussian Mechanism for Private Federated Learning (Fed-SGM), which proves amplification scaling like $1/\sqrt{b}$ for dense Gaussian sketches. For Fed-SGM, the choice of $b$ becomes more consequential, not less.

Two further papers constrain the design. An improved analysis of per-sample and per-update clipping in federated learning shows that per-update clipping is safe for any threshold only if client step sizes shrink with heterogeneity. We therefore use per-update clipping and report sensitivity to the client step size rather than hiding it. A Practical Recipe for Federated Learning Under Statistical Heterogeneity documents the "client drift trap" from raising local epochs. This is why local epochs appear in our sensitivity grid.

The leakage literature sets the evaluation standard. Learning To Invert (LTI) trains an inversion network through the defense and breaks heuristic compression, reaching reconstruction MSE 0.014 under sign compression. This is why we rely on formal DP rather than on compression as a defense, unlike Mixed Quantization Enabled Federated Learning to Tackle Gradient Inversion Attacks. Do Gradient Inversion Attacks Make Federated Learning Unsafe? shows that released batch-normalization statistics leak, so our models use group normalization or keep those statistics local.

Research Program

We will build the method on the cosine-transform basis of FL-CS-DP with simple band truncation in place of basis-pursuit reconstruction, which removes the heavy server solve. A layer-band variant will also be built. The components are the guard-band cutoff rule, per-update clipping after truncation, and quantile-adaptive $C_t$. Client-side error feedback on dropped coefficients will be an optional switch, since the file flags error feedback as interacting badly with drift.

The analysis has two parts. The first is a bias-variance account of the rule under heterogeneity. Larger client disagreement lowers the energy of the mean update relative to individual norms, so the rule should keep fewer bands on more heterogeneous partitions. The second is a check that the accounting really is unchanged from DP-FedAvg.

Baselines

The naive reference is uncompressed DP-FedAvg with a fixed clipping norm, at the same $\epsilon$, client sampling rate and round count. Every method must beat it. Its well-known weakness is noise energy growing with the dimension $d$.

The strongest existing competitor on the compression question is FL-CS-DP with its best reported fixed ratio, alongside FL-TOP with proxy-selected coordinates. They are the strongest because they are the published methods that already obtain the compression boost under formal DP. On Fashion-MNIST, FL-CS-DP reports 78% at 5% compression versus 61% for standard DP, and FL-TOP reports 0.81 at $r=0.5\%$. We rerun both on identical partitions and budgets. We give them the same adaptive clipping so that the only difference is how the compression level is chosen. Their ratio is tuned in two ways: on a public proxy split (the fair setting), and by oracle search on private validation data (an optimistic ceiling whose privacy cost is ignored). Beating the proxy-tuned versions, and staying within one standard deviation of the oracle while using no tuning, counts as success.

Fed-SGM is run with $b$ swept over the same grid and the same budget. It is the main competitor that obtains its gain through amplification rather than truncation. Its own reported result is a noise-scale drop from 0.8 to 0.0883 at equal $\epsilon$ on EMNIST ByClass. The file reports no accuracy figure on Lacuna for Fed-SGM that we could cite.

For an automated but dense competitor, we use adaptive clipping plus DP-FedEXP server steps. DP-FedEXP reports 94.57% on MNIST under central DP. For adaptive clipping, the file gives only the "<1% more noise" overhead and no reported accuracy on Lacuna.

Three ablations isolate the ingredients. The first is dense DP aggregation followed by the same band-truncation rule applied at the server. This has the same noise-energy reduction but no clipping benefit, and it isolates the clip-after-truncation claim. The second is the rule with a fixed $C$, which isolates the adaptive clipping. The third is the rule without guard bands, which isolates exploration.

Why not just…?

Why not just tune a fixed compression ratio on public proxy data, as FL-TOP does? This is the real threat. If the best ratio barely moves across $\epsilon$, heterogeneity and rounds, a proxy-tuned constant is simpler and adequate. FL-TOP itself notes that selection degrades when private data departs from the proxy, but no number quantifies how much. Our first decisive experiment measures how far the oracle ratio shifts across Dirichlet $\alpha$ and $\epsilon$, and whether proxy tuning keeps up. If the shift is small and proxy tuning matches, we will report that compression-level tuning is a non-problem, and the contribution narrows to the clipping finding.

Why not just denoise the dense private aggregate at the server? Post-processing is free, so server-side truncation achieves the same noise-energy reduction at the same privacy cost, and compression would then contribute only bandwidth savings. The ablation above tests exactly this. Our hypothesis is that clipping the truncated vector at a lower adaptive norm yields most of the gain. If server-side denoising matches, we will say so. That would be a useful correction to the "compression boosts privacy" narrative.

Why not just use a hyperparameter-free dense method such as HyFreeDP or adaptive clipping? These methods fix the learning rate and clipping norm, but their noise energy still scales with $d$. Cattan et al.'s layer-selection gains show that dimension choice matters beyond the learning rate and clip norm. The dense adaptive baseline measures this directly.

Why not rely on compression itself as the privacy defense? LTI breaks sign compression and pruning. The mixed-quantization defense was tested only against optimization attacks. We include it in the leakage audit for completeness, and we expect it to fail.

Proposed Experiments

Datasets follow the file: Fashion-MNIST and the Premier Healthcare EHR task, if access allows, for direct comparison with FL-CS-DP and FL-TOP; CIFAR-10 under Dirichlet $\alpha\in\{0.1, 0.5, 100\}$ using the FedZoo-Bench protocol with client sampling in its recommended 0.1–0.4 range; EMNIST and StackOverflow next-word prediction from the adaptive-clipping suite and LEAF; and SST-2 with BERT-Base as in Fed-SGM. Budgets are $\epsilon\in\{1,2,4,8\}$.

The headline metric is sensitivity rather than peak accuracy. For each method we report the median and worst-case accuracy across a grid of client learning rates and local epochs, crossed with $\alpha$ and $\epsilon$. We also report uplink bytes.

There are four decisive checks. The first is the oracle-ratio shift test described above. The second is the server-denoising ablation. The third is whether the rule keeps fewer bands as $\alpha$ decreases, as the bias-variance account predicts. The fourth is a leakage audit. In that audit, LTI is trained through the complete pipeline, including the public cutoff sequence, at batch sizes 1 and 8. Reconstruction MSE and Image Identifiability Precision are compared against DP-FedAvg at matched $\epsilon$, and against undefended and mixed-quantization federated training, where LTI reaches 0.012 MSE even under $\sigma=0.1$ noise. This confirms that removing the tuning step does not open a side channel that the formal accounting misses.