Choosing What to Send, Privately: Server-Chosen Sparse Masks with Online Sparsity Control for Differentially Private Federated Learning

Federated learning trains a shared model while phones or hospitals keep their data local. Each client sends only a model update to a server. Those updates are expensive to transmit, and they are also dangerous: an attacker holding an update can often reconstruct the images or text behind it. Two tools address these problems together. Sparsification sends only a small subset of coordinates, which cuts bandwidth. Differential privacy (DP) clips each client's update to a maximum length and adds calibrated Gaussian noise, which bounds what any single client can reveal. Combining them helps in a specific way: noise is added only to the coordinates that are actually sent, so a well-chosen small subset carries more signal per unit of noise. The catch lies in "well-chosen." Picking the coordinates a client finds most important (its largest-magnitude entries) leaks which coordinates those are. Picking them at random is safe but wasteful. And the right number of coordinates depends on the privacy budget, the dataset, and the stage of training, so finding it by trial and error spends privacy budget itself.

We propose that the server choose the coordinate subset, or mask, from its own previous noisy aggregates. Because those aggregates are already DP outputs, this choice costs no additional privacy. The server also adjusts the mask size every round with a controller driven by one cheap private statistic: what fraction of each client's clipped update falls inside masks of a few candidate sizes. This yields a biased, top-magnitude-style sparsifier that needs neither a public proxy dataset nor multiple non-colluding servers, and whose sparsity level adapts within a single training run without a privacy-consuming search. We then test directly whether the resulting sparse, noisy updates resist modern generative-prior inversion attacks, including when the server picks masks maliciously.

Core Claim

The prior work leaves a specific opening. Fed-SMP sparsifies before adding noise under client-level DP. It shows that top-k selection beats random selection and that an optimal compression ratio exists. However, it picks top-k indices using a server-side public proxy dataset and fixes the ratio for the whole run. DP-SparFL uses only random sparsification, precisely to avoid index leakage, and rescales a static clipping bound by $\sqrt{s}$. Clover hides client-chosen top-k indices behind a three-server secret-sharing protocol that fails if two servers collude. None of these picks indices from information the server already legitimately holds, and none adapts the sparsity level against the DP noise online.

Our claim is that three privacy-free signals are enough to recover most of top-k's utility advantage without its index leakage. The first is the previous noisy aggregates, which can be smoothed with momentum. The second is the known noise variance. The third is a small DP query about how concentrated client updates are.

The controller picks mask size $k_t$ in round $t$ to minimize an estimate of the error in the aggregated update:

$$ \hat{E}_t(k) = \big(1 - \hat{\rho}_t(k)\big)\,\hat{G}_t^2 + \frac{k\,\sigma^2 C_t^2}{m^2}. $$

In this expression, $\hat{\rho}_t(k)$ is the privately estimated average fraction of a client's clipped-update energy that lies inside the size-$k$ mask. $\hat{G}_t^2$ is the squared norm of the mean clipped update, estimated from the previous noisy aggregate by subtracting the known noise variance. $\sigma$ is the noise multiplier, $C_t$ is the current clipping bound, and $m$ is the number of participating clients.

The first term is the signal lost to masking. The second term is the noise injected into the $k$ transmitted coordinates, since Gaussian noise of standard deviation $\sigma C_t$ is added per coordinate and averaged over $m$ clients. This objective makes the regime dependence formalized by Farokhi something the algorithm tracks rather than something practitioners guess. That paper finds that sparsification helps at small privacy budgets and hurts at large ones. Under our objective, when the budget is large, $\sigma$ is small and the controller should grow $k$ toward dense training. When the budget is tight, it should shrink $k$.

Research Program

The mechanism has four parts.

The first part is the mask. The server keeps an exponential moving average of past noisy aggregates, which is pure post-processing. It takes the top $(1-\beta)k_t$ coordinates of that average and adds $\beta k_t$ uniformly random coordinates. The random share avoids lock-in on stale coordinates and covers early training, when the average is mostly noise. A useful property is that when noise dominates, the top-k of the average is effectively random, so the method degrades gracefully toward random sparsification.

The second part is concentration estimation. Each sampled client computes its masked update, clips it to $C_t$, and reports a short vector of energy fractions $\rho_i(k_j)$ over a geometric ladder of about five candidate sizes. The client clips this vector coordinatewise to $[0,1]$ and privatizes it with Gaussian noise. We account for it jointly with the main mechanism using Rényi DP, following the joint analysis that Andrew et al. use for their clipping bit. Our target is to keep this query under a few percent of total budget, which we will verify rather than assume.

The third part is clipping. The clipping bound $C_t$ follows Andrew et al.'s private quantile update, but it is applied to the norm of the masked update rather than the dense one. That paper leaves open how sparsification changes quantile dynamics. The research file flags the same question as poorly understood, noting that compression reshapes the norm distribution. We will measure whether the optimal target quantile shifts as $k_t$ changes. We will also derive the $\sqrt{k/d}$-style scaling that DP-SparFL uses for random masks, and check whether it breaks for top-k-style masks, whose captured energy is far above $k/d$.

The fourth part is accounting. Because the mask is a function of past DP outputs, per-round sensitivity is exactly $C_t$ on the transmitted coordinates, and standard subsampled-Gaussian accounting applies. We give up the extra privacy amplification that random per-client sparsification earns in Privacy Amplification via Compression and DP-SparFL. Whether better coordinate choice outweighs that lost amplification is the central empirical question, not a foregone conclusion.

We treat gradient leakage as the evaluation lens, with no separate defense added. Deep Leakage from Gradients found that pruning more than 20% of gradients stopped its attack. GIAS reconstructs recognizable images with 99% of gradient values removed. Sparsity alone is therefore no defense, and we make no claim that it is. What we test is whether the formal DP guarantee at practical budgets suppresses generative inversion on the sparse updates our method produces. We also test whether a malicious server can exploit its control over the mask. A server could concentrate masks on first-layer weights, which are the most revealing for inversion.

Baselines

The naive reference is dense DP-FedAvg with a hand-tuned static clipping bound. It is the starting point in Fed-SMP's comparisons, and every sparse method must beat it at equal privacy budget.

The strongest existing competitor is Fed-SMP with top-k indices chosen from a public proxy dataset and a static ratio tuned per task. It targets exactly our accuracy-at-fixed-budget question with a biased sparsifier. Its reported result is "up to 11% higher accuracy than DP-FedAvg" on Fashion-MNIST, while saving 99% of communication at comparable accuracy on Fashion-MNIST and SVHN. We will reimplement it on identical client partitions, models, rounds, cohort sizes, and accountant. We will give it the best of a grid over compression ratios, and we will report how much privacy budget that grid would have consumed had it been run on private data. We run it with two kinds of proxy data: an aligned proxy drawn from the same distribution, and a mismatched proxy drawn from a different dataset. Beating it means matching its accuracy with an aligned proxy while requiring no proxy and no sweep, and exceeding it with a mismatched proxy.

The random-sparsification competitor is DP-SparFL's random masking with $\sqrt{s}C$ clipping and amplification-aware accounting. Its scheduling machinery is removed so the comparison isolates the sparsifier. It is the strongest option if amplification matters more than coordinate choice. No reported numerical accuracy was found on Lacuna for DP-SparFL, so we compare against our own reimplementation.

For the adaptive-clipping-only line, we run Andrew et al.'s dense adaptive clipping. No reported numerical result was found on Lacuna for it either.

For the cryptographic line, we report Clover's roughly 75% CIFAR-10 accuracy under DP only as a reference point. Its privacy model differs from ours, since it relies on three non-colluding servers.

Four ablations isolate the key ingredients. The first swaps the noisy-aggregate mask for a random mask at the same adaptive $k_t$. The second freezes $k_t$ at the controller's average value, which tests whether adaptivity matters beyond finding a good constant. The third drops the random exploration share. The fourth adds client-side error feedback, which carries unsent residuals forward to the next round.

Why not just…?

Why not just tune the sparsity ratio once on public data and compose it with adaptive clipping? This is the most serious threat. If the optimal ratio transfers well across datasets and budgets, a fixed well-chosen ratio plus Andrew et al.'s clipping may match our controller. Farokhi's regime result and Fed-SMP's observed interior optimum both suggest the optimum moves with the budget, but neither shows it moves within a run. The frozen-$k_t$ ablation settles this. If it matches the adaptive controller at every budget we test, we will report the controller as unnecessary and keep only the proxy-free mask as the contribution.

Why not just use random sparsification with its privacy amplification? Amplification reduces noise for free. For low-concentration updates, random masking could beat any deterministic mask. DP-SparFL and Fed-SMP's random variant are our direct comparison, and we sweep budgets because we expect the answer to depend on the regime. We expect random masks to win at very small budgets and early in training, where the mask we select is itself noise. We expect our mask to win once the aggregate stabilizes.

Why not just use a public proxy dataset, as Fed-SMP does? When a well-aligned proxy exists, it may choose better coordinates than a noisy aggregate. Our honest expectation is parity in that case. The case for our method rests on non-IID cross-device data, such as FEMNIST writers and Shakespeare roles, where no aligned proxy exists. The mismatched-proxy runs quantify that case.

Why not just quantize instead? Mixed quantization drops DLG reconstruction PSNR from 55.3 to 8.1 and keeps 96.67% MNIST accuracy. However, it gives no formal DP guarantee, was not tested against generative attacks, and caps savings near 4×. We include 8-bit quantization of our masked payload as an orthogonal add-on rather than a rival.

Proposed Experiments

For utility, we use FEMNIST and Shakespeare from LEAF, StackOverflow next-word prediction as used by Andrew et al., and Dirichlet non-IID CIFAR-10 and Fashion-MNIST/SVHN matching Fed-SMP. We run client-level budgets of $\epsilon \in \{1, 2, 4, 8\}$ at fixed $\delta$. We report accuracy at a fixed budget, total uplink bytes to target accuracy, the trajectory of $k_t$, and the share of budget spent on the concentration query. The decisive utility check is that our method beats random masking at moderate budgets, and matches or beats Fed-SMP when its proxy is aligned, without that method's budget-consuming ratio sweep. We also expect the controller to push $k_t$ toward dense as $\epsilon$ grows. Failure to show that trend would undercut the error model.

For leakage, we attack updates from the trained systems with DLG and GIAS on CIFAR-10. DLG's reported undefended baseline is MSE of about 0.0069 on CIFAR-100, and GIAS reports about 15.58 dB PSNR on ImageNet with no defense. We also use the diffusion-prior user-level attack of DiffULA on CelebA, measuring PSNR, SSIM, and face detection rate. We attack in two settings: an honest server, and a malicious server that places the entire mask on the first layer. The decisive check is whether reconstructions stay near chance at budgets where utility is acceptable. If the malicious mask raises reconstruction quality meaningfully at a fixed budget, we will report that server-chosen masks require a per-layer cap on mask share. We will then add that cap and re-measure.