In federated learning, many devices train a shared model without sending their raw data anywhere. The strongest deployed privacy recipe has two parts. Each device adds a small integer-valued noise share to its model update. A cryptographic protocol, secure aggregation, then reveals only the sum of all updates to the server, and the noise shares add up to exactly the amount needed for a formal differential privacy guarantee. That guarantee is a worst-case bound, identical for every client. In practice, though, clients are not identical. A client whose data looks unlike everyone else's sends an update that points somewhere unusual and is usually larger. Two things follow. The fixed size limit (the clip) applied before noising distorts that client's update more, and attacks that unmix a summed gradient back into individual contributions find that client easiest to separate. At the moderate-to-large privacy budgets used in practice, the worst-case number hides a double penalty on atypical clients: they get worse accuracy and face higher real reconstruction risk.
This note proposes to measure that double penalty and to test a specific remedy. The remedy is personalized federated learning, meaning training methods that give each client a few private parameters, such as a local classifier head or local normalization layers. Our hypothesis is that these private parameters soak up exactly the client-specific signal that makes a client's shared update distinctive. If they do, they make the shared update both easier to clip without bias and harder to isolate, at the same formal privacy level and the same communication cost. The deliverable is a measured map of per-client leakage and per-client utility under distributed discrete noise and non-IID data (data whose distribution differs from client to client). It also includes a simple training order, "absorb, then clip," that would turn the hypothesis into a usable protocol.
Core Claim
Our central claim concerns the shared update that leaves each client. For client $i$, call it $g_i$, and call the round's average shared update $\bar g$. We define the client's distinctiveness as
$$\delta_i = 1 - \cos(g_i, \bar g),$$
which is near zero when the client's shared update points the same way as everyone else's and approaches one or more when it does not.
We claim that, under distributed discrete noise with a single global clip norm, a client's clipping distortion and its empirical reconstruction risk both rise with $\delta_i$. We also claim that, at fixed $(\epsilon, \delta)$, moving from plain federated averaging (FedAvg) to a body/head split lowers the spread of $\delta_i$ across clients and therefore narrows both gaps. The body/head split means a shared feature extractor plus local heads, as in FedRep, or local normalization layers, as in FedBN.
The prediction that would make this new is not that personalization improves accuracy, which is known. It is that personalization changes who is vulnerable inside a differentially private aggregate, and that this effect remains when the formal privacy accounting is held exactly equal.
How this relates to existing work
Three papers motivate the proposal. Disparate Impact in Differential Privacy from Gradient Misalignment shows that, in centralized private training, clipping bends the average update away from minority groups. It reports minority-class accuracy of 26.3% under standard private SGD versus 65.5% with its global-scaling fix on unbalanced MNIST. That fix needs a trusted curator who sees raw per-example gradients, which secure aggregation forbids.
Concealing Sensitive Samples against Gradient Leakage argues that inversion succeeds when individual contributions are weakly entangled, that is, when they lie in nearly separate directions. The Cocktail Party Attack exploits exactly this. It treats a summed fully connected layer gradient as a linear mixture and separates it with independent component analysis, recovering inputs from sums over as many as 1,024 samples. Neither paper asks how data heterogeneity across clients shapes that separability.
Other work supplies the mechanisms we reuse. Distributed noise shares that sum to a known distribution come from the Distributed Discrete Gaussian, the Skellam mechanism and the Poisson Binomial Mechanism. We use the last because its bounded support avoids clipping the noise itself, and that clipping would otherwise confound our clipping-bias measurements. Communication-efficient aggregation comes from Project-and-Perturb in The Fundamental Price of Secure Aggregation, and tight privacy accounting from the FFT accountant of Koskela et al.. These works analyze distributed mean estimation or synthetic, uniform data (PBM uses $n=1000$, $d=250$). None measures how leakage or clipping bias is distributed across heterogeneous clients. That distribution is our object of study.
One paper constrains the design. Breaking Secure Aggregation recovers labels with 100% accuracy by having a malicious server tamper with batch-normalization scale and shift parameters it ships to clients. FedBN-style local normalization removes that lever. This is a second, separate reason the personalized split could matter, and our experiments must keep it apart from the distinctiveness effect.
Research Program
The first component is measurement. We train FedAvg, FedBN and FedRep under PBM noise shares with secure aggregation simulated exactly over a modular integer ring. Every configuration is accounted to the same $(\epsilon, \delta)$ with the FFT accountant, uses the same clip norm selection procedure, and uses the same bits per coordinate. For every client in every round we log $\delta_i$, the pre-clip norm, and the clipping distortion $\lVert g_i - \mathrm{clip}(g_i)\rVert / \lVert g_i\rVert$. We then attack the noised, securely aggregated shared update and score how well each client's data is recovered. The attacks are the Cocktail Party Attack on the first shared fully connected layer, with the layer width set at least as large as the total number of samples mixed, to satisfy that attack's width constraint, and the generative-prior attack GIAS from Gradient Inversion with Generative Image Prior. Scoring uses each recovered image's best match: PSNR (pixel-level reconstruction quality), SSIM (structural similarity) and LPIPS (perceptual distance). We test the claim through the rank correlation between $\delta_i$ and per-client leakage, and through how far that correlation's slope and the spread of $\delta_i$ shrink under personalization.
The second component is a protocol, "absorb, then clip." Each round, the client first takes several steps on its private parameters (head or normalization layers) while the shared body is frozen. Only then does it compute the shared update, clip it, discretize it, and add its noise share. FedRep's alternating schedule already approximates this. We make the order explicit and vary how many absorption steps are taken. We set the clip norm privately each round with the federated quantile estimator of Differentially Private Federated Quantiles with the Distributed Discrete Gaussian Mechanism, charging its cost to the same privacy budget. This is the step that makes a small $\delta_i$ spread pay off: a lower clip quantile can be used without disproportionately crushing atypical clients, and that buys a better signal-to-noise ratio at equal $\epsilon$.
The third component is an active-attack check. We run the batch-normalization tampering attack against FedAvg and FedRep, which share normalization layers, and against FedBN, which keeps them local, each with and without noise shares. This separates the "removes the attacker's lever" benefit from the distinctiveness benefit.
Baselines
The naive reference point is FedAvg with secure aggregation and no noise. It shows how much leakage the summation alone permits and gives the utility ceiling. Prior evidence is that the summation permits a lot, through both independent-component unmixing and the tampering attack.
The second reference is FedAvg with PBM noise shares and a privately estimated clip quantile, the standard deployable distributed-privacy pipeline as of the Poisson Binomial Mechanism and Project-and-Perturb. It is the strongest existing method for this exact threat model, an untrusted server with formal privacy guarantees. We give it the same accountant, the same bits per coordinate, and a clip-quantile and learning-rate grid as large as the one given to our method. There is no reported result found on Lacuna for its accuracy under non-IID splits; the closest numbers in the file are communication figures, about 0.24 bits per parameter on F-EMNIST for Project-and-Perturb. We will therefore report it ourselves.
The strongest non-private personalized reference is FedRep. It reports about 87.7% versus 42.6% for FedAvg on CIFAR-10 with two classes per client. FedBN reports about 71% versus 62% on SVHN in the Digits benchmark. These numbers bound how much accuracy privacy costs each architecture.
The key ablation keeps the FedRep architecture but aggregates the head too, with noise. It has the same parameters and the same noise; only the absorption of client-specific signal into local parameters is removed. A second ablation varies the number of absorption steps from zero upward.
Beating the strongest baseline means two things at equal $(\epsilon, \delta)$ and bits. The worst-decile client accuracy must be higher by more than seed-to-seed variation. The leakage gap between the top and bottom $\delta_i$ quartiles, measured in LPIPS, must be smaller, and mean accuracy must not be worse than the noised FedRep without the private clip schedule.
Why not just…?
Why not just use secure aggregation without noise, since the server only sees a sum? Two results in the file say the sum is not enough. The Cocktail Party Attack separates sums of up to 1,024 samples, and the tampering attack recovers labels with 100% accuracy from summed gradients. Our no-noise baseline remeasures this on the heterogeneous splits. If atypical clients are recovered first, that directly supports the measurement claim.
Why not just tune the clip norm adaptively under FedAvg and skip personalization? This is the real threat to the utility half of the claim, since private quantile clipping alone might close much of the minority-accuracy gap. It cannot change $\delta_i$, though, because the shared update's direction is untouched. Our prediction is that it leaves the leakage gap intact. The FedAvg-plus-adaptive-clip baseline decides this. If it matches our method on both worst-decile accuracy and the leakage gap, the personalization claim is false and we will say so.
Why not just port DPSGD-Global-Adapt to the federated setting? Its global scaling needs the maximum update norm across contributors. Under secure aggregation that value can only be obtained through a further private estimate, which is what our quantile step already provides. We include a client-level version, scaling by the private norm quantile instead of clipping, as an extra arm so the comparison is not hypothetical.
Why not just add more noise? Lowering $\epsilon$ reduces every client's leakage, but the private-training literature in the file shows utility falling steeply as noise grows. The question here is the shape of the trade-off at fixed $\epsilon$. Our method has to deliver a strictly better per-client frontier, and we sweep $\epsilon$ to show whether it does.
Proposed Experiments
The experiments use four heterogeneity regimes. FEMNIST from LEAF gives natural differences between writers. CIFAR-10 with two classes per client gives label skew, matching FedRep's protocol. The Digits benchmark gives feature shift, matching FedBN's protocol. CelebA, grouped by attribute, lets us track named minority groups as in the disparate-impact study. We sweep $\epsilon$ from 1 to 16 with 100 to 1,000 clients per round, holding the bits per coordinate fixed. All differences are reported over five seeds.
Three checks are decisive. First, under noised FedAvg, the rank correlation between $\delta_i$ and per-client reconstruction quality should be clearly positive on at least three of the four regimes. If it is not, the double-penalty premise fails and the rest is moot. Second, the absorption-free ablation should recover most of FedAvg's leakage gap, while full FedRep with absorption should not. This attributes the effect to absorption rather than to having fewer shared parameters. Third, local normalization should block the tampering attack, while independent-component unmixing of the shared fully connected layer still succeeds on high-$\delta_i$ clients absent noise. That would show the two protective mechanisms are distinct.
A negative outcome would also be informative. If personalization lowers mean leakage but concentrates the remaining risk on the same atypical clients, the conclusion is that formal client-level guarantees need per-client empirical auditing in heterogeneous deployments. We would report that.
References
- Exploiting Shared Representations for Personalized Federated Learning
- FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
- Disparate Impact in Differential Privacy from Gradient Misalignment
- Concealing Sensitive Samples against Gradient Leakage in Federated Learning
- Cocktail Party Attack: Breaking Aggregation-Based Privacy in Federated Learning Using Independent Component Analysis
- The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
- The Skellam Mechanism for Differentially Private Federated Learning
- The Poisson Binomial Mechanism for Unbiased Federated Learning with Secure Aggregation
- The Fundamental Price of Secure Aggregation in Differentially Private Federated Learning
- Tight Differential Privacy for Discrete-Valued Mechanisms and for the Subsampled Gaussian Mechanism Using FFT
- Breaking Secure Aggregation: Label Leakage from Aggregated Gradients in Federated Learning
- Gradient Inversion with Generative Image Prior
- Differentially Private Federated Quantiles with the Distributed Discrete Gaussian Mechanism
- LEAF: A Benchmark for Federated Settings