Federated learning lets hospitals, phones, or companies train one model together without pooling raw data. Each participant (a "client") trains locally and sends only model updates to a coordinating server. Two findings pull against each other. Collaboration helps the smallest participants most: in multi-hospital studies, the sites with the least data gain the most accuracy from joining. But the updates those same small participants send are the easiest to invert back into their private records. Gradient inversion attacks rebuild training inputs from a shared update, and they work best on small batches, with a good image prior, and early in training. Current defenses, whether differential privacy (adding calibrated noise so no single record changes the output much), pruning, or subspace substitution, are applied at the same strength to every client. They are judged by average accuracy. Nobody asks whether the privacy tax lands on the very clients who had the most to gain.
This note proposes a simple accounting and a mechanism built on it. A client's gain from federation can be split into two parts. The first is what it receives: what the shared model learned from everyone else. The second is what it gets back from sending its own updates, which pull the shared model toward its data. Leakage comes only from sending. We hypothesize that the split depends on the kind of heterogeneity. When clients differ mainly in which labels they hold, a small client's gain is almost all received, so it can send little or nothing and keep most of the benefit. When clients differ in what their inputs look like (a different scanner, a different handwriting style), the shared features do not cover the small client unless it contributes. If this holds, each client's privacy spending should follow its own received/sent split, not a single global setting.
Core Claim
Uniform privacy defenses are badly calibrated under non-IID data (data whose distribution differs from client to client) for a measurable reason. They charge every client for sending, including clients whose collaborative gain does not depend on sending. A tiered scheme can do better on both privacy and equity at equal average accuracy. In that scheme, a client sends nothing, sends a small noised projection, or sends a full defended update, according to its measured sent-gain. We make the claim testable by writing a client's gain over local-only training as
$$G_i = \underbrace{A_i(\text{global model trained without } i,\ \text{then locally adapted}) - A_i(\text{local only})}_{\text{received gain } R_i} + \underbrace{A_i(\text{full federation}) - A_i(\text{global model trained without } i,\ \text{then locally adapted})}_{\text{sent gain } S_i},$$
where $A_i$ is client $i$'s accuracy on its own held-out data. The prediction is that for small clients $S_i/G_i$ is near zero under label skew and large under feature skew. It should also be large for any client whose inputs lie outside what the other clients' data covers.
Relation to Prior Work
The motivating evidence comes from real-world collaboration studies. Federated Learning for Breast Density Classification ran unprotected federated averaging across seven mammography sites. It found local kappa rose from 0.64 to 0.68 and cross-site kappa from 0.18 to 0.26, and it names reconstruction attacks as the open risk. FedMAP found that site gain correlates inversely with standalone accuracy, with gains up to 14.3% for weak CPRD regions, and it left differential privacy to future work. Neither study asks where the gain comes from, which is the question we ask.
The threat model comes from inversion work. Gradient Inversion with Generative Image Prior searches a generator's latent space (GIAS, about 15.58 dB PSNR against 13.78 dB for pixel attacks). This keeps reconstructions useful at batch size 32 and even after 99% gradient pruning. That paper constrains our design: we cannot rely on batch size or sparsity to protect small clients. Evaluating Gradient Inversion Attacks and Defenses shows that withholding BatchNorm statistics and labels weakens attacks, so we evaluate under those realistic restrictions as well as permissive ones. CENSOR reports that leakage peaks in early rounds, which we use for tier scheduling.
The closest method is FedDPA. It keeps high-Fisher-information parameters local so that DP noise does not touch them, and it adapts the clipping constraint. Its per-parameter decision is the same for every client, it is never tested against inversion, and it reports only average accuracy. Our decision is per-client, driven by measured sent-gain, and evaluated per client on both leakage and gain.
The idea of shrinking the noised dimension comes from two papers. FL-TOP noises only the Top-K coordinates, chosen with public data, and reaches 0.81 on Fashion-MNIST against 0.56 for full DP. DP adapters reach 91.82% by privatizing 2.46% of parameters, where full DP-SGD falls to about 53%. We keep the small-subspace principle but drop the public-data requirement, as described below. FedRep supplies the model shape: a shared feature extractor plus private per-client classification heads. Its authors explicitly leave open what the shared-extractor updates leak. We answer that for each tier.
Research Program
The first component is a measurement protocol for $R_i$ and $S_i$. For each client, and for groups of clients by size, we train the federation with that client excluded. We then fine-tune the resulting shared extractor locally with a fresh head. This leave-one-out step is expensive and is used only for analysis. In deployment we replace it with a cheap proxy computed by the client alone: the drop in local validation accuracy when its own contribution is withheld for a window of rounds. We check the proxy against the exact decomposition.
The second component is the tiered sending scheme on a FedRep-style model. Tier 0 clients only receive the shared extractor and train their head locally. They send nothing, so the rest of the system learns nothing about their data. Tier 1 clients project their extractor update onto a rank-$k$ subspace, then clip and noise it with Gaussian DP. The subspace is taken from the top singular directions of recent aggregated updates, which the server already releases under DP or secure aggregation. Because the subspace is computed only from outputs that are already private, choosing it costs nothing extra in privacy and needs no public dataset, unlike FL-TOP. Tier 2 clients send full defended extractor updates, using DP-FedAvg-style noising, optionally combined with CENSOR.
Clients choose a tier from the proxy $S_i$. Every client starts in Tier 1 during the early high-leakage rounds, and may move to Tier 0 or Tier 2 after a warm-up period. There is an honest tension here. Under feature skew, the directions a small client needs may be exactly the ones other clients' updates do not span, so a projection onto their subspace may throw away the useful signal. The experiments measure this directly rather than assuming it away. If it happens, those clients fall back to Tier 2.
The third component is a per-client attack evaluation. We run GIAS, a pixel-space inversion, and the mGAN-AI client-identity attack from Beyond Inferring Class Representatives against each client's sent messages. We report the best-of-attacks PSNR and LPIPS, alongside the formal $\epsilon$ where one applies.
Baselines
The naive reference points are local-only training, which has no leakage but also no collaborative gain, and undefended FedAvg or FedRep, which have full gain and full leakage. The undefended FedRep result gives the ceiling for gain: FedRep reports about 87.7% on CIFAR-10 with 2 classes per client, against 42.6% for FedAvg (FedRep).
The strongest privacy-utility competitor is FedDPA, because it already targets personalized DP under non-IID data. Its reported results are 49.75% on CIFAR-10 at Dirichlet $\alpha=1$, 77.27% on FEMNIST at $\epsilon=4$, and 62.63% on SVHN at $\epsilon=4$ (FedDPA). We rerun it with our partitions, model, round count, and a matched hyperparameter search budget, and compare at equal $\epsilon$ for all clients that send. Beating it means two things together. At equal or better average accuracy, the bottom size-quartile of clients must keep more of their gain $G_i$, and their best-of-attacks reconstruction quality must be lower.
The strongest heuristic inversion defense is CENSOR, which reports about 86% CIFAR-10 accuracy, matching its undefended baseline. We apply it uniformly and compare per-client leakage at equal accuracy.
Two ablations isolate the key ingredient. The first applies the same tier mix at random rather than by measured $S_i$. The second replaces the subspace built from aggregated updates with a random rank-$k$ subspace. If random tiering does as well, the received/sent accounting adds nothing.
Why not just…?
Why not just have small clients never send, since they gain the most from receiving? This is Tier 0 applied to everyone, and it is a real threat. Under pure label skew it may capture almost everything. If so, that is a useful negative result, and the tiered scheme reduces to it. The decomposition exists to show where this rule fails. We expect it to fail under feature skew, for example FEMNIST writers or corrupted-style CIFAR-10 clients, where $S_i$ should be large. The experiments report Tier-0-for-all as its own baseline in each heterogeneity regime.
Why not just use secure aggregation? It hides individual updates from the server, and mGAN-AI's authors agree it blocks their attack. But the file notes that secure aggregation breaks the server-side inspection that clustering and personalization rely on, gives no protection against collusion, and handles client dropouts poorly. Our scheme is compatible with it. We run it both with and without secure aggregation, and we note that Tier 0 needs no cryptography at all.
Why not just use per-client adaptive clipping or FedDPA's parameter masking? Both change how each client sends. Neither asks whether a client needs to send. If FedDPA already matches our small-client gain at lower leakage, the proposal fails. The head-to-head comparison with FedDPA is built to settle this.
Why not just use larger batches or pruning? GIAS already reconstructs images at batch size 32 and after 99% pruning (Jeon et al.), and small sites often cannot form large batches. We include batch size 32 plus 90% pruning as a cheap baseline, not as a candidate solution.
Proposed Experiments and Decisive Checks
For label skew we use CIFAR-10 with Dirichlet partitions at $\alpha \in \{0.1, 1\}$ and 2-class splits, with client sizes varying 20× so that clear small clients exist. For natural feature skew we use FEMNIST. As a controlled feature-skew case we give each CIFAR-10 client a fixed corruption style, and we flag that this is synthetic. To check natural heterogeneity beyond vision, we use FedCVD (FedCVD), for which no reported result was found on Lacuna. SVHN supports direct comparison with FedDPA's numbers. Privacy budgets are $\epsilon \in \{1, 4, 8\}$. Metrics are per-client $G_i$, $R_i$, and $S_i$; accuracy of the worst quartile and the average; best-of-attacks PSNR/LPIPS per client; and formal $\epsilon$.
The first decisive check tests the hypothesis itself. Under label skew, the median $S_i/G_i$ for small clients should fall below 0.2, and under feature skew it should rise above 0.5. If the ratio does not change with the type of heterogeneity, the case for tiering collapses.
The second check tests the proxy: the cheap $S_i$ estimate must rank clients consistently with the leave-one-out decomposition, which we measure as Spearman correlation.
The third check is the main comparison. At equal $\epsilon$ and within one point of FedDPA's average accuracy, the tiered scheme must raise worst-quartile gain retention and lower worst-quartile reconstruction PSNR, and it must beat the random-tier ablation.
The fourth check tests Tier 1 under feature skew, reporting how much of $S_i$ survives projection onto the subspace built from other clients' aggregated updates. This tells us whether the no-public-data subspace is a real contribution or only works when heterogeneity is mild.
References
- Federated Learning for Breast Density Classification: A Real-World Implementation
- FedMAP: Unlocking Potential in Personalized Federated Learning through Bi-Level MAP Optimization
- Gradient Inversion with Generative Image Prior
- Evaluating Gradient Inversion Attacks and Defenses in Federated Learning
- CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Sampling
- Dynamic Personalized Federated Learning with Adaptive Differential Privacy
- Constrained differentially private federated learning for low-bandwidth devices
- Differentially Private Adapters for Parameter Efficient Acoustic Modeling
- Exploiting Shared Representations for Personalized Federated Learning
- Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning
- FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Data