Does Personalization Move the Leak? Auditing Decoupled Federated Learning Under Differential Privacy

Federated learning trains a shared model across many devices or institutions without collecting their data. Each client sends only model updates, meaning gradients or weight differences. Two facts complicate this picture. First, clients' data differ, so many methods now let each client keep part of the model private (a "personalized" head or adapter) and share only the rest. Second, shared gradients are not harmless: an attacker who sees them can often reconstruct training images or the client's label distribution. The current answer is differential privacy (DP), which clips each update and adds calibrated noise so that any one person's influence is provably bounded. Recent methods combine the two ideas. They add DP noise only to the shared part and leave the private part noise-free. This buys personalization accuracy at a fixed privacy budget $\epsilon$.

This proposal tests an assumption hidden in that design. The assumption is that keeping a component private reduces what the shared component reveals. Backpropagation suggests the opposite may happen. The private head sits on top of the shared layers, and it is fit to the client's skewed labels. Every gradient sent for the shared layers therefore passes through it, and may carry the client's label skew more sharply than a generic model would. We propose to measure, at matched formal $\epsilon$, how leakage of labels, images, and client identity changes as a client personalizes more. We also test one cheap architectural change that cuts this coupling. In short, the personalization–generalization knob may also be a privacy knob, and formal $\epsilon$ at the budgets people actually use may not show it.

Core Claim

At the loose but common budgets reported for personalized federated learning, such as $\epsilon = 4$, the DP bound allows a wide range of actual leakage. We claim that where a method lands in that range depends on how strongly it personalizes. Two predictions compete. In the first, the private head absorbs client-specific information, so shared updates look more generic and leak less; this is the implicit hope behind noise-free private modules. In the second, gradients reach the shared layers through a head tuned to local skew, so the label histogram is written more clearly into them; this is the mechanism Building Gradient Bridges exploits.

To make the second prediction concrete, write the shared-layer gradient for one sample as $g = J^\top \nabla_z \ell(h(z), y)$. Here $z$ is the feature produced by the shared layers, $J$ is the Jacobian of $z$ with respect to the shared weights, $h$ is the prediction head, $y$ is the label, and $\ell$ is the loss. Whatever head $h$ is used during local training determines which label-dependent signal lands in $g$.

The constructive part of the claim is simple. During local training, compute the shared layers' gradient through a globally shared head. The personalized head is trained on detached features, so its gradient never flows into the shared layers. Then add client-side clipping and noise to the shared update only. We expect this to give a better three-way frontier than current methods. The three axes are personalized accuracy, accuracy on unseen clients, and measured leakage. If neither prediction dominates and leakage stays flat across the personalization sweep, that is also a useful result. It would show that the free-noise private module is safe in practice, not just in the accounting.

What prior work settles and what it leaves open

Two methods supply the designs we audit. Dynamic Personalized Federated Learning with Adaptive Differential Privacy (FedDPA) keeps parameters with high Fisher information local and noise-free. It adds DP noise to the rest, with a penalty that keeps updates near the clipping threshold. It reports 77.27% on FEMNIST at $\epsilon = 4$ against 74.23% for BLUR+LUS. SDFLoRA does the analogous split for language-model adapters, putting noise only into the shared low-rank module. Neither paper runs a reconstruction or label-recovery attack on the shared updates. Both evaluate privacy only through $(\epsilon, \delta)$. Our work keeps their architectures and asks what an attacker actually recovers.

On Bridging Generic and Personalized Federated Learning (FED-ROD) supplies the structural idea we reuse. It trains a generic head with a class-balanced loss next to a personalized head. It reports 48.5% generic and 62.5% personalized accuracy on CIFAR-100 with Dirichlet 0.3 partitioning, without any privacy mechanism. We borrow its two-head layout, but our purpose differs: we use the generic head to stop label-skewed gradients from reaching the shared layers. We also avoid its reliance on sharing class-frequency vectors, since those are themselves sensitive.

The attack literature motivates the study and constrains its design. Gradient Bridges shows that withholding the classifier does not protect labels. Its closed-form attack recovers over 80% of labels from early-layer gradients at batch sizes up to 512. The paper studies split architectures without DP, so we do not know how this leakage behaves once personalization and formal noise are combined. Gradient Inversion with Generative Image Prior (GIAS) shows that a generative prior defeats large batches, reconstructing batches of 32 about as well as earlier attacks managed with batches of 4. This rules out "use bigger batches" as a defense. Beyond Inferring Class Representatives shows that a server watching one client's updates over time can extract identity-specific traits. That result makes client re-identification a natural third leakage measure in personalized settings.

Evaluating Gradient Inversion Attacks and Defenses warns that many attacks quietly rely on batch-normalization statistics. We therefore fix and report whether such statistics are shared. On the utility side, PLRV-O and Differentially Private Adapters document how full-model Gaussian DP collapses at tight budgets. Their results justify noising only a small shared subspace, not the whole model.

Research Program

The first component is a controlled personalization sweep. On a fixed backbone, we vary how much of the model stays private. The settings run from nothing (DP-FedAvg), to the classifier head only, to head plus last block, and finally to FedDPA's Fisher-selected mask at several thresholds. Clipping and noise on the shared part are calibrated so that every setting reaches the same user-level $\epsilon$ under the same accountant. Utility is measured two ways. Personalized accuracy is measured on each client's own test split. Generalization is measured on held-out clients that receive only the shared model plus a freshly fit head.

The second component is the attack suite, run against the transmitted shared updates. It has three parts. The first is Gradient Bridges label-distribution recovery, scored as label-recovery accuracy. The second is cosine-similarity gradient inversion and GIAS image reconstruction, scored with PSNR, SSIM, and LPIPS. The third is a client-linkage attack: a classifier trained to tell which client produced a given update, scored with AUC. We run each attack under two noise placements. In local noise, the client perturbs its update before sending it. In central noise, the server perturbs the aggregate, so an honest-but-curious server still sees clean individual updates. We check which placement each audited method actually uses, because this alone may decide whether its $\epsilon$ means anything against the server.

The third component is the intervention, which we call routed shared gradients. The shared layers receive gradients only through a global head that is aggregated like the shared layers. The personalized head trains on detached features. This removes the dependence of the shared gradient on the private head while keeping the private head free of noise.

The fourth component is a short accounting check. Personalized clients keep a hidden state that depends on their data. Sample-level DP analyses that noise only the shared part may therefore be unsound. Changing one sample changes the private head, and through it every later shared gradient from that client, so the per-round sensitivity is no longer simply the clipping norm $C$. User-level accounting avoids this problem. We will state when each kind of guarantee holds and report only guarantees that are valid.

Baselines

The naive reference points are two extremes. Local-only training shares nothing, so it leaks nothing but gives no help to unseen clients. Undefended FedAvg with a fine-tuned personal head sets the leakage ceiling. DP-FedAvg with client-side Gaussian noise over the full model is the uniform-noise baseline. It runs at the same $\epsilon$ and the same number of rounds, and its clipping norm is tuned over the same grid. In FedDPA's report, DP-FedSAM reaches 45.07% on CIFAR-10 at $\alpha = 1$, below FedDPA's 49.75%.

The strongest competitor is FedDPA. It is the only method in the file that targets both personalization and formal DP, and it reports 77.27% on FEMNIST and 49.75% on CIFAR-10 ($\alpha = 1$) at $\epsilon = 4$. We will re-implement it on our data splits and backbone, with an equal hyperparameter-tuning budget. We first check our re-implementation against these reported numbers before using it. SDFLoRA is the strongest adapter-based competitor, but its reported utility gains are relative ("up to 3.20%" on QQP). No reported absolute number found on Lacuna for SDFLoRA at a matched $\epsilon$, so it enters only in the optional language extension.

The strongest empirical defense without formal DP is the combination studied in Evaluating Gradient Inversion Attacks and Defenses: 90% gradient pruning plus Intra-InstaHide. It reports 86.10% CIFAR-10 accuracy (undefended baseline: 93.37%) and raises reconstruction LPIPS to 0.73 (undefended: 0.45), but without personalization. We will run it with our personal heads. It tells us whether formal DP buys anything over this heuristic on the leakage axes. The key ablation is our routed method with routing turned off, so the shared gradient again flows through the private head. Everything else stays identical, which isolates the one ingredient we add.

Our method beats a baseline in one of two ways. It may match the baseline's personalized and unseen-client accuracy within seed variance while cutting label-recovery accuracy, reconstruction quality, or linkage AUC by a margin larger than the seed spread. Alternatively, it may match the baseline's leakage while gaining accuracy.

Why not just…?

Why not just trust $\epsilon$ and pick a smaller one? A smaller budget does tighten every bound, but uniform Gaussian DP degrades sharply at tight budgets. PLRV-O reports Gaussian DP-SGD at 50.25% on SST-2 at $\epsilon \approx 0.2$. Full-model DP-SGD in the adapter study drops to roughly 53% where private adapters keep 91.82%. Practitioners therefore operate near $\epsilon = 4$, where the bound is loose. At that budget, two methods with the same $\epsilon$ can leak very differently, and our sweep measures exactly how differently. If leakage turns out flat across the sweep, this objection wins, and we will report it as such.

Why not just train locally and share nothing? Local-only training removes leakage entirely. It is included as a baseline because some low-data, highly skewed clients may genuinely prefer it. The cost appears on held-out clients, which local training cannot serve. It also appears on clients with little data, which is why FED-ROD and FedDPA beat pure local training. The unseen-client evaluation quantifies this cost directly.

Why not just withhold the head or prune gradients, without DP? Gradient Bridges shows that withholding the head still lets over 80% of labels be recovered. The evaluation study finds that pruning alone must exceed 99% to stop image inversion, at about a 10% accuracy cost. GIAS still reconstructs images at 99% pruning. We include the strongest such heuristic as a baseline instead of assuming it fails.

Why not just use secure aggregation? If the server sees only the sum of updates, per-client inversion and linkage become much harder. This is a real threat to the relevance of our threat model in cross-silo settings. It is weaker in cross-device settings, where participation changes from round to round. There, differences between successive aggregates can isolate one client's contribution, as the unlearning attack in DRAUN illustrates. We will run the attacks on aggregates under realistic dropout, and we will scope our conclusions to settings without secure aggregation if aggregate leakage turns out negligible.

Why not just reuse FED-ROD's generic head as-is? Our intervention resembles it, and that is deliberate. FED-ROD was not designed or evaluated for privacy. Its balanced loss also requires sharing class counts. The routing-off ablation and a version of FED-ROD with DP added test whether the gradient routing, rather than the two-head layout, is what reduces leakage.

Decisive Experiments

Experiments use CIFAR-10 and CIFAR-100 under Dirichlet label skew with $\alpha$ in {0.1, 0.3, 1}, and FEMNIST with clients defined by writer. FEMNIST is the most natural test of client linkage, because each client is a real person with a handwriting style. All runs use the same backbone, client-side user-level accounting at $\epsilon$ in {1, 4, 8}, and five seeds. Batch-normalization statistics are handled the same way in every run and are reported.

The first decisive check is the slope of leakage against personalization strength at fixed $\epsilon$, for each of the three attacks. A clearly positive slope supports the gradient-bridge prediction and shows that noise-free private modules raise leakage beyond what $\epsilon$ reveals. A negative slope supports the design FedDPA and SDFLoRA assume. The second check compares local and central noise placement for FedDPA's mask. If central noise leaves Gradient Bridges label recovery near the undefended level, then reported $\epsilon$ offers no protection against the server in that configuration. The third check is the routing ablation on the three-way frontier. Routing must lower label recovery and linkage AUC relative to the routing-off variant without costing more than seed-level personalized or unseen-client accuracy. If it does not, the intervention is dropped and the audit stands on its own.

An optional extension repeats the sweep and audit on SDFLoRA's shared adapter using GLUE tasks. There we attack the shared adapter updates to see whether input tokens can be recovered, which checks whether the finding holds outside vision.