Spending Each Device's Privacy Budget Where the Server Is Blind: Local DP for Width-Nested Federated Learning

Federated learning trains one model across many devices without collecting their raw data: each device computes an update on its own data and sends only that update to a server. Two practical facts break the textbook version. First, those updates are not safe to reveal. An attacker who sees one device's gradient can often run an optimization that reconstructs the training images behind it, a family of attacks called gradient inversion. Second, devices differ widely in memory and compute, so many systems give weaker devices a thinner slice of the model. In these "width-nested" schemes, every device trains the core channels, and only stronger devices also train the outer channels. The standard defense that assumes nothing about the server's honesty is local differential privacy (LDP), where each device clips its own update to a fixed norm and adds calibrated noise before sending it. Under LDP, the server's reconstruction of each parameter is only as clean as the number of noisy reports averaged into it. In a nested model, that number is large for the core channels and small for the outer ones.

This note proposes a tier-aware local privacy mechanism. Each device keeps its formal privacy guarantee exactly as before. What changes is how the device splits its fixed clipping budget across the nested slices it trains, with each slice's share set by how many devices the server will average on that slice. The split has a closed form. It changes nothing about the noise a device adds, so no individual device gives up any privacy. We pair the mechanism with a per-tier gradient-inversion audit. The audit asks whether small devices, which send only the dense, information-rich core of the model, are empirically more exposed at the same nominal privacy level, and whether reallocating the budget changes that exposure.

Core Claim

Take a width-nested model with slices $s = 1, \dots, S$ ordered from core to outermost. Let $d_s$ be the number of parameters in slice $s$, and let $n_s$ be the number of devices per round whose tier includes slice $s$; $n_s$ shrinks as $s$ grows. Every device uses the Gaussian mechanism: it clips its full update to norm $C$ and adds noise of standard deviation $\sigma C$ to every coordinate it sends, so its $(\epsilon,\delta)$ guarantee depends only on $\sigma$. Within that single clip, the device may clip each slice separately to $C_s$, as long as $\sum_s C_s^2 = C^2$. This keeps the total sensitivity at $C$ and the guarantee unchanged. After averaging, the server's noise-to-signal ratio on slice $s$ is roughly

$$\rho_s = \frac{\sigma^2 C^2 d_s}{n_s\, C_s^2},$$

because $d_s$ noise coordinates of variance $\sigma^2C^2$ are averaged over $n_s$ reports against a signal of norm at most $C_s$. Minimizing a weighted sum $\sum_s w_s \rho_s$ under the budget constraint gives, by a standard Lagrange argument,

$$C_s^2 \propto \sqrt{w_s\, d_s / n_s},$$

where $w_s$ is the importance of slice $s$ to the loss. We start with $w_s = 1$ and later estimate it from noisy per-slice norm statistics. A tier-$t$ device applies the same rule restricted to slices $1..t$.

The uniform default effectively makes $C_s^2 \propto d_s$. It ignores $n_s$, so it starves the outer channels that only a few strong devices can train. The claim is that, at the same per-device $\epsilon$, tier-aware allocation recovers a meaningful part of the accuracy that naive LDP costs in nested federated learning. A second claim, tested rather than assumed, is that inversion risk under LDP is unequal across tiers in practice even when the formal bound is equal.

Where This Sits

This work builds on three pieces of prior work. HeteroFL slices one global network by channel width and aggregates "inclusively," so core parameters receive updates from all clients and outer parameters only from high tiers. FjORD orders neurons by importance through Ordered Dropout, which it shows matches SVD for linear maps. That ordering pushes dominant features into the core slice. Neither paper adds privacy. That uneven aggregation is exactly what makes uniform LDP wasteful: the noise on core parameters averages down across all clients, while outer parameters keep the noise of the few strong devices. We supply the missing privacy layer and exploit the uneven counts rather than suffer them.

The closest privacy work is CENTAUR. It shares only a representation extractor under DP and keeps classifier heads local, which reduces clipping distortion with a proven $O(\sqrt{d})$ gain in the linear case. CENTAUR still requires an identical extractor on every client. Our mechanism is compatible with its head-local design, but it targets the part CENTAUR fixes: the extractor itself varies in width across devices.

The motivation for the inversion audit comes from two papers. Inversion Influence Functions show that leakage is directional, since noise along the Jacobian's small singular values protects best, and unequal across classes. That suggests SVD-ordered core slices may carry more reconstructible signal per sent coordinate. MineGrad/PEFTLeak show that small parameter subspaces (rank $\le 64$ adapters) still leak 82.8% of CIFAR-10 patches at batch size 32 under an active server. Being "small" is therefore no protection, which constrains us to formal noise rather than size-based arguments.

Two papers constrain the design. SHE-LoRA rejects DP for low-rank factors because noise multiplies through $B \times A$. We therefore restrict this proposal to width-nested dense layers, where that amplification does not arise, and leave low-rank variants explicitly out of scope. Privacy Aware Learning shows that local privacy shrinks the effective sample size. That is the reason per-slice report counts $n_s$, not total clients, are the right quantity to plan around.

Research Program

We first build the mechanism on top of HeteroFL and FjORD. This means slice-wise clipping with the closed-form split, followed by one isotropic Gaussian draw per device per round. Privacy is accounted per device across rounds with standard composition, since LDP must hold even against a server that keeps every report. We will also derive a corrected server-side estimator that rescales each slice by its known $C_s$, so that slices clipped differently are combined without bias in direction.

Second, we analyze when allocation matters. The predicted gain grows with the spread of $n_s$ across slices, so it is largest when strong devices are rare. It shrinks to nothing when all tiers are equally common or the model is homogeneous. We will vary the tier mix and check that the measured gains follow this prediction. If they do not, the noise-to-signal proxy is the wrong objective.

Third, we run the per-tier audit. For each tier, $\epsilon$ level, and allocation scheme, we attack individual noisy reports with a pixel-space optimization attack and with the generative-prior attack GIAS from Gradient Inversion with Generative Image Prior. We also run its meta-learned variant GIML across rounds, because LDP reports accumulate over time. Alongside the attacks, we compute the $I^2F$ lower bound using the released implementation from Inversion Influence Functions, adapted to tier-specific Jacobians. Checking whether this cheap estimate tracks the attack results across tiers is itself a contribution, because $I^2F$ was developed only for homogeneous models.

Baselines

The naive reference point is HeteroFL (and FjORD) with uniform local Gaussian noise. Each device clips its whole update to $C$ and adds the same noise, which is what one gets by combining the two literatures without thought. It is the direct ablation of our key ingredient, because the noise, $\epsilon$, accounting, data split, and architectures are identical and only the clipping split differs. We also report the same nested models with no privacy as a utility ceiling. HeteroFL reports 61.64% global accuracy on non-IID CIFAR-10 with PreResNet18 tiers 'a'–'e', against 58.99% for FedAvg (HeteroFL). We found no reported result on Lacuna for HeteroFL or FjORD under local DP, so we will run the uniform baseline ourselves.

The strongest existing method on this question is CENTAUR (CENTAUR), deployed the only way it can be under hardware limits: with an extractor small enough for the weakest tier, shared by everyone, and with local heads. CENTAUR reports 53.23% CIFAR-10 accuracy at $\epsilon = 1$, against 46.17% for DP-FedAvg. We will rerun it under our local-noise threat model, client count, data partition, and per-device $\epsilon$, rather than compare to that figure directly, because its DP setting is not identical to ours.

A second strong reference is our mechanism combined with CENTAUR-style local heads. If adding local heads accounts for all the gain, our contribution is the nesting rather than the allocation, and we will say so. All methods receive the same tuning budget over clip norm, learning rate, and local steps, chosen by a fixed-size grid. We use the same rounds and seeds, three seeds each.

Beating the baselines means two things. Utility: at matched per-device $\epsilon \in \{1, 4\}$, higher global and per-tier accuracy than uniform-LDP HeteroFL, and higher accuracy than smallest-extractor CENTAUR on large-tier devices without losing accuracy on small-tier ones. Privacy: no worse reconstruction quality, measured by PSNR and SSIM, than the uniform baseline for any tier.

Why not just…?

Why not just shrink everyone to the smallest model and use CENTAUR? This is the most serious competitor. It removes the uneven-count problem by construction, and CENTAUR already handles clipping distortion well. The cost is capacity for every device that could run more, the "lowest common denominator" problem that motivates HeteroFL. HeteroFL itself gains about 2.6 points over FedAvg without privacy. Under LDP that capacity advantage could disappear if outer slices are too noisy to learn. Our main experiment settles this directly: if smallest-extractor CENTAUR matches tier-aware nested training on large-tier accuracy at $\epsilon = 1$, nesting under local DP is not worth it, and we would report that as the finding.

Why not just give strong devices a larger $\epsilon$, or drop weak devices? Raising $\epsilon$ for rich devices does clean up the outer slices, but it ties privacy to hardware, so owners of expensive phones are protected less. Our mechanism avoids exactly that inequity by holding $\epsilon$ fixed. Dropping weak devices reintroduces the sampling bias that heterogeneous federated learning exists to remove, and it reduces $n_s$ on the core slice too. We include a "drop tiers d–e" run so the reader can see the trade.

Why not just use secure aggregation or encryption with central DP? If the server only ever sees sums, inversion of individual reports is impossible, and central DP's utility is better. The proposal's threat model is a server that may itself be curious or compromised. SHE-LoRA notes that its shared-key protocol breaks if the server colludes with one client. It also reports that full encryption costs vastly more communication and encryption time than its selective variant, which is a heavy load for the low-tier devices nesting is meant to serve. These approaches complement LDP rather than replace it, and we do not claim LDP beats them on utility.

Why not just send logits on public data (FedDF) instead of gradients? FedDF fuses mixed architectures without parameter averaging, but it needs a representative unlabeled dataset. Such data is often unavailable in domains like rare cancer detection, and logits on that data still leak through model inversion. We treat it as out of scope rather than a baseline, because it changes the data assumption.

Proposed Experiments

The main testbed is CIFAR-10 and CIFAR-100 with PreResNet18 in HeteroFL tiers 'a'–'e' and in FjORD ordered-dropout form. We use the non-IID partition of HeteroFL, 100 clients, and three tier mixes: uniform across tiers, 80% weak devices, and 95% weak devices. FEMNIST with the FjORD CNN serves as a naturally partitioned check. The per-device privacy grid is $\epsilon \in \{0.25, 1, 4\}$. Utility metrics are global accuracy and accuracy for each tier's submodel.

Three checks are decisive. First, the gain over uniform LDP should grow with the skew of the tier mix, as the $\sqrt{d_s / n_s}$ rule predicts, and vanish for a homogeneous model. A gain below one point at the 80% weak-device mix would count against the idea. Second, comparing against smallest-extractor CENTAUR determines whether nested capacity survives local noise at all.

Third, the per-tier inversion audit uses GIAS and a pixel-space attack on CIFAR-10 at batch sizes 1, 4, and 32, plus GIML across 50 rounds. GIAS reports about 15.58 dB PSNR against 13.78 dB for pixel attacks without defense (Gradient Inversion with Generative Image Prior). The audit tests whether core-only reports from tier-'e' devices reconstruct measurably better than full-width reports at equal $\epsilon$, and whether the $I^2F$ bound ranks tiers in the same order as the attacks. If small tiers are more exposed in practice, the formal guarantee is uniform but empirical risk is not, and the next step would be tier-dependent noise directions guided by $I^2F$. If they are not, the result shows that LDP's uniform guarantee holds in practice under nesting, which is also worth knowing.