Federated learning trains one model across many devices that keep their data local and send only model updates to a server. Real fleets mix microcontrollers, phones and servers, so recent systems such as HeteroFL and FjORD let each device train a nested slice of a shared network: weak devices train only the first few channels of each layer, and strong devices train the full width. These updates can still be inverted. Gradient inversion attacks reconstruct a device's training images from the update it sends. The standard protection is differential privacy (DP), a guarantee that bounds how much any one participant can change what the server sees. DP works by capping each update's size ("clipping") and adding Gaussian noise. Existing DP recipes assume every device trains the same parameters. Under nesting that assumption fails in a specific way. The core channels are averaged over every device, so their noise averages away. The outer channels are averaged over only the few strong devices, so they stay noisy. The strongest devices end up paying most of the utility cost of privacy. Weak devices, which have no outer channels to protect, waste privacy budget they could have spent on the core.
We propose to treat the nested slices as separate groups inside a single DP guarantee. Each tier of device clips each slice to its own size limit. The server adds a different amount of noise to each slice. Both are chosen so that every device gets exactly the same $(\epsilon,\delta)$ guarantee, while the noise left after averaging is roughly even across slices. We pair this with the first per-tier measurement of gradient inversion in nested federated learning, with and without DP. The two results together say whether weak devices leak more, whether strong devices lose more accuracy, and whether a slice-aware allocation fixes both.
Core Claim
The claim has three parts. In width-nested federated learning under client-level DP with uniform clipping and noise, accuracy loss falls mostly on the wide slices, because the number of contributors per slice shrinks from all clients to a few. A per-tier, per-slice allocation of clipping and noise, holding every client's privacy guarantee fixed, recovers a large share of the full-width model's accuracy at the same $\epsilon$. That allocation should not increase empirical reconstruction quality for any tier.
The idea differs from its nearest neighbours as follows. Projected Federated Averaging with Heterogeneous Differential Privacy projects noisy updates onto a subspace estimated from relaxed-privacy clients. It handles differences in privacy budgets, assumes one shared architecture, and needs those relaxed clients. We assume equal budgets and different architectures, and need no public clients. Constrained differentially private federated learning for low-bandwidth devices (FL-TOP) shows that confining DP noise to a small fixed parameter subset improves the ratio of signal to noise: Fashion-MNIST accuracy rises to 0.81 from 0.56. Its subset is identical for all devices and chosen from public data. We build on its lesson that where noise is spent matters. Here, though, the subsets are dictated by device hardware and overlap in a nested way.
Mechanism
Partition the network's width into nested slices $s=1,\dots,T$, where slice 1 is the core channel block shared by every tier and slice $s$ is the extra block added at tier $s$. A tier-$t$ client trains slices $1..t$. It clips its slice-$s$ update to norm $C_{t,s}$. The server sums slice $s$ over its $n_s$ contributing clients and adds Gaussian noise with standard deviation $\sigma_s$. One client's influence on the noisy release is then a Gaussian mechanism whose effective sensitivity is the noise-scaled norm. Every client gets the same guarantee as a standard Gaussian mechanism with noise multiplier $z$ provided
$$\sum_{s=1}^{t} \frac{C_{t,s}^2}{\sigma_s^2} \le \frac{1}{z^2} \quad \text{for every tier } t.$$
This follows because the release is a Gaussian mechanism with diagonal covariance, and its privacy loss depends only on the client's sensitivity measured in noise units. Subsampling amplification and Rényi-DP accounting therefore carry over unchanged from DP-FedAvg.
The noise remaining in the averaged slice-$s$ update is $\sigma_s/n_s$. Its signal is bounded by the typical clipped norm on that slice. We choose $\{C_{t,s}, \sigma_s\}$ to minimize a proxy for utility loss, the sum over slices of clipping bias plus residual noise variance, subject to the constraint above. Clipping bias is estimated from privately released norm quantiles in the style of Adaptive Clipping for Differential Private Federated Learning in Interpolation Regimes. That paper adapts a single global clipping radius as gradients vanish. We reuse its private norm-estimation step per tier and per slice, because narrow slices underfit and keep large gradients while the full-width model converges.
Two effects follow directly from the constraint. A tier-1 device can spend its whole budget on the core slice, so it clips the core more loosely than a full-width device can. Outer slices with small $n_s$ receive a larger share of every high-tier client's budget, which partly offsets their small contributor count. The allocation is solved in closed form under the quadratic proxy and refreshed every few rounds. Its inputs, the per-tier counts $n_s$, are already public under HeteroFL scheduling.
Research Program
The work has three stages. The first is a leakage audit. We run HeteroFL (HeteroFL) and FjORD's ordered dropout (FjORD) and attack each tier's update. FjORD shows that ordered dropout pushes the highest-variance features into the lowest channel indices, much as a singular value decomposition orders components, so a narrow update may be unusually informative per coordinate. We use three attacks:
- optimization-based gradient matching under the relaxed, realistic assumptions of Evaluating Gradient Inversion Attacks and Defenses in Federated Learning, meaning no batch-norm statistics and no label priors;
- GIAS, the generative-prior attack of Gradient Inversion with Generative Image Prior;
- SPEAR (SPEAR), run on the fully connected ReLU layers where its exact recovery is defined, with labels supplied by RLU (Recovering Labels from Local Updates).
This answers the open question raised in the file: are narrow tiers easier to invert because the attack's search space is small, or harder because capacity is limited? The answer matters for whether equal $\epsilon$ is even the right goal.
The second stage measures the utility side under uniform DP. We record per-slice residual noise and per-tier test accuracy to confirm, or refute, that the damage concentrates in the wide tiers.
The third stage builds the slice-aware allocation and evaluates it on the same grid. We also run a secure-aggregation analysis, because nested aggregation shrinks the crowd that hides each outer slice.
Baselines
The naive reference point is HeteroFL or FjORD with standard DP-FedAvg. That means one clipping norm for the whole update, isotropic noise, and HeteroFL's count-normalized averaging per coordinate. This is what a practitioner would build today, and it is the configuration the proposal must beat. No reported result was found on Lacuna for DP-trained HeteroFL or FjORD on any benchmark, so we establish this baseline ourselves. The non-private ceiling is reported: HeteroFL reaches 61.64% global accuracy on non-IID CIFAR-10, against 58.99% for FedAvg (HeteroFL).
The strongest existing competitor is per-tier adaptive clipping: AdaptDP-FedAvg's private quantile-based radius, run separately for each tier but with isotropic noise within a client. It adapts to the different gradient scales across tiers without our slice-level budget split, so it is the most likely to close the gap by itself. A second competitor is the least-common-denominator design: a single model the width of the smallest tier, trained with DP-FedAvg. HeteroFL's report that a tiny model reaches 77% standalone but over 90% inside heterogeneous aggregation suggests this design loses capacity, but under DP its small dimension also means less noise, so it must be measured. An FL-TOP-style baseline restricts DP training to a fixed top-$K$ subset, chosen so its size fits the smallest tier.
The key ablation keeps per-tier clipping but forces $\sigma_s$ equal across slices. This isolates whether the gain comes from slice-level noise allocation or merely from tier-level clipping.
Fairness is enforced in four ways. Every method uses the same client partition and tier assignment, the same number of rounds and local steps, and the same accountant at $\epsilon \in \{1, 3, 8\}$ with $\delta = 10^{-5}$. Each gets an equal hyperparameter budget of 20 configurations over noise multiplier, learning rate and target clipping quantile. Any data-dependent tuning is charged to the privacy budget.
Beating the strongest competitor means two things at matched $\epsilon$, over five seeds with non-overlapping confidence intervals. The full-width model must gain at least two accuracy points, and no tier's accuracy may drop by more than half a point.
Why not just…?
Why not just use secure aggregation instead of DP? GuardHFL reports that cryptographic baselines are 4.4x to 75.6x slower than its own protocol, which is costly for edge devices. There is also a structural problem specific to nesting. An outer slice is summed only over the few full-width clients in a round. If only one or two strong devices participate, the aggregate of that slice is close to an individual update. We will measure attack fidelity on outer-slice aggregates as a function of $n_s$ to quantify how much protection secure aggregation provides there. Secure aggregation can be layered on top of our mechanism, but it does not replace a guarantee for a single client.
Why not just run AdaptDP-FedAvg separately per tier? This is the real threat. If tier-level clipping alone recovers most of the accuracy, the slice-level allocation adds nothing worth its complexity. The equal-$\sigma_s$ ablation, and the per-tier adaptive clipping baseline itself, settle this directly. We expect a residual gap because per-tier clipping cannot change the fact that outer slices average noise over few contributors. Only shifting noise across slices addresses that. If no gap appears, we will report the simpler method as the recommendation.
Why not just train one small model for everyone with DP, or prune gradients? Training only a small model removes heterogeneity entirely. HeteroFL's 77% versus over-90% contrast argues against this without DP, and FlexLoRA's "bucket effect" makes the same point for adapter ranks (Federated Fine-tuning of LLMs under Heterogeneous Tasks and Client Resources). We run it anyway because its DP noise is cheap. Gradient pruning without DP is weaker. Evaluating Gradient Inversion Attacks and Defenses finds that pruning must exceed 99% of coordinates to stop attacks, which costs about 10 accuracy points, and GIAS still reconstructs images when 99% of gradient values are removed.
Proposed Experiments
The vision experiments use CIFAR-10 with HeteroFL's five width tiers, 'a' through 'e' (100% down to 6.25% width), under both IID and non-IID splits. Language and handwriting experiments use FEMNIST and Shakespeare with FjORD's ordered-dropout tiers. For each tier and $\epsilon$, we report test accuracy of each tier's submodel and of the full-width model. We also report reconstruction PSNR, SSIM and LPIPS at batch sizes 1, 8 and 32 for each attack. As a calibration point, the file reports LPIPS of 0.45 with no defense and 0.73 under the combined GradPrune and InstaHide defense at batch size 32 on CIFAR-10 (Evaluating Gradient Inversion Attacks and Defenses). We report empirical leakage alongside the formal $\epsilon$ because the two can disagree across tiers.
Three checks decide the outcome. First, under uniform DP, residual noise per slice should scale as $1/n_s$ and full-width accuracy should fall more than tier-'e' accuracy. If it does not, the motivating mechanism is wrong and the proposal stops there. Second, the slice-aware allocation must beat per-tier adaptive clipping by the margin defined above at $\epsilon=3$. Third, on the attack side, no tier under the new allocation may show lower LPIPS (that is, better reconstruction) than under uniform DP at the same $\epsilon$. The looser core clipping for weak devices is exactly where extra leakage would appear if the formal accounting misses something. The leakage audit also yields a standalone finding: whether narrow, ordered-dropout tiers are more invertible per sample than wide ones, which no paper in the file has measured.
References
- Projected Federated Averaging with Heterogeneous Differential Privacy
- Constrained differentially private federated learning for low-bandwidth devices
- Adaptive Clipping for Differential Private Federated Learning in Interpolation Regimes
- HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
- FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
- Evaluating Gradient Inversion Attacks and Defenses in Federated Learning
- Gradient Inversion with Generative Image Prior
- SPEAR: Exact Gradient Inversion of Batches in Federated Learning
- Recovering Labels from Local Updates in Federated Learning
- GuardHFL: Privacy Guardian for Heterogeneous Federated Learning
- Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources