In federated learning, many devices train a shared model without sending their raw data to a server. When the devices differ in hardware, methods such as HeteroFL and FjORD let weak phones train a narrow slice of the network while strong devices train the full width. The slices are nested, so the innermost core is updated by everyone and the outermost channels only by the strongest devices. Differential privacy (DP), the standard guarantee that the trained model reveals little about any one participant, is still applied to these systems as if every device sent the same kind of update. Every update is cut down to one shared maximum norm, called the clipping bound, and one noise level covers all coordinates. We argue this wastes most of the privacy budget in a structured way. A weak device touches only the core block, so under per-client privacy accounting it has already spent less privacy than a strong device. Meanwhile the outer blocks are averaged over few contributors and are drowned in noise that was sized for the whole population.
The proposal is to treat the nested block structure as the unit of privacy design. Each block gets its own noise level, and each device tier gets its own clipping bound on each block it trains. These values are chosen by a small optimization so that every tier meets the same per-client privacy target. The statistics that optimization needs, namely how large each tier's updates are on each block, are tracked privately during training at a small, accounted cost, following the one-bit feedback idea of adaptive clipping. The same tracked bounds then serve as the dynamic range of the uplink quantizer, so compression requires no extra scale parameters and no separate tuning. The work also answers a question that heterogeneous FL has never measured: once realistic noise and quantization are in place, do narrow-tier updates leak more or less under gradient inversion than full-width ones?
Core Claim
Nested heterogeneous federated learning gives each client a different sensitivity vector over parameter blocks. Individual (per-client) Rényi DP accounting can exploit this directly. Write $s_j$ for the standard deviation of the Gaussian noise added to block $j$ of the aggregate, and $c^{(k)}_j$ for the clipping bound that tier-$k$ clients use on block $j$, where tier $k$ trains blocks $1,\dots,k$. One round of the mechanism then gives a tier-$k$ client Rényi DP of order $\alpha$ with
$$\varepsilon_k(\alpha) = \frac{\alpha}{2}\sum_{j\le k}\left(\frac{c^{(k)}_j}{s_j}\right)^2 .$$
This expression, before subsampling amplification and composition over rounds, follows from treating the block-wise Gaussian mechanism as one Gaussian mechanism with a per-client sensitivity. Requiring $\varepsilon_k \le \rho$ for every tier leaves slack for the lower tiers. That slack can be spent on looser core-block clipping for weak devices, or on lower core-block noise, depending on where the estimated error is largest. We choose $(s_j, c^{(k)}_j)$ each round to minimize a plug-in estimate of aggregate error. The estimate sums three terms per block: noise variance divided by the squared number of contributors to that block, clipping bias estimated from the privately tracked norm quantiles, and stochastic-rounding variance at the chosen bit-width. This is a small convex program with on the order of $K^2$ variables for $K$ tiers, and the server solves it at negligible cost.
The claim is new in one specific way. It is neither per-tier clipping with a single noise level, which gains nothing because sensitivity is set by the largest bound, nor a fixed analytic rescaling. It is a joint allocation of noise and clipping across the nesting structure, justified by individual accounting and fed by privately estimated statistics.
Relation to Prior Work
Two papers supply the method, and one supplies the setting. Differentially Private Learning with Adaptive Clipping tracks a target quantile of update norms using noisy one-bit client feedback and geometric updates, at less than 1% extra noise. We reuse that estimator, but run one tracker per (tier, block) pair. The paper assumes a homogeneous model and a single clipping bound $C_t$; that assumption breaks when update norms differ systematically across widths, which is the gap this proposal addresses. HeteroFL and FjORD supply the nested-width setting. HeteroFL does this through inclusive aggregation of slices. FjORD does it through Ordered Dropout, which drops neurons in a fixed order and is equivalent to SVD ordering for linear maps. Neither paper considers privacy. We take their architectures as given and change only how updates are clipped, noised and quantized.
Several papers motivate the proposal and also constrain its design. DPQuant shows that DP noise calibrated in $\ell_2$ inflates the error of quantization bounded in $\ell_\infty$, and it spends a small privacy budget on layer sensitivity estimates. This is the reason we quantize the clipped update before noise is added, and use the privately tracked bound as the quantizer range, rather than quantizing noisy vectors. DP-HyPO and Practical DP Hyperparameter Tuning with Subsampling represent the approach of running separate private tuning trials. The second of these reports that its scaling-law extrapolation fails for adaptive optimizers. Both assume one homogeneous model. We avoid separate trials entirely by tuning inside a single training run. A Meta-learning Framework for Tuning Parameters of Protection Mechanisms derives closed-form protection parameters, but only for a single round and for homogeneous clients. Our accounting is multi-round and tier-aware by construction.
On leakage, Inverting Gradients reconstructs images from trained networks using a cosine-similarity loss. Gradient Inversion with Generative Image Prior strengthens the attack with a generator prior. Concealing Sensitive Samples argues that reconstruction succeeds when per-sample updates are weakly entangled. These papers lead to opposite predictions for narrow tiers. Fewer parameters give the attacker fewer equations, but low capacity and the SVD-like ordering in FjORD may concentrate dominant input features in a small, easily inverted core. Mixed Quantization Enabled FL claims that keeping quantization scales private defeats the DLG inversion attack, but it gives no formal guarantee and evaluates single rounds. We test whether that protection survives multiple rounds and a generator prior.
Research Program
The first component is the accountant and the allocator. We implement individual Rényi DP accounting for block-structured Gaussian noise with Poisson client sampling, and verify it against an ordinary accountant in the homogeneous special case. We then implement the per-round convex allocation of $(s_j, c^{(k)}_j)$ and bit-widths $b_j$. Its inputs are private quantile estimates, the number of expected contributors per block, and a communication budget per tier.
The second component is the in-run private tuner. Each (tier, block) tracker spends a fixed, accounted share of $\rho$. We will measure how the total tuning cost grows with $K$. We will also test a pooled variant that shares one tracker per block and models tier offsets with a single learned scale. This variant keeps the tuning tax flat as the number of tiers grows.
The third component is a leakage study aimed at the honest-but-curious server. Under central DP the server sees individual clipped updates, so we evaluate the distributed-noise variant, in which each client adds its share $s_j/\sqrt{n_j}$ of the block noise before quantization, with $n_j$ the number of expected contributors to block $j$. This is the realistic deployment without secure aggregation. It also produces a sharp prediction. Core blocks have the most contributors, so each client's noise share there is smallest. Weak devices, which send only the core, would then send the least-noised individual updates. The allocator may therefore need an extra per-client floor on the noise share, and we will measure its cost.
The fourth component, as an extension if time allows, applies the same nesting argument to rank-heterogeneous LoRA. This follows FedRoRA, which decomposes each update into directions and a diagonal magnitude matrix $S$ ($\Delta W=\tilde B S\tilde A$) and explicitly leaves DP to future work. Treating ranked directions as nested blocks gives the analogous allocation.
Baselines
The naive reference point is DP-FedAvg on HeteroFL slices with one fixed clipping bound and one noise level. The bound is tuned by grid search on the private task, and the grid's privacy cost is either ignored or charged, reported both ways. This shows what practitioners do today.
The strongest existing method is adaptive quantile clipping from Differentially Private Learning with Adaptive Clipping, applied globally to the heterogeneous updates. It is the strongest competitor because it already removes manual clipping search at small privacy cost, and it is what one would deploy on HeteroFL without new ideas. The file reports only qualitative results for it ("adaptive clipping to the median outperformed the best-tuned fixed clip" on Shakespeare), so there is no reported result found on Lacuna for adaptive clipping under heterogeneous architectures. The non-private anchor is HeteroFL's reported 61.64% global accuracy on non-IID CIFAR-10 with PreResNet18, against 58.99% for homogeneous FedAvg, from HeteroFL. There is no reported result found on Lacuna for any differentially private HeteroFL or FjORD run.
Three ablations isolate the key ingredient. The first uses per-tier adaptive clipping with a single shared noise level; it should gain little, because sensitivity is the maximum clipping bound. The second uses block-wise noise but ignores per-tier slack, forcing all tiers to the top tier's constraint. The third uses the full allocator, but with quantizer ranges set by a separate heuristic instead of the tracked bounds.
All methods share the same data partitions, tier proportions, number of rounds, client sampling rate, total $\varepsilon$ at fixed $\delta$ with every tuning query charged, and uplink bits per round. Our method counts as better if, at matched $\varepsilon$ and bits, it raises worst-tier accuracy and global accuracy by more than the seed-to-seed spread across five seeds. It must do so at at least two privacy levels, without reducing inversion PSNR margins relative to the global adaptive-clipping baseline.
Why not just…?
Why not just run a separate adaptive clipping tracker per tier? This is the obvious fix, and it is ablation one. It fails for a structural reason. With a single noise level on the aggregate, the noise must be scaled to the largest per-tier bound, so weak tiers get smaller bounds but no reduction in noise, and their updates are simply shrunk. Gains appear only when noise is also split by block and the per-tier privacy slack is spent. If ablation one matches the full method, the core claim is wrong, and we will report that.
Why not just rescale one clipping bound analytically by width, for example $C\sqrt{r_k}$ for width ratio $r_k$? This is a real threat, because it costs no privacy budget. Nothing in the file shows that update norms follow such a law. Non-IID data, static batch normalization and FjORD's self-distillation plausibly distort norms tier by tier, and the scaling-law failure reported by Practical DP Hyperparameter Tuning with Subsampling warns against assuming one. We will include this rule, combined with block-wise noise, as a baseline. If it comes within noise of the privately tracked version, we will recommend it and drop the tracker, keeping only the allocation.
Why not just train the smallest model everywhere with standard DP? This removes heterogeneity. However, HeteroFL's own non-private comparison, 61.64% against 58.99% on CIFAR-10, shows the capacity lost by homogeneous training. Under DP the smaller model also has fewer coordinates to noise, which could narrow that gap. We therefore run homogeneous DP-FedAvg on the core-width model as a baseline at every privacy level, to show whether heterogeneity still pays under privacy.
Why not just tune everything on public proxy data? Proxies are often unavailable in regulated domains, as Federated and Differentially Private Learning for Electronic Health Records notes. More importantly for us, per-tier norm statistics depend on the private non-IID client population. We will still run a variant tuned on a proxy, using CIFAR-100 bounds transferred to CIFAR-10, to measure how much the tracker adds.
Proposed Experiments
Experiments use three benchmarks with user-level structure. The first is CIFAR-10 with PreResNet18 under HeteroFL's non-IID splits and five width tiers ($r$ from 1.0 down to 0.0625). The second is FEMNIST from LEAF, with natural writer partitions. The third is Shakespeare with an LSTM under Ordered Dropout, as in FjORD and adaptive clipping. We sweep $\varepsilon \in \{1, 3, 8\}$ at $\delta = 10^{-5}$ and two uplink budgets, 8 bits per coordinate and 4 bits per coordinate on average. We report global accuracy, worst-tier accuracy, total privacy cost including tuning, and bits per round.
The decisive utility check has two parts. The first is whether the full allocator beats global adaptive clipping on worst-tier accuracy at $\varepsilon = 3$ while ablation one does not. The second is whether that gain grows as tier widths spread further apart, which the $1/n_j^2$ noise term predicts.
The leakage study attacks individual client updates with the cosine-similarity attack of Inverting Gradients, which reports about 14 dB PSNR on trained ResNet-20 in the non-private setting, and with the generator-prior attack GIAS, which reports 15.58 dB on CIFAR-10. We report PSNR per tier, per block, and per round. As a calibration point for quantization-only defenses, we reproduce the DLG attack on LeNet-5 against Mixed Quantization, which reports PSNR falling from 55.3 dB to 8.1 dB. We then give the attacker updates from multiple rounds to see whether keeping scales private still holds.
The decisive leakage check is whether narrow-tier updates reconstruct better than wide-tier ones under distributed noise. If they do, a per-client noise floor becomes necessary, and we report its accuracy cost. Either outcome would be the first measured answer to how nested-width federated learning changes gradient leakage.
References
- Differentially Private Learning with Adaptive Clipping
- HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
- FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
- DPQuant: Efficient and Private Model Training via Dynamic Quantization Scheduling
- DP-HyPO: An Adaptive Private Framework for Hyperparameter Optimization
- Practical Differentially Private Hyperparameter Tuning with Subsampling
- A Meta-Learning Framework for Tuning Parameters of Protection Mechanisms in Trustworthy Federated Learning
- Inverting Gradients - How easy is it to break privacy in federated learning?
- Gradient Inversion with Generative Image Prior
- Concealing Sensitive Samples against Gradient Leakage in Federated Learning
- Mixed Quantization Enabled Federated Learning to Tackle Gradient Inversion Attacks
- Federated and Differentially Private Learning for Electronic Health Records.
- LEAF: A Benchmark for Federated Settings