In federated learning, phones and sensors train a shared model on their own data and send back only model updates. Two problems collide here. First, those updates leak. Gradient inversion attacks, meaning attacks that search for inputs whose gradient matches the one a client sent, can rebuild training images. Second, devices differ widely in compute. Systems such as HeteroFL and FjORD therefore give each weak device a narrow slice of the global network. The usual fix for leakage is DP-SGD, which clips every example's gradient and adds noise at every step. This is the worst possible fit for those weak devices. Per-example clipping is costly, the privacy budget drains with every local step, and recent denoising attacks can strip the added noise back out.
We propose a different approach for each training round on a weak client. The client adds a single random linear term to its local loss, plus a strong pull toward the current global weights. It then optimizes that perturbed loss to convergence with an ordinary optimizer. This is objective perturbation, a differential privacy mechanism that makes the solution of a randomized loss private, so the path taken to reach it costs no privacy. The idea fits the heterogeneous setting for three reasons. Its privacy cost depends on the dimension of the model being released, so it is cheapest for exactly the narrow submodels weak devices train. It needs no per-example clipping. And the noise it puts into the released update is not isotropic Gaussian noise. It is the random term filtered through the inverse curvature of the client's own data. That is a kind of noise diffusion-based gradient denoisers were never trained to remove.
Core Claim
Client $k$ holds $n_k$ examples, receives global weights $w_t$, and is assigned a submodel with $d_k$ parameters. It minimizes
$$F_k(w) = L_k(w) + \frac{\mu}{2}\|w - w_t\|^2 + \frac{1}{n_k} b^\top w,$$
where $L_k$ is its average training loss, $\mu$ is the strength of the pull toward the global weights, and $b$ is a Gaussian vector drawn fresh each round. Near the solution, the released weights differ from the non-private local solution by roughly
$$\Delta w \approx -\frac{1}{n_k}\left(H_k + \mu I\right)^{-1} b,$$
with $H_k$ the Hessian (curvature matrix) of the client's loss. This form comes from a single Newton step on the perturbed objective.
We make three claims, each testable.
The first concerns convexity. If $\mu$ exceeds the most negative curvature $\rho$ of the loss on a bounded domain, then $F_k$ is strongly convex (it has a single minimum and curves upward everywhere). Bounded inputs and smooth activations give the loss such a bound $\rho$; this property is called weak convexity. Under that condition the classical objective-perturbation argument applies per round. Inexact solvers are handled by Approximate Minima Perturbation, described below.
The second concerns where privacy is cheap. The per-round privacy penalty from the curvature term grows with the rank of each example's Hessian, which we bound by $d_k$. The guarantee is therefore tightest for narrow submodels. Rounds, not local steps, are what compose.
The third concerns attacks. Because $\Delta w$ is colored by $(H_k+\mu I)^{-1}$, which depends on the client's private data, denoisers trained on clean surrogate gradients under an isotropic forward-noise model should lose much of their advantage.
The first claim is a conditional formal guarantee, and we state it as conditional on $\rho$. The third is an empirical hypothesis that we try hard to falsify, including against a cheaper mechanism that imitates the same noise shape.
Relation to Prior Work
The method comes from the objective-perturbation literature. Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners gives modern Rényi-differential-privacy accounting and shows the guarantee attaches to the minimizer, so epochs and optimizer tuning are free. It covers only centralized linear learners. We reuse its accounting per federated round on submodels that have been convexified by the proximal term. Towards Practical Differentially Private Convex Optimization introduces Approximate Minima Perturbation (AMP). AMP adds secondary noise scaled by the final gradient norm when a solver stops early. We adopt it unchanged, because edge devices will not reach exact minima.
Functional Mechanism and Differential Privacy Preservation for Deep Auto-Encoders perturb Taylor-expanded loss coefficients. The second adds normalization layers to bound sensitivity. We borrow the normalization idea to bound per-example gradients without clipping. We avoid Taylor truncation, whose error grows with depth. Differentially Private Objective Perturbation: Beyond Smoothness and Convexity shows privacy can survive non-convexity through sampling or discretization. That is an alternative route if proximal convexification proves too restrictive.
Two papers constrain the design. Differentially private distributed convex optimization via objective perturbation is the closest distributed precedent. It perturbs function coefficients once, but it assumes identical strongly convex nodes and was tested only on 2D logistic regression. We instead re-perturb every round on heterogeneous deep submodels and pay round-wise composition. Differentially Private Learning Beyond the Classical Dimensionality Regime shows objective perturbation's error blows up when parameters outnumber samples. This warns that the narrow-submodel advantage holds only while $d_k$ stays well below $n_k$. We test that boundary directly rather than assume it away.
The heterogeneity side supplies the setting. HeteroFL carves width-scaled submodels with static batch normalization. FjORD uses ordered dropout, which nests submodels and removes ambiguity about which neuron is which. Neither paper analyzes privacy. Our method sits inside both training loops as a drop-in client procedure.
The leakage side supplies the threat, and it motivates the proposal. Gradient Diffusion: A Perturbation-Resilient Gradient Leakage Attack (Mjölnir) treats additive gradient noise as a forward diffusion process and removes it. Gradient Inversion Attacks Beyond SGD inverts multi-step Adam updates by matching updates rather than raw gradients. Do Gradient Inversion Attacks Make Federated Learning Unsafe? shows batch-normalization statistics leak, and that single-image clients are fully identifiable. Weak, data-poor clients are exactly these.
Research Program
We will first build the client procedure and its accounting. This step has four parts. We bound inputs and use normalization layers so per-example gradient norms are bounded. We estimate the weak-convexity constant $\rho$ of each submodel tier on public data, using a Lanczos eigenvalue routine, and set $\mu$ above it with a safety margin. We run L-BFGS or local SGD to a gradient-norm tolerance. Finally, we apply AMP's correction. Per-round guarantees compose across rounds with Rényi accounting and client subsampling. Choosing $\mu$ from public data, not private data, keeps the choice itself free of privacy cost.
Second, we will derive and check empirically how the per-round $\epsilon$ scales with $d_k$ and $n_k$ across HeteroFL's width levels from 1.0 down to 0.0625. Based on the dimension dependence of the curvature term, we expect it to rise with width.
Third, we will run a full attack battery against each tier. The battery comprises update-matching inversion, Mjölnir with its diffusion model retrained on surrogate clean updates, and batch-normalization matching where static batch normalization exposes statistics. As a secondary question, we will test whether FjORD's ordered submodels are more invertible than HeteroFL's at equal width.
A known tension remains. The proximal pull that buys convexity also limits how far a client can move each round, which may slow convergence for non-IID clients. We measure this tension rather than hide it.
Baselines
The naive reference point is plain HeteroFL and FjORD with no privacy mechanism. This fixes the accuracy ceiling and the leakage floor. On CIFAR-10 under balanced non-IID splits, HeteroFL reports 61.64% global accuracy against 58.99% for FedAvg (HeteroFL). We reproduce these numbers before adding anything.
The strongest existing defense is Fed-CDP from Gradient-Leakage Resilient Federated Learning. It is client-side per-example clipping plus Gaussian noise with a decaying clip bound. Of the defenses in the file, it is the one that defends against an honest-but-curious server, and it gives formal client-level guarantees. It reports 97.48% on MNIST against a 97.98% non-private baseline. We found no reported CIFAR-10 accuracy for Fed-CDP on Lacuna.
We run Fed-CDP on the same submodel tiers, data splits, round count and per-device compute budget. We tune it with an equal number of trials, and we count Fed-CDP's tuning trials against its privacy budget, since its guarantee depends on its hyperparameters. Our method's own tuning is free. We will report an untuned-Fed-CDP comparison as well, so the tuning asymmetry is visible.
Beating Fed-CDP has three parts, all assessed at matched final $\epsilon$. Weak-tier accuracy must be higher. Wall-clock time on the weak tier must be lower. Mjölnir and update-matching reconstruction quality (PSNR, perceptual distance LPIPS, and identifiability precision) must be no better for the attacker.
Two further heuristic baselines come from the literature. One is 2-bit update quantization, which SoK: Gradient Leakage in Federated Learning reports blocks current attacks at under 5% accuracy cost. The other is FedDF (Ensemble Distillation for Robust Model Fusion in Federated Learning), which fuses heterogeneous models through predictions on unlabeled server data.
Three ablations isolate the key ingredient. The first applies the proximal term alone without $b$, which tests whether regularization by itself explains any defense. The second is isotropic output perturbation: Gaussian noise added to the non-private local solution, calibrated to the same $\epsilon$. The third, and most important, is Hessian-shaped output perturbation. It adds Gaussian noise with covariance proportional to $(H_k+\mu I)^{-2}$, built from a curvature estimate, to the non-private solution. This isolates whether the defense comes from the shape of the noise or from the mechanism that produces it.
Why not just…?
Why not just run DP-SGD (Fed-CDP) on each submodel? It is the incumbent and gives a clean guarantee. Its costs, however, fall on exactly the clients that heterogeneity frameworks exist to serve. Per-example clipping is expensive on weak hardware, which the Fed-CDP authors acknowledge. Privacy composes over every local step, so weak clients must either train few steps or spend large budgets. And its additive Gaussian noise is precisely what Mjölnir was built to remove, reaching PSNR 18.25 at $\epsilon=1$ on CIFAR-100. The matched-$\epsilon$ comparison in our Baselines section decides this question. If Fed-CDP matches us on weak-tier accuracy and attack resistance at equal compute, our method has no reason to exist.
Why not just quantize updates to 2 bits? The SoK reports that quantization stops current attacks cheaply. It is a heuristic, though: it has no guarantee against attacks designed for low precision, and there is no membership-inference argument behind it. We include it as a baseline and also run attacks adapted to quantized updates. If an adapted attack fails against quantization, we will report that quantization is a strong practical competitor for reconstruction. Our formal-guarantee argument would then rest on membership-inference audits alone.
Why not just use distillation like FedDF, which never sends gradients? FedDF needs a representative unlabeled server dataset, which is often unavailable in medical or proprietary settings. It also ships whole client models, and its authors say these carry inversion and membership risk that would need differential privacy. Our method can supply that privacy to FedDF's client models. We test FedDF only where public data exist, for example with ImageNet as the distillation set for CIFAR-10, which FedDF reports works.
Why not just add Hessian-shaped Gaussian noise to the output and skip the perturbed optimization? This is the real threat. To first order it produces the same released distribution, so any denoising resistance we find may come from the noise shape alone. Our answer has two parts. Calibrating output perturbation requires a bound on the sensitivity of the solution, and that bound degrades with $1/\mu$ and needs a curvature estimate computed on private data, which itself costs privacy. Objective perturbation gets its shaped noise without either. The Hessian-shaped ablation settles the empirical half. If it matches us on attack resistance and accuracy at the same accounted $\epsilon$, we will say the contribution is the noise shape and recommend whichever mechanism is cheaper.
Why not just rely on many local steps and large batches? Update-matching inversion survives up to 8 local steps and batch size 16 (Gradient Inversion Attacks Beyond SGD). Single-image clients reach identifiability precision 1.0 in the medical setting (Do Gradient Inversion Attacks Make Federated Learning Unsafe?). Weak, data-poor clients are not protected by this "natural defense". The no-defense baseline under these settings will show it directly.
Proposed Experiments
The main testbed is CIFAR-10 with PreResNet18 at HeteroFL width levels from 'a' to 'e'. We add FEMNIST for writer-level non-IID data and FjORD's ordered dropout, and MNIST to compare directly with Fed-CDP's reported 97.48%. We attack faces on LFW, attack CIFAR-100 under Mjölnir, and use the chest X-ray setting to test single-image identifiability. We report accuracy per width tier and overall, the accounted $\epsilon$ after all rounds, reconstruction PSNR, LPIPS and identifiability precision per tier, membership-inference advantage measured with planted canary examples, and device wall-clock time.
Four checks decide the proposal.
The first is the denoiser check. We retrain Mjölnir's diffusion model on surrogate updates generated under our mechanism, giving the attacker full knowledge of it. We then compare its reconstruction PSNR against the Fed-CDP and Hessian-shaped output-perturbation releases at matched $\epsilon$. Our denoising-resistance claim survives only if PSNR against our releases is clearly lower than against Fed-CDP's and no higher than against the shaped baseline.
The second is the scaling check. Accounted per-round $\epsilon$ should rise with submodel width at fixed $n_k$. The weak-tier utility advantage over Fed-CDP should shrink, then reverse, as $d_k/n_k$ approaches one. That reversal is the boundary predicted by Differentially Private Learning Beyond the Classical Dimensionality Regime.
The third is the convexity audit. On held-out rounds we measure the smallest eigenvalue of the local objective's Hessian to confirm $\mu > \rho$ holds in practice. We also run canary-based membership audits to check that empirical leakage stays within the claimed $\epsilon$. Failures would show the conditional guarantee is not being met.
The fourth is the cost of convexity. We compare convergence rounds and non-IID accuracy against the no-$b$ proximal ablation. This separates the price of convexification from the price of privacy.
References
- Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners
- Towards Practical Differentially Private Convex Optimization
- Functional Mechanism: Regression Analysis under Differential Privacy
- Differential Privacy Preservation for Deep Auto-Encoders: an Application of Human Behavior Prediction
- Differentially Private Objective Perturbation: Beyond Smoothness and Convexity
- Differentially private distributed convex optimization via objective perturbation
- Differentially Private Learning Beyond the Classical Dimensionality Regime
- HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
- FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
- Mjölnir: Breaking the Shield of Perturbation-Protected Gradients via Adaptive Diffusion
- Gradient Inversion Attacks Beyond SGD
- Do Gradient Inversion Attacks Make Federated Learning Unsafe?
- Gradient-Leakage Resilient Federated Learning
- SoK: Gradient Leakage in Federated Learning
- Ensemble Distillation for Robust Model Fusion in Federated Learning