Federated learning trains one model across many devices that never share raw data. The devices send only model updates, and the server combines them. Two facts about this setup pull against each other. Updates leak data: an attacker holding an update can often reconstruct the images or sentences that produced it. The standard fix is differential privacy (DP), which bounds each client's influence by clipping its update to a fixed length and then adding Gaussian noise. That noise is the same size in every coordinate of the model. Meanwhile, the server-side adaptive optimizers that make non-IID federated training work, meaning Adam-style rules that rescale each coordinate by a running estimate of its variance, already know that coordinates differ greatly in how much signal they carry.
This proposal connects the two. The server's Adam state is computed only from past noisy, already-private outputs, so under DP's post-processing rule it is public and costs no extra privacy budget. We propose using it to reshape the geometry of the privacy mechanism itself. In the next round, clients clip and noise their updates in the coordinate system defined by the server's preconditioner (the per-coordinate rescaling the optimizer applies), and the server maps the result back. The privacy guarantee is unchanged. The noise, however, is redistributed toward coordinates whose updates are naturally large and away from those whose updates are small, which roughly equalizes signal-to-noise across coordinates. We then ask a question that formal DP accounting cannot answer: at the same privacy budget $\epsilon$, does this reshaping change how well gradient inversion attacks actually reconstruct client data?
Core Claim
Our claim is that the server's own debiased second-moment estimate is a free, privacy-preserving description of update geometry. Two things follow if it is used to shape clipping and noise. First, federated Adam under DP should reach higher accuracy at fixed $\epsilon$ than applying isotropic noise and correcting the optimizer afterwards. Second, the resulting privacy–utility–leakage frontier should differ measurably from the isotropic one, and we will measure that difference rather than assume it.
Concretely, let $\hat v_t$ be the server's second-moment estimate after round $t$. Let $\Phi_t$ be the known variance the DP noise contributes to it; DP-Adam shows this bias equals $(\sigma C/B)^2$ in the centralized case. Let $\nu$ be a small floor. The debiased estimate is $\tilde v_t = \max(\hat v_t - \Phi_t, \nu)$, and we define the shaping matrix $D_t = \mathrm{diag}(\tilde v_t)^{\gamma/2}$, where $\gamma \in [0,1]$ is a single shaping exponent. Client $i$, holding local update $\Delta_i$, sends
$$u_i = \mathrm{clip}_C\!\left(D_t^{-1}\Delta_i\right) + \mathcal{N}(0, \sigma^2 C^2 I),$$
where $\mathrm{clip}_C$ rescales a vector to norm at most $C$ and $\sigma$ is the noise multiplier. The server forms the pseudo-gradient (the averaged update that the server's optimizer treats as a gradient) as $D_t \bar u$ and feeds it to its adaptive optimizer. Setting $\gamma = 0$ recovers ordinary DP federated Adam.
Because $D_t$ depends only on released noisy aggregates, the sensitivity in the transformed space is still $C$, and the Gaussian mechanism accounting is identical. The noise covariance in parameter space becomes $\sigma^2 C^2 D_t^2$. The motivation comes from a heuristic model in which update magnitudes scale with $\tilde v^{1/2}$: under that model, $\gamma = 1$ equalizes per-coordinate signal-to-noise ratio, and intermediate values trade that equalization against clipping bias.
Relation to Prior Work
Two lines of work motivate the proposal. The first shows that server adaptivity helps under DP. AdDPNFL pairs server-side momentum and scaling with tighter accounting and reports large gains on non-IID CIFAR-10. Fed-SPA combines random-coordinate sparsification with a server Adam, and the Sketched Gaussian Mechanism projects updates into a random sketch before noising and aggregates with AMSGrad. All three keep the noise isotropic, either in parameter space or in a random subspace, and use the adaptive state only to consume noisy updates, never to shape the mechanism. The second line diagnoses what goes wrong. DP-Adam finds that DP noise inflates Adam's second moment and squashes low-signal coordinates, and proposes subtracting the known noise variance. It works only in centralized settings. We reuse its correction as a component, inside the federated server optimizer, and go further by shaping the mechanism. FedDuA supplies evidence that coordinate-wise anisotropy matters for federated server steps, but it has no DP.
Other work constrains the design. Adaptive clipping in interpolation regimes adapts only a scalar clipping radius over time. We adapt the shape of the clipping region, and the two can be combined. LA-LoRA shows that noise interacting with model structure, in its case low-rank factors, changes utility a great deal at fixed $\epsilon$. This supports the view that noise geometry matters, and it gives us a large-model test bed.
On the attack side, three papers supply methods. Inverting Gradients matches gradient directions under a total-variation image prior. Gradient Inversion Attacks Beyond SGD recovers labels analytically from fully connected layers and matches adaptive-optimizer updates. Learning To Invert trains an inversion network that treats defenses as data augmentation, and it breaks Gaussian noise at $\sigma = 0.1$ with MSE 0.012. That last result is the main reason we insist on empirical leakage measurement: a mechanism can have a valid $\epsilon$ and still leak recognizable data at practical noise levels. None of these attacks has been run against noise whose covariance follows a server preconditioner. Such noise could put more noise on the first-layer and classifier coordinates that attacks exploit, or less.
Research Program
The work has three parts.
The first is the mechanism: preconditioned federated Adam as defined above. It includes a warm-up of isotropic rounds, needed before $\tilde v_t$ is informative. It also includes the federated form of the debiasing term. With $m$ sampled clients and client-level noise, the noise variance entering the server's second moment is $(\sigma C)^2/m$ per coordinate, multiplied by the diagonal of $D_t^2$ once shaping is active. The server knows this term exactly and subtracts it.
The second is analysis on the synthetic anisotropic quadratics used by AdDPNFL and FedDuA. There we can compute the stationary error as a function of $\gamma$ in closed form and check whether the optimal $\gamma$ tracks how anisotropic the problem is.
The third is a leakage study. Each defended configuration is attacked by all three attack families. We use a threat model in which the adversary observes a single client's noised update, which is the worst case when secure aggregation is absent or fails. The inversion network is retrained on updates drawn from each mechanism's exact noise distribution, so that it is adaptive in the sense of Learning To Invert.
Baselines
The naive reference point is DP-FedAvg with static clipping and isotropic noise. The file reports its collapse to 48.46% on CIFAR-10 with Dirichlet $\alpha = 0.1$ in AdDPNFL. We will also report non-private FedAvg and federated Adam as utility ceilings.
The strongest existing competitor is AdDPNFL, the adaptive DP method for non-IID data. It reports 73.19% on CIFAR-10 with $\alpha = 0.1$ and 84.45% on MNIST, against 78.79% for DPFedAvg. AdDPNFL uses truncated concentrated DP accounting, so we will rerun every method, AdDPNFL included, under one common Rényi DP accountant. We will use identical client sampling, rounds, local steps, architectures and data partitions, and give each method the same hyperparameter search budget over learning rates, clipping radius, and, for our method, $\gamma$. We quote the published numbers only as sanity checks. Fed-SPA serves as a second strong baseline at $\epsilon = 1$, where it reports 92.65% on MNIST (with $p = 0.05$) and 63% on CIFAR-10. For the large-model setting, LA-LoRA's 60.68% on Tiny-ImageNet with Swin-B at $\epsilon = 1$ is the reference. There we apply preconditioning to the adapter parameters only.
The ablation that isolates the key ingredient is DP federated Adam with DP-Adam debiasing but $\gamma = 0$. It keeps every component except the shaped mechanism.
Beating the baselines means a higher final accuracy, averaged over three seeds with non-overlapping confidence intervals, than both the debiased-isotropic ablation and AdDPNFL at $\epsilon \in \{1, 4\}$ on CIFAR-10 at $\alpha = 0.1$. It also means reconstruction quality that is no better for the attacker at matched $\epsilon$, measured by higher LPIPS and lower PSNR under the adaptive inversion network. A utility gain paired with increased empirical leakage would count as a mixed result, and we would report it that way.
Why not just…?
Why not just apply the DP-Adam correction on the server and keep isotropic noise? This is the most serious threat. If the main loss under DP is the optimizer squashing noisy coordinates, debiasing alone recovers most of the gain. Debiasing, however, fixes only how the optimizer scales steps. It cannot improve the raw signal-to-noise ratio of small-magnitude coordinates, which is set when the noise is added. The $\gamma = 0$ ablation settles this directly. If the best $\gamma$ is near zero on the real benchmarks, the shaping idea is refuted and the contribution reduces to a federated version of DP-Adam.
Why not just use per-layer clipping or an adaptive clipping radius? Per-layer clipping captures coarse geometry at no extra cost, and adaptive clipping handles how update norms change over time. Both are plausible substitutes. We will run a per-layer variant of our method in which $D_t$ is constant within each layer, and a combination of AdaptDP-style radius scheduling with $\gamma = 0$. If per-layer shaping matches coordinate-wise shaping, we will recommend the simpler per-layer version.
Why not just rely on secure aggregation or larger batches to stop inversion? Secure aggregation hides individual updates. mGAN-AI itself notes that it would block that attack. Secure aggregation, though, supplies neither a formal guarantee on the released model nor a remedy for DP's utility loss, which is our main target. Larger batches weaken attacks, yet Inverting Gradients recovers recognizable images from batches of 100, and edge devices often cannot afford large batches. We will report attack success at batch sizes 1, 8 and 32 so that readers can see where shaping matters relative to batch size.
Proposed Experiments
The utility experiments use CIFAR-10 and CIFAR-100 with Dirichlet partitions at $\alpha \in \{0.1, 0.5\}$, EMNIST with natural writer partitions, and Tiny-ImageNet with Swin-B LoRA fine-tuning. We sweep $\epsilon \in \{0.5, 1, 4, 8\}$ at fixed $\delta$ and $\gamma \in \{0, 0.25, 0.5, 0.75, 1\}$, and log per-coordinate signal-to-noise ratios so that the mechanism can be observed directly rather than inferred.
The leakage experiments use CIFAR-10 and ImageNet subsets for images, and CoLA for text through the inversion network from Learning To Invert. We report PSNR, LPIPS, MSE, label-recovery accuracy and sentence-recovery rate. We also break down where the shaped noise falls, measured as total noise variance on the first layer and on the classifier, because this is where we expect the leakage difference to arise.
The decisive checks are three. The first asks whether the best $\gamma$ is bounded away from zero, with an accuracy gain over the debiased-isotropic ablation. The second asks whether the optimal $\gamma$ predicted by the quadratic analysis tracks the measured anisotropy of $\tilde v_t$ across benchmarks. The third asks whether, at matched $\epsilon$ and matched accuracy, the adaptive inversion network's reconstruction quality moves up or down under shaping. That last result tells practitioners something formal accounting cannot: whether redistributing a fixed privacy budget also moves real-world leakage.
References
- DP-Adam: Correcting DP Bias in Adam's Second Moment Estimation
- Differentially Private Federated Learning on Non-iid Data: Convergence Analysis and Adaptive Optimization
- Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimization
- Sketched Gaussian Mechanism for Private Federated Learning
- FedDuA: Doubly Adaptive Federated Learning
- Adaptive Clipping for Differential Private Federated Learning in Interpolation Regimes
- Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models
- Inverting Gradients - How easy is it to break privacy in federated learning?
- Gradient Inversion Attacks Beyond SGD
- Learning To Invert: Simple Adaptive Attacks for Gradient Inversion in Federated Learning
- Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning