In federated learning, many devices or hospitals train one shared model without pooling their data. Each sends only a model update to a server, which averages the updates. Several recent methods improve that average with information about loss-landscape curvature, meaning how sharply the loss rises when the weights move. Each client reports how sharp or how sensitive its local loss surface is, and the server upweights or downweights it accordingly. This helps when clients' data differ or some clients' data are corrupted. These methods were designed without privacy in mind, and that matters in two ways. First, the curvature report is a second channel of information about private data that no privacy accountant charges for. For a fully connected layer it is a sum of squared per-example gradients, which gives an attacker extra equations about the inputs. Second, when updates carry differential-privacy noise (random noise calibrated so that no single person's data can be inferred), weighting clients by curvature can magnify the noise on exactly the clients it trusts most.
We propose to treat curvature-based weighting as a privatized statistic that must be paid for. The client releases a few curvature numbers per layer through the same Gaussian noise mechanism as its update, inside one privacy budget. The server then combines updates with a weight that shrinks automatically toward uniform averaging as the privacy noise grows. Around this design we run a leakage audit. It measures how much an unprivatized curvature report helps known gradient-reconstruction attacks. It also tests whether pushing models into flat regions, the standard fix for poor accuracy under privacy noise, makes their updates easier to invert. The deliverable is a method, a matched-budget comparison, and an empirical answer to when curvature weighting is worth its privacy cost.
Core Claim
Curvature-weighted aggregation under client-level differential privacy fails in two ways that prior work has not examined, and one principled weighting rule addresses both.
Take a layer $l$ and client $k$. Let $\hat F_{kl}$ be the client's privatized curvature summary, here the mean diagonal Fisher information of that layer, where Fisher information is the average squared gradient, a cheap stand-in for curvature. Let $s_l^2$ be the per-coordinate variance of the privacy noise added to that layer's update. We aggregate the privatized updates $\tilde\Delta_{kl}$ as
$$ \theta_l^{\text{new}} = \theta_l + \sum_k \frac{w_{kl}}{\sum_{k'} w_{k'l}}\,\tilde\Delta_{kl}, \qquad w_{kl} = \frac{\hat F_{kl}}{1 + \hat F_{kl}\, s_l^2}. $$
The weight comes from treating each client model as a Gaussian belief whose precision (inverse variance) is its Fisher information, which is the view that motivates Fisher-weighted aggregation. Adding independent noise of variance $s_l^2$ turns precision $F$ into $1/(1/F + s_l^2)$.
The rule has two limits. When $s_l \to 0$ it recovers Fisher-weighted aggregation. When the noise dominates, every weight approaches $1/s_l^2$ and the rule falls back to plain averaging. Plain averaging is then the right choice, because there is little reliable curvature signal left to exploit.
The claim is testable at a matched $(\epsilon,\delta)$ budget, meaning the same formal privacy guarantee for every method, with the curvature report charged to that budget. Under that condition, this rule should beat uniform averaging of flatness-trained clients at moderate $\epsilon$ under non-IID or corrupted data (client datasets that differ in distribution or contain noisy samples), and should tie it at small $\epsilon$. Unaccounted curvature reports, by contrast, should measurably strengthen reconstruction attacks.
What Prior Work Establishes and Leaves Open
Two lines of work supply the curvature-weighting mechanism we privatize.
Leveraging Function Space Aggregation for Federated Learning at Scale (FedFish) derives, from a second-order expansion of the distance between model outputs, that each parameter should be weighted by its diagonal Fisher information. It reports a 7% next-token accuracy gain over FedAvg on Stack Overflow. From Optimization to Generalization: Fair Federated Learning against Quality Shift via Inter-Client Sharpness Matching (FedISM) instead has each client report a single sharpness number, which the server uses to reweight clients. It shows that loss-based reweighting falls into a "fairness trap": it overfits corrupted clients.
Neither method adds privacy noise. Neither asks what the curvature report reveals. Our weight is FedFish's weight corrected for noise, released at FedISM-like low dimension (per layer rather than per parameter).
Three papers motivate the flatness side and serve as the strongest competitors on utility. Make Landscape Flatter in Differentially Private Federated Learning (DP-FedSAM) runs sharpness-aware minimization (SAM, an optimizer that seeks flat regions by minimizing the worst-case loss in a small neighborhood) on each client under client-level privacy. It shows flat models suffer less from clipping and noise, but it aggregates uniformly. Differentially Private Sharpness-Aware Training (DP-SAT) shows that naive SAM under privacy wastes budget on a second gradient. It avoids the extra cost by reusing the previous privatized gradient, which is free under post-processing. We borrow that accounting discipline. Differentially Private Empirical Risk Minimization under the Fairness Lens decomposes the accuracy lost to privacy into terms driven by the Hessian trace. It shows high-curvature groups are hurt most, which is why curvature should inform weighting at all.
Two papers constrain the design. Rethinking the Flat Minima Searching in Federated Learning shows that locally flat client models need not average into a globally flat model under non-IID data. Local flatness alone is therefore not sufficient, and server-side weighting has a role. Data-Centric Defense: Shaping Loss Landscape with Augmentations to Counter Model Inversion argues the opposite of the utility literature: inversion optimizers converge more easily in flat basins, so it deliberately steepens basins around sensitive targets. That paper works in a centralized setting and studies model inversion rather than federated gradient inversion. Its tension with DP-FedSAM is untested, and we test it.
The attack literature supplies our audit tools. SPEAR: Exact Gradient Inversion of Batches in Federated Learning recovers batches exactly from fully connected ReLU gradients. It exploits the fact that roughly half of backpropagated signals are zero, and reports PSNR above 100 for batches up to 25. Gradient Inversion with Generative Image Prior uses a generative model as a prior over images. Gradient Inversion Attacks Beyond SGD matches the updates produced by adaptive optimizers. Learning To Invert: Simple Adaptive Attacks for Gradient Inversion in Federated Learning trains a network to invert gradients and breaks heuristic defenses. Gradient-Leakage Resilient Federated Learning shows that server-side privacy does not stop client-side interception. That result is why we privatize on the client.
Research Program
Leakage mechanism. For a fully connected layer, the gradient is $\sum_i \delta_i x_i^\top$. Here $x_i$ is example $i$'s input to the layer and $\delta_i$ is the backpropagated error for that example. The diagonal Fisher of the same layer is $\sum_i \delta_i^2 \odot x_i^2$, where $\odot$ is element-wise multiplication. The Fisher is therefore a second-moment sum over the same per-example factors. It adds an independent set of equations to the disaggregation problem SPEAR solves. We will formalize when this extra information makes the decomposition identifiable beyond SPEAR's batch limit. We will then build a Fisher-augmented variant of SPEAR and of the generative-prior attack.
Method. Each client trains locally with DP-SAT-style recycled sharpness steps. It computes per-layer mean Fisher values, clips them to a bound $B$, and adds Gaussian noise. That noise is composed with the update noise in one Rényi differential privacy accountant, so curvature is never free. The server forms the weights from the rule in the Core Claim.
We will compare three ways of splitting the budget between update and curvature report:
- a fixed fraction;
- a split chosen on a held-out federation;
- a post-processing-only estimator that infers curvature from squared privatized updates minus the known noise floor, at zero extra budget but with higher variance.
Flatness audit. For every trained model, we will record two quantities: the top Hessian eigenvalue, estimated by power iteration, and attack success. This tests whether flatter federated models are easier to invert, at a fixed $\epsilon$.
Baselines
The naive reference is DP-FedAvg: uniform sample-size averaging of clipped, noised client updates. DP-FedSAM reports 81.90% accuracy for it on IID EMNIST. Every method must clear this to justify any machinery.
The strongest utility competitor is DP-FedSAM with uniform aggregation, from Make Landscape Flatter in Differentially Private Federated Learning. It reports 85.90% on IID EMNIST and 87.70% for its top-k variant. It also reports a generalization gap that falls from 17.35% to 10.75%. It is the strongest baseline because it already captures the flatness benefit under privacy without any weighting. If weighting helps, it must help beyond what local flatness provides.
We will run it on the same Dirichlet partitions, with $\alpha \in \{0.3, 0.6\}$ for EMNIST and CIFAR-10/100. The privacy levels match its reported range, $\epsilon \in [4,10]$, plus $\epsilon = 1$. Client sampling, local steps, and the $\rho$ tuning grid are identical. Beating it means higher accuracy at equal total $\epsilon$, outside seed variance over five seeds, on non-IID or corrupted federations.
The strongest robustness competitor is FedISM's sharpness-matched aggregation from From Optimization to Generalization. It reports a 10.85% gain over FedAvg and a 3.78% gain over FedGA on corrupted clients of RSNA ICH with 20% corruption, without privacy. We will run it in two forms. One sends its sharpness scalar unprivatized, giving its true utility and true leakage. The other sends the scalar through a Gaussian mechanism charged to the budget. That second form is the fair privatized competitor.
FedFish, run on privatized updates with unprivatized Fisher diagonals, serves as the curvature-weighting reference. No reported result found on Lacuna for FedFish under differential privacy.
Two ablations isolate the key ingredient. Running our method with $s_l^2$ set to zero in the weight gives the privatized curvature report without noise-awareness. Running per-client scalars instead of per-layer summaries isolates granularity.
For leakage, the reference attacks are SPEAR and Learning-to-Invert. Learning-to-Invert reports MSE 0.012 on CIFAR-10 under noise $\sigma = 0.1$, against 0.150 for an optimization attack. "Beating" here means that adding unprivatized curvature metadata lowers reconstruction error or extends the recoverable batch size relative to these numbers. It also means that our privatized report shows no such gain.
Why not just…?
Why not just privatize FedISM's single sharpness number with a small slice of budget? This is the most serious threat. A scalar costs almost nothing to release, so it may capture most of the benefit. We include it as a baseline. The ablation that keeps scalar granularity but adds the noise-aware weight separates two possible sources of improvement: weighting granularity, and accounting for noise.
If privatized scalar FedISM with noise-aware shrinkage matches the per-layer version, we will report that a single number suffices. The contribution would then reduce to the shrinkage rule and the leakage audit. The prediction that would distinguish them is that per-layer weighting matters under heterogeneity that is layer-specific, such as feature-level domain shift, more than under label corruption.
Why not just use DP-FedSAM and average uniformly? Under heavy noise our own weighting rule predicts exactly this, so at small $\epsilon$ it is the expected winner, not a strawman. Two results suggest it falls short elsewhere. Rethinking the Flat Minima Searching in Federated Learning shows that local flatness does not transfer to the global model under strong non-IID data, reporting a 10.94% gap over a local-SAM baseline at $\alpha = 0$ on CIFAR-100. FedISM shows uniform aggregation underserves corrupted clients. The $\epsilon$ sweep locates the crossover. If uniform averaging wins everywhere, curvature weighting has no place under privacy, and that is itself the finding.
Why not just weight clients by validation accuracy on server data, as FedVal does? FedVal and RFedLR need a clean server dataset. Siloed medical federations such as RSNA ICH and ISIC 2019 often lack one. We run FedVal with a small held-out clean set as an upper reference that is unavailable in deployment. A proxy-free method closing most of that gap is the bar.
Why not just rely on large batches or gradient pruning to stop leakage? SPEAR reconstructs batches up to 25 exactly and scales to 100. Learning-to-Invert breaks sign compression and pruning. Evaluating Gradient Inversion Attacks and Defenses in Federated Learning needs 90% pruning plus InstaHide and pays roughly 7 points of accuracy (93.37% to 86.10%). Neither heuristic is a substitute for formal accounting.
Proposed Experiments and Decisive Checks
Utility. Experiments use EMNIST and CIFAR-10/100 with Dirichlet partitions, following DP-FedSAM's protocol. RSNA ICH and ISIC 2019 use 10–30% corrupted clients, following FedISM's protocol. We report accuracy at $\epsilon \in \{1,4,8\}$ with total budget matched across all methods. On medical tasks we report accuracy separately on corrupted and clean clients. We also report the top Hessian eigenvalue of the global model.
Leakage. Reconstruction runs on CIFAR-10 and a 64×64 ImageNet subset. Attacks are SPEAR, the generative-prior attack, and Learning-to-Invert, each given (a) the update alone, (b) the update plus unprivatized per-parameter Fisher, (c) the update plus unprivatized per-layer summaries, and (d) our privatized report. We measure PSNR, SSIM, and LPIPS, plus the largest batch size at which recovery is exact.
Decisive checks. The proposal fails if any of the following holds. First, at matched total $\epsilon$ the noise-aware rule does not beat both DP-FedSAM and privatized FedISM on at least the non-IID and corrupted-client settings. Second, the zero-noise-term ablation performs as well, which would mean shrinkage is unnecessary. Third, Fisher-augmented attacks show no gain over gradient-only attacks, which would mean the leakage concern is empty.
A separate result settles the flatness tension with Data-Centric Defense. That result is the correlation between global-model sharpness and attack success across DP-FedAvg, DP-FedSAM, and our method at fixed $\epsilon$. A clear negative correlation would mean flatness-seeking methods need to be audited for privacy beyond their formal guarantee.
References
- Leveraging Function Space Aggregation for Federated Learning at Scale
- From Optimization to Generalization: Fair Federated Learning against Quality Shift via Inter-Client Sharpness Matching
- Make Landscape Flatter in Differentially Private Federated Learning
- Differentially Private Sharpness-Aware Training
- Differentially Private Empirical Risk Minimization under the Fairness Lens
- Rethinking the Flat Minima Searching in Federated Learning
- Data-Centric Defense: Shaping Loss Landscape with Augmentations to Counter Model Inversion
- SPEAR: Exact Gradient Inversion of Batches in Federated Learning
- Gradient Inversion with Generative Image Prior
- Gradient Inversion Attacks Beyond SGD
- Learning To Invert: Simple Adaptive Attacks for Gradient Inversion in Federated Learning
- Gradient-Leakage Resilient Federated Learning
- FedVal: Different good or different bad in federated learning
- Towards Robust Parameter-Efficient Fine-Tuning for Federated Learning
- Evaluating Gradient Inversion Attacks and Defenses in Federated Learning