Blind Precision Aggregation for Private Federated Learning

Federated learning lets many devices train a shared model by sending weight updates to a central server instead of their raw data. Two lines of work have grown up largely in isolation. One asks how the server should combine those updates when devices hold wildly different data — so-called statistical heterogeneity, where naive averaging causes "client drift" — and the best answers weight each device's update by a per-parameter confidence measure, such as a Fisher information matrix (a sensitivity profile saying which weights matter most for that device's data). The other line asks how much private information leaks from the updates themselves, since gradient inversion attacks can reconstruct training images from the vectors clients send. This proposal starts from a simple observation that no existing work has examined: the confidence profiles that make uncertainty-aware aggregation work are themselves a privacy side channel, and they may be a far richer one than the updates they accompany.

The core claim is twofold. First, the per-parameter precision, Fisher, or dropout-rate statistics transmitted by methods like FedFish and BayesADMM materially strengthen gradient inversion, because they hand the attacker exactly the per-parameter magnitude information whose absence makes large-batch inversion under-determined. Second, this side channel can be removed at essentially no cost in accuracy or communication: each client can multiply its own update by its own precision before sending anything, so the server performs the precision-weighted sum without ever seeing a per-client confidence map. Adaptive client weighting survives, but it moves from the server (where it invites manipulation and punishes privacy-defended clients) to the client, where it costs nothing to disclose.

Background and the Open Problem

Under non-IID data, devices optimize different loss surfaces, and simple averaging pulls the global model away from all of them. Three families of fixes exist. Representation-splitting methods like FedRep keep classifier heads local (87.7% on CIFAR-10 with two classes per client, versus 42.6% for FedAvg) but share the representation with plain averaging, which still drifts. Personalization methods like Ditto regularize local models toward the global one (~6% average accuracy gain, ~10% lower cross-device variance) but leave the global aggregator untouched. Precision-weighted methods are the most direct: FedFish aggregates in function space with each client weighted by its diagonal Fisher information (a 7% next-token accuracy gain on Stack Overflow over FedAvg), and BayesADMM derives the same idea from a Bayesian reading of federated ADMM, reporting up to a 7% accuracy boost over FedAvg, FedProx, and FedDyn on heterogeneous CIFAR-100. MetaVD similarly transmits variational dropout statistics and fuses posteriors via a product of Gaussians.

All three precision methods require the client to send its confidence profile alongside its update — doubling payload, in BayesADMM's case, and publishing a per-parameter map of where the client's data is most informative. On the attack side, SPEAR shows that when an adversary can exploit structural sparsity analytically, batch reconstruction becomes exact (PSNR above 100 dB at batch size 25), and GIAS shows generative priors recover images even under 99% gradient pruning. Neither has ever been pointed at a Fisher or precision payload. The one paper that treats Fisher information as a privacy asset, FedDPA, masks high-Fisher parameters and keeps them local under differential privacy (49.75% on CIFAR-10 at Dirichlet α=1, beating DP-FedSAM's 45.07%) — but it then aggregates the remaining parameters with plain DP-FedAvg, discarding the anti-drift benefit entirely. Nobody has asked whether the precision channel leaks, and nobody has designed an aggregator that keeps precision weighting while closing that channel.

Core Claim

Precision-weighted aggregation and privacy are usually treated as a trade-off. We claim they are separable. The server needs the sum of precision-weighted updates, not any individual client's precision. So let each client $k$ compute its local update $\Delta w_k$ and a diagonal precision (inverse-variance or Fisher) estimate $F_k$, and transmit only the product, with normalization handled by a low-dimensional scalar:

$$w_{t+1} \;=\; w_t \;+\; \eta \,\frac{\sum_{k} F_k \Delta w_k}{\sum_{k} \mathrm{tr}(F_k)}$$

where $\mathrm{tr}(F_k)$ is the trace (sum of diagonal entries), sent as a single number so the server can normalize the combined step. The server sees one precision-scaled vector and one scalar per client — the same payload size as plain FedAvg, half of BayesADMM's — and the fixed point is (up to normalization) the same precision-weighted consensus that gives FedFish and BayesADMM their accuracy gains under heterogeneity. We call this blind precision aggregation: the weighting is adaptive in the sense that low-confidence parameters are automatically discounted, but the adaptation happens entirely client-side.

This relocation also dissolves two further tensions catalogued in the literature. Server-side adaptive weighting schemes that score clients by update geometry or reported metrics can be gamed by malicious clients and, as CENSOR and DCS² imply but never test, will systematically down-weight clients whose updates look "divergent" because they applied a privacy defense. In blind precision there is no server-visible trust signal to hijack, in the spirit of the active-manipulation attack LIA-SA (which recovers label distributions with 100% accuracy from aggregated gradients by planting a "fishing" Batch Normalization layer), and no geometric anomaly signal to punish defended clients. We will verify both claims experimentally rather than assert them.

Research Program

The work has three parts. Part one is an attack study that quantifies the side channel. We extend the optimization-based inversion of GIAS and the analytical inversion of SPEAR to consume, in addition to $\Delta w_k$, the diagonal $F_k$ (or dropout rates, in the MetaVD variant). Since a diagonal Fisher is approximately the square of per-parameter gradients, it constrains the inversion's magnitude estimates per parameter and should shrink the batch-size regime in which attacks degrade — the working hypothesis is that attacks given $(\Delta w_k, F_k)$ succeed at batch sizes where update-only attacks collapse, and that FedFish-style payloads therefore leak strictly more than FedAvg payloads. This is a measurement contribution in its own right, since no published attack has been run against any uncertainty-aware aggregator.

Part two is the defense design. Blind precision aggregation needs three components worked out: how clients estimate $F_k$ cheaply (squared local gradients, as in FedFish, avoiding FedDPA's per-epoch full FIM computation); how the normalization trace is protected (it is a single scalar per client per round, and we will test whether an active server can extract class-frequency information from traces alone, mirroring LIA-SA's methodology); and how the method composes with existing update-level defenses. The last point is genuinely open: CENSOR samples descent directions orthogonal to the true gradient, and DCS² projects onto decoy subspaces — both assume the transmitted vector is the gradient, whereas blind precision transmits a rescaled version, so we must check that the defense's zero-correlation or subspace guarantees survive the rescaling, and re-derive them where they do not. A second defensive layer, borrowed from FedDPA, retains the very highest-Fisher parameters locally instead of transmitting them at all — but, unlike FedDPA, we keep precision weighting on everything that is shared, so the anti-drift property is preserved.

Part three is analysis. We will characterize when the client-side product $F_k \Delta w_k$ is information-equivalent to the pair $(F_k, \Delta w_k)$ — it is not, in general, since a diagonal rescaling destroys the per-parameter gradient magnitudes the attacker needs — and when a determined adversary can partially invert the rescaling (for example, using a generative prior meta-learned across rounds, GIML-style, to estimate typical $F_k$ values). This sets honest bounds on what the defense guarantees: blind precision should reduce leakage to FedAvg levels, not to zero.

Proposed Experiments

We use the benchmarks already standard across these literatures. Utility is measured on heterogeneous CIFAR-100 and CIFAR-10 partitioned by Dirichlet concentration $\alpha \in \{0.1, 0.3, 1.0, 5.0\}$ and by restricted classes per client, on FEMNIST (natural writer-level non-IID), and on TinyImageNet for larger models, using the DH-Benchmark framework for the non-IID partitions and the MetaVD codebase for baseline implementations of FedAvg, FedProx, and FedBE. Metrics are global test accuracy, per-client accuracy variance (following Ditto's fairness metric), communication payload per round, and, under attack, reconstruction PSNR/SSIM and label-recovery rate.

The decisive checks are four. First, utility: blind precision should recover most of the precision-weighting gain that BayesADMM demonstrates (up to a 7% accuracy boost over FedAvg on heterogeneous CIFAR-100) at FedAvg-level payload. Second, leakage: running GIAS and SPEAR against FedFish (update + Fisher sent separately) versus blind precision (product only) at matched batch sizes, with GIAS's reported 15.58 dB PSNR at batch size 32 on 64×64 ImageNet as the reference scale; success means the side channel measurably helps the attacker against FedFish and stops helping against ours. Third, active-server robustness: an LIA-SA-style fishing model against both designs, where the reference point is LIA-SA's 100% label recovery on CIFAR-10/100 under secure aggregation. Fourth, composition: stacking CENSOR or DCS² on top (CENSOR holds ~86% CIFAR-10 accuracy while defending; DCS² holds 65.68% on TinyImageNet and drives reconstruction SSIM on CIFAR-10 to 0.15), and measuring whether blind precision preserves those defenses' guarantees better than server-side precision weighting does — including whether alignment-based client weighting (in the spirit of MOTION's trajectory analysis) unfairly strips defended clients of influence, which we measure as the weight share received by defended versus undefended clients.

Baselines

The "do nothing" reference is plain FedAvg on the same partitions, whose collapse under heterogeneity is well documented — 42.6% on CIFAR-10 with two classes per client in the FedRep evaluation — and which our method must beat by a wide margin for the precision machinery to be worth anything at all. The strongest utility competitor is FedFish, which performs exactly the precision-weighted aggregation we are trying to protect; we will re-implement it from its paper on CIFAR-100, FEMNIST, and TinyImageNet at the same Dirichlet partitions, compute budget, and tuning grid as ours (its published gains are a 7% next-token improvement on Stack Overflow; no reported result found on Lacuna for FedFish's CIFAR-100 accuracy, so we treat its published algorithm, not a published number, as the bar). The comparison is fair by construction because blind precision uses the same $F_k$ estimator, the same local training, and the same server learning-rate search; the only difference is where the multiplication happens. Beating FedFish means matching its accuracy within about 1% while cutting its attack-surface metric (reconstruction PSNR under GIAS/SPEAR given the transmitted payload) by a large margin and halving payload. The strongest privacy-utility competitor is FedDPA, whose reported numbers — 49.75% on CIFAR-10 at α=1 and 77.27% on FEMNIST at ε=4 — anchor the DP-based alternative; we run it at its published settings on our partitions, and beating it means higher accuracy at equal or better attack resistance without a formal privacy budget. Two ablations isolate the key ingredient: first, server-side precision (sending $\Delta w_k$ and $F_k$ separately, FedFish-style, but with our normalization) versus client-side precision with identical weighting mathematics — this isolates where the multiplication happens as the sole cause of any leakage difference; second, Fisher masking alone (FedDPA-style retention of high-Fisher parameters with plain averaging of the rest) versus masking plus blind precision, which isolates the contribution of keeping precision weighting on shared parameters.

Why not just…?

Why not just add differential-privacy noise to the precision vectors before sending them? This is the most serious alternative, and we treat it as such. The evidence against it is that SPEAR bypasses basic DP noise and clipping for exact analytical inversion, and RESFL documents that DP noise disproportionately degrades underrepresented groups — so a noisy precision channel risks the worst of both worlds: residual leakage and skewed aggregation. But the honest position is that no one has measured how much noise the precision channel actually needs, and if a lightly-noised Fisher vector matches blind precision on both utility and attack resistance, the simpler design should win. Our experiments include exactly this sweep: DP noise on the $F_k$ channel at several budgets, evaluated on the same accuracy and PSNR metrics, so the comparison is decided empirically rather than by assertion.

Why not just use secure aggregation so the server never sees individual precision vectors? LIA-SA demonstrates that secure aggregation is not a defense against an active server: by manipulating a Batch Normalization layer before distribution, the server recovers exact per-client label distributions with 100% accuracy from the encrypted aggregate. A precision payload under secure aggregation leaves the same linear-disentanglement surface available. Blind precision removes per-client payloads from the wire entirely, so there is no per-client quantity to fish for — but we will run the LIA-SA attack against both designs rather than assume this, since the trace scalar is itself a per-client quantity an active server could probe.

Why not just drop precision weighting and defend plain FedAvg with CENSOR or DCS²? This works if you are willing to give up the heterogeneity gain, but the whole motivation for precision weighting is that plain averaging drifts badly under non-IID data — the gap BayesADMM (up to 7% over FedAvg on heterogeneous CIFAR-100) and FedFish exist to close. We will run CENSOR-equipped FedAvg as a baseline at severe skew (α=0.1), and if it matches blind precision's accuracy there, that would genuinely undercut the proposal and we would report it. Our expectation, grounded in FedDPA's own result that masking without precision weighting leaves utility well below undefended levels, is that defended FedAvg pays a utility tax at high heterogeneity that blind precision does not.

Why not just personalize everything with FedRep or Ditto and stop sharing sensitive updates? Personalization reduces but does not eliminate what crosses the wire: FedRep still averages shared representations and Ditto still sends updates to maintain the global model, and SPEAR shows shared fully connected layers admit exact recovery. Moreover, FedFish documents a client-server loss barrier from plain averaging of shared representations that personalization alone does not fix. Blind precision is complementary: it can sit underneath a FedRep-style split, weighting the shared-representation updates, and we will test that composition on FEMNIST and CIFAR-10.

Decisive Checks and Closing

The proposal stands or falls on three numbers. If the attack study shows that FedFish-style payloads do not improve inversion over update-only payloads at matched batch sizes, the premise fails and the defense is solving a non-problem — this is a genuine risk, since no prior measurement exists in either direction. If DP-noised precision matches blind precision on utility and leakage, the simpler design wins and we should say so. And if blind precision loses more than about 1% accuracy relative to server-side FedFish on heterogeneous CIFAR-100, the client-side relocation is not free and the trade-off is real. If instead all three checks come out as hypothesized, the result is a federated aggregator that keeps the documented anti-drift benefits of precision weighting under statistical heterogeneity, halves communication relative to Bayesian variants, and closes a privacy channel that the uncertainty-aware aggregation literature has been opening without noticing — with adaptive client weighting recast as a purely local computation that no curious or malicious server can observe, manipulate, or turn against the clients who protect themselves.