In federated learning, many clients train one shared model without pooling their data. Each round, the server sends out model weights, clients compute updates on their private data, and the server combines those updates. Homomorphic encryption, meaning encryption that still allows sums to be computed on the hidden values, lets the server add up client updates without seeing any single one. These encrypted schemes are increasingly cheap. Some encrypt only a sliver of the parameters, and some compress updates before encryption. Almost all of them assume an "honest-but-curious" server, one that follows the protocol but reads whatever it is shown.
The strongest privacy attacks of the last few years break exactly that assumption. They do not invert gradients passively. Instead the server sends doctored weights: it rescales a single batch-normalization layer, sharpens one class boundary, or turns frozen layers into identity maps. The resulting updates then leak individual training examples, and they leak them even after secure summation. This note proposes a single-server protocol that takes weight selection away from the server. Clients jointly decrypt only a verified aggregate and compute the next model themselves. The server's remaining levers are forging the sum, choosing which clients count, and planting fake clients. A cheap linear checksum, a decryption-time membership rule, and update norm bounds constrain those levers. The central empirical question is whether fake clients with bounded updates can still steer the model into a trap. The protocol is only worth having if they cannot do so in practical numbers of rounds.
Core Claim
Every active reconstruction attack catalogued by Dealing Doubt works through one of three channels:
- The server chooses the weights clients train on. This covers trap weights, fishing, batch-normalization manipulation, PEFTLeak, and sending different models to different clients.
- The server chooses which ciphertexts enter the sum. For example, it can sum the victim's ciphertext with only its own fake updates and subtract them afterwards.
- The server corrupts the sum itself.
The claim is that, in threshold-decryption federated learning, the first channel can be removed at almost no cost. The second and third can be closed with linear-only verification. Linear verification suffices because aggregation is a sum. So the non-linearity limitation that confines vFHE, a checksum scheme for encrypted matrix products, does not bite here. What remains is a quantifiable residual: a coalition of $f$ fake clients, each with update norm at most $C$, among $n$ participants, can move the averaged model by at most $fC/n$ per round. Planting a trap that requires a parameter displacement of size $D$ therefore needs at least
$$T \ge \frac{nD}{fC}$$
rounds. Honest updates also pull against the trap, so this bound is optimistic for the attacker. The program measures $D$ for each known attack and checks whether real training pushes back fast enough to keep traps from ever forming.
Research Program
The protocol builds on threshold multiparty homomorphic encryption as in Secure Aggregation in Federated Learning using Multiparty Homomorphic Encryption. That work shares the decryption key among clients and compresses updates while keeping them addable, but it is analyzed only against a semi-honest server. We reuse its encryption layer and add three components.
The first component is client-side model derivation. After threshold decryption, every client holds the same plaintext average and applies the public update rule to the previous model, starting from a publicly committed initialization. Clients cross-check a hash of the resulting model through a gossip exchange. The server never transmits weights. This makes model inconsistency attacks and one-shot weight doctoring impossible by construction. The idea of client-side unmasking comes from Janus, which gives clients the final unmasking step but relies on two servers that must not collude. We drop the second server and the non-collusion assumption.
The second component is a blind linear checksum on the aggregate, adapted from vFHE. After all ciphertexts are committed, clients derive a random vector $h$ that the server never learns. Each client additionally encrypts the tag $\langle h, \Delta_i \rangle$, where $\Delta_i$ is its update. Upon decryption, clients check that the sum of tags equals $\langle h, \sum_i \Delta_i \rangle$. A server that adds any offset $\delta$ to the aggregate is caught unless it guesses an offset with $\langle h,\delta\rangle = 0$.
The third component is a membership and norm rule enforced at decryption. Clients release their decryption shares only if two conditions hold. The aggregate must cover a signed list of at least $k$ registered clients that includes themselves. And each included update must carry a lightweight proof that it is clipped to norm $C$. The norm check is a much smaller statement than the full pruning-compliance proofs in Preserving Privacy and Security in Federated Learning, whose server takes about 3.5 minutes to verify proofs from 100 users. The membership rule turns the "isolate the victim" attack into a requirement to control $k-1$ registered identities. Defending against fake identities at registration is treated as an explicit assumption, not something this protocol solves.
We also make the protocol compatible with selective encryption in the style of SHE-LoRA and Evaluating Selective Encryption Against Gradient Inversion Attacks. Both encrypt only the parameters judged most revealing, and both assume an honest server. The plaintext portion of each update is committed and covered by the same checksum, so even unencrypted columns cannot be forged. Finally, we develop the slow-steering attack. A coalition of $f$ fake clients optimizes clipped updates over many rounds to drive the batch-normalization parameters toward the LIA-SA trap, or toward the fishing boundary or PEFTLeak identity configuration. We record how many rounds each attack needs as a function of $f/n$ and $C$.
Baselines
The naive reference point is plain federated averaging with no protection, together with standard secure aggregation without verification, both facing an active server. Breaking Secure Aggregation shows how badly the second fares: its LIA-SA attack recovers labels through secure aggregation with 100% accuracy on CIFAR-10, CIFAR-100, and ImageNet. Reproducing this number establishes that our attack implementations work.
The strongest efficiency baseline is the semi-honest threshold-encryption protocol of Secure Aggregation in Federated Learning using Multiparty Homomorphic Encryption. It reports 95.20% CIFAR-10 accuracy against a 95.48% uncompressed, non-secure baseline. We run it with the same model, client count, compressor, and rounds, adding only our three components. Beating it means staying within 0.5 points of its accuracy and within 10% of its wall-clock and bandwidth cost, while reducing LIA-SA label recovery from 100% to chance and fishing and PEFTLeak reconstruction to the level of passive attacks. For selective encryption we use SHE-LoRA under identical LoRA settings on SST-2 and GLUE tasks. It reports defeating DAGER, a passive text-reconstruction attack, while encrypting 0.03% of parameters. We test whether that protection survives once an active server is allowed, and whether it holds again once our components are added.
The strongest defense aimed specifically at active attacks is OASIS. It defends with client-side image augmentation and reports at most a 1.5% accuracy drop on CIFAR-100, with reconstruction PSNR (peak signal-to-noise ratio, an image-fidelity score) falling to about 15–20 dB on ImageNet under the attacks it tests. It runs on the same vision data and attacks, with augmentation strength tuned on a held-out validation split using the same tuning budget we give our clipping bound $C$. Janus serves as the verifiability baseline. It reports 93.18% MNIST accuracy against 94.1% without secure aggregation, and we evaluate it both with honest servers and with the two servers colluding.
Two ablations isolate the key ingredient. In the first, the server still sends weights, but clients verify a consistency hash among themselves. In the second, clients derive the model locally, but the checksum and membership rule are disabled.
Why not just…?
Why not just have clients check that they all received the same model? Gossiping a model hash defeats the inconsistency attacks that motivate Janus. It does nothing against attacks where everyone receives the same doctored model, which is exactly how LIA-SA and fishing operate. Our first ablation implements this check. We expect it to leave LIA-SA label recovery at 100%. If it does not, local model derivation is unnecessary and the proposal shrinks to the checksum.
Why not just run a detector on received weights that flags suspicious structure such as extreme batch-norm scales or identity-like layers? This is the most serious threat to the proposal, because it is cheap and MineGrad itself suggests client-side parameter checks would block PEFTLeak. Its weakness is that it is heuristic. A server that knows the detector can search for traps that stay inside its thresholds. We will build the best detector we can, then mount adaptive attacks constrained to evade it. If adaptive attacks fail to recover more than passive inversion does at acceptable false-positive rates, the detector wins and we will say so. Our protocol's guarantee does not depend on enumerating trap shapes.
Why not just add local differential privacy? Local differential privacy means each client adds noise to its own update before sending it. Fishing for User Data finds that it stops fishing only at a severe cost in accuracy. SHE-LoRA reports that LoRA's matrix products amplify the noise. Dealing Doubt notes that fake participants dilute distributed noise. We include local differential privacy at noise levels matched to equal reconstruction PSNR and compare accuracy at that level.
Why not just augment inputs as OASIS does? OASIS is specific to vision and does not address label leakage through batch normalization or tabular data. CIC-IDS2017 network-flow records and text are where we expect it to fail. The vision runs will show whether it holds against large-batch fishing.
Proposed Experiments
The vision experiments use CIFAR-10, CIFAR-100, and ImageNet at batch sizes up to 256. They test against LIA-SA, class and feature fishing, PEFTLeak on a LoRA-tuned vision transformer, and model inconsistency attacks. Leakage is measured by label-recovery accuracy, fraction of reconstructed patches, PSNR, and LPIPS, a learned measure of perceptual image similarity. For reference, PEFTLeak reports reconstructing 85.9% of CIFAR-100 patches at batch size 32 and 72.6% at batch size 128.
The text experiments use LoRA fine-tuning on SST-2 and GLUE tasks against DAGER and LAMP, measured by ROUGE-1. CIC-IDS2017 provides a tabular regime. There, EdgeDetect, which combines binarized updates with additive encryption under an honest-server assumption, reports 15.1 dB PSNR under passive inversion and 98.0% accuracy.
Cost is reported as client and server wall-clock time, bytes per round, and verification overhead per round, measured against the threshold-encryption baseline. The vFHE checksum's reported overhead of 2.1–2.2% on a 64×64 encrypted matrix product sets the order of magnitude we expect.
The decisive check is the slow-steering curve. For each attack we measure its required displacement $D$ and plot rounds-to-trap against fake-client fraction from 1% to 20%, at clipping bounds chosen to cost under 1 point of accuracy. If traps form within a typical training horizon of a few hundred rounds at 5% fake clients, the protocol only relocates the problem to identity admission, and the paper reports that bound as its main result. If honest drift prevents traps from forming at all fractions we test, the protocol gives single-server, malicious-server protection against active inversion for roughly the cost of a semi-honest protocol.
References
- Dealing Doubt: Unveiling Threat Models in Gradient Inversion Attacks under Federated Learning, A Survey and Taxonomy
- vFHE: Verifiable Fully Homomorphic Encryption with Blind Hash
- Secure Aggregation in Federated Learning using Multiparty Homomorphic Encryption
- Janus: Dual-Server Multi-Round Secure Aggregation with Verifiability for Federated Learning
- Preserving Privacy and Security in Federated Learning
- SHE-LoRA: Selective Homomorphic Encryption for Federated Tuning with Heterogeneous LoRA
- Evaluating Selective Encryption Against Gradient Inversion Attacks
- Breaking Secure Aggregation: Label Leakage from Aggregated Gradients in Federated Learning
- OASIS: Offsetting Active Reconstruction Attacks in Federated Learning
- MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning
- Fishing for User Data in Large-Batch Federated Learning via Gradient Magnification
- EdgeDetect: Importance-Aware Gradient Compression with Homomorphic Aggregation for Federated Intrusion Detection