Institutions that legally cannot pool their data — hospitals, courts, banks — increasingly adapt shared foundation models through federated fine-tuning: each site keeps its records local, attaches a small trainable "adapter" module to a large frozen backbone, and sends only that module to a coordinating server. This cuts communication dramatically — synchronizing lightweight adapters rather than full models reduced communication overhead by over 98% in multilingual translation — and it is widely assumed that transmitting roughly 1% of the parameters also protects the underlying records. That assumption is in a strange state. The strongest attacks on adapter gradients, MineGrad and ReCIT, both require a malicious server that tampers with the frozen backbone before dispatching it, something a client can defeat by hashing the received model against the official public release. Meanwhile MineGrad reports that ordinary optimization-based inversion "completely failed" in the adapter setting, and the one successful passive reconstruction, UTR, is text-only. For the case real silos actually face — an honest-but-curious server, a verified frozen backbone, images — nobody has measured whether adapter updates leak, and the defenses on offer all cost something silos cannot spare: differential privacy destroys utility, encryption carries heavy engineering and key-management assumptions, and Shadow Defense requires each hospital to train and run a StyleGAN3 locally.
This note proposes to settle the question and close the gap. We will audit passive leakage of adapter updates under verified backbones, explain the result geometrically, and — if leakage is real — neutralize it with a decoy-entangled release rule that needs no noise, no cryptographic keys, and no locally trained generative model. The urgency is sharpened by a tension already visible in the literature: Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classification shows that out-of-domain medical silos pay roughly a 4% balanced-accuracy tax per order of magnitude of parameter reduction, so silos have no accuracy budget left to spend on privacy.
Core Claim
The claim has two halves. First, the privacy of low-rank adapter updates under a passive adversary is not an all-or-nothing fact but a geometry: an adapter gradient is an average of per-sample outer products between upstream activations and downstream errors, so it is a low-dimensional linear sketch of the batch, and its invertibility should be governed by three measurable quantities — the adapter rank $r$, the batch size $B$, and how entangled (mutually aligned versus orthogonal) the per-sample sketches are, the same quantity Concealing Sensitive Samples against Gradient Leakage in Federated Learning identified as the decisive factor for full gradients. Second, because a passive attacker with a verified public backbone must regularize inversion with a public generative prior — the strategy that took ImageNet reconstruction from 17.49 dB to 20.05 dB PSNR in GIFD, building on Gradient Inversion with Generative Image Prior — the defender can turn that necessity into a trap: release an update that is equally consistent with a batch of decoy inputs drawn from the attacker's own public manifold, so the attacker's optimizer converges toward data that was never private.
Formally, the passive attack we audit is
$$ x^{\star} = \arg\min_{x}\; \mathcal{D}\!\left(g(x;\theta_{\text{pub}}),\, g_{\text{obs}}\right), $$
where $g(x;\theta_{\text{pub}})$ is the vector of adapter gradients a candidate dummy batch $x$ produces through the public, hash-verified backbone $\theta_{\text{pub}}$, $\mathcal{D}$ is cosine or $L_2$ distance as in standard gradient inversion — reconstructing private data by gradient descent on dummy data until its gradients match the intercepted ones — and $g_{\text{obs}}$ is the update actually transmitted. Optimization runs in intermediate feature space with a generative prior, following GIFD. The defense releases
$$ \Delta_{\text{rel}} = \Delta_{\text{priv}} + \lambda\,\Delta_{\text{decoy}}, $$
where $\Delta_{\text{priv}}$ is the client's true adapter update on its real batch, and $\Delta_{\text{decoy}}$ is an update computed on decoy inputs sampled from a public generative model (BigGAN or StyleGAN2 — the exact priors a passive attacker must use), labeled by the client's current local model so the decoy task signal is benign. The scalar $\lambda$ is tuned locally so the excess loss on real data stays small; the released update is by construction at least as consistent with public decoys as with private records, so any prior-regularized inversion is pulled toward the decoys. A second, orthogonal ingredient restructures local batching so per-sample contributions to the release are deliberately aligned rather than separable, directly attacking the low-entanglement condition that Concealing Sensitive Samples shows inversion depends on.
Research Program
The audit comes first, because its outcome shapes everything else. We extend feature-domain generative inversion to adapter gradients of a frozen ViT-B/16 across two regimes: in-domain silos (CIFAR-100, Caltech101, Flowers102) where the attacker's natural-image generative prior matches the silo data, and out-of-domain silos (Fed-ISIC2019, HAM10000, the setting of Probing PEFT) where it does not. This second regime is scientifically the most interesting: MineGrad engineered its statistical sieves from public proxy statistics, which a server simply does not have for histopathology, so we will learn whether domain mismatch between the attacker's prior and the silo's data is itself a meaningful shield — and whether a column-space constraint in the spirit of ReCIT's token extraction, applied to the adapter down-projection, partially substitutes for the missing prior. We also test longitudinal passive profiling across rounds, since Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning shows successive updates let an attacker isolate user-specific features rather than generic class appearance, and we test whether the global adapter of a dual-adapter scheme — the shared half of Dual-Personalizing Adapter for Federated Foundation Models, which transmits about 0.06% of parameters and assumes a benign server — leaks local data even though the local adapter is withheld.
Alongside the audit we build the geometric account: closed-form expressions for the information content of the adapter sketch as a function of $r$, $B$, and the spectral alignment of per-sample outer products, calibrated against observed attack success. This tells us where in design space passive inversion provably collapses — possibly vindicating "privacy by sparsity" in vision, which MineGrad disproved only for active poisoning — and where the defense must do real work.
The defense is then evaluated on exactly the regimes the audit identifies as leaky, and — critically for silo adoption — with zero additional client-side training: decoys are sampled from public generative models, not trained locally, which is what distinguishes this from Shadow Defense, whose local StyleGAN3 imposes exactly the compute burden edge silos lack. A text leg extends the audit to FEDLEGAL's privacy task: FEDLEGAL demonstrated that DLG and TAG invert sensitive legal text from full fine-tuning of RoBERTa (no quantitative recovery score is reported), and left adapters unexamined even though adapters are the obvious answer to its ~6 GB-per-client communication bottleneck; we will run a UTR-style word-bag inversion against adapter-based federated RoBERTa on the FEDLPA silos and check whether decoy-entanglement (with a public language-model prior in place of a GAN) blunts it.
Baselines
The naive reference point is undefended federated adapter averaging in the style of FedDPA: a frozen backbone with a shared adapter aggregated across silos. This gives both the leakage ceiling (what a passive attacker extracts with no defense) and the utility floor; on the Flan task federation FedDPA reports ROUGE-1 of 80.22 on local tasks and 77.12 on unseen shift tasks, and on the medical silos the relevant reference slope from Probing PEFT is roughly a 4% balanced-accuracy drop per order of magnitude of parameter reduction (no absolute balanced accuracies on Fed-ISIC2019 are reported in the file for its LoRA/VPT/SBA variants). Every method is run on identical silo partitions, the identical hash-verified backbone, with equal communication bytes and an equal tuning budget (fixed grid over learning rates, ranks, and defense hyperparameters), and every attack receives the public backbone, the public generative prior, and an equal reconstruction-iteration budget.
The strongest existing defense competitor is SHE-LoRA, which defeats the DAGER inversion attack while encrypting only 0.03% of parameters and cuts communication by up to 99.71% and encryption runtime by up to 99.87% relative to full homomorphic encryption. We will run it unchanged on our silos and attacks; beating it means matching its utility and confidentiality without its shared-decryption-key assumption (which one colluding client breaks, as its authors note) and without ciphertext-management engineering. The second defense baseline is CENSOR, orthogonal-subspace cold-posterior sampling, which holds about 86% CIFAR-10 accuracy with 0.00236% overhead on full gradients but has never been tested where trainable parameters are confined to rank $r \le 64$; this run doubles as the ablation that isolates our key ingredient, because if orthogonal sampling alone survives the low-rank bottleneck, the decoy mechanism is unnecessary. Attack-side baselines are PEFTLeak, the active attack of MineGrad / Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning — 85.9% patch recovery at batch 32 on CIFAR-100 with 0.88 SSIM, and 72.6% at batch 128 — which serves as the leakage ceiling even though it requires backbone tampering our threat model excludes, and GIFD at 20.05 dB PSNR on ImageNet as the passive full-gradient reference. Ablations of the defense itself are decoy mixing without entangled batching, entangled batching without decoys, and a $\lambda$ sweep; if pure entangled batching alone closes the leak, we will report that honestly and drop the decoy machinery.
Why not just…?
Why not just add differential-privacy noise to the adapter updates? The file's evidence is consistent that DP stops inversion only at noise levels that destroy the utility these silos cannot spare: UTR found DP effective at higher noise levels but "severely impacting the model's performance," SHE-LoRA observes that LoRA's low-rank matrix multiplications amplify injected noise, and Shadow Defense shows DP is indiscriminate, obscuring task-critical diagnostic signal like a blindfold. We will nevertheless run DP-SGD at matched utility as a baseline; the question is empirical, and our decoy mechanism is a targeted attempt to get DP-like ambiguity without DP-shaped damage.
Why not just use homomorphic encryption, as SHE-LoRA already does? This is the most serious alternative, and we treat it as such by running it as the headline baseline. Three things keep a crypto-free defense in scope. First, SHE-LoRA assumes all clients share one decryption key, so a single compromised client colluding with the curious server collapses the boundary — a failure mode our defense does not have. Second, homomorphic encryption addresses confidentiality but not the audit question, which stands regardless: if passive inversion on adapter gradients fails anyway in out-of-domain silos, encryption there is wasted cost. Third, our defense composes with SHE-LoRA rather than competing, covering deployments that will never manage keys or ciphertext expansion. If SHE-LoRA matches our utility at acceptable overhead under our attacks in our silos, we will say so plainly.
Why not just share logits and distill instead of gradients? FedED and ADAFEDSELECKD transmit ensemble predictions rather than weights, but both need proxy data matched to the silo domain — precisely what authentic legal and histopathology silos lack, as FEDLEGAL emphasizes — and AdaFedSelecKD still synchronizes a shared global adapter, which is exactly the object whose leakage we audit. Soft predictions are also not leak-free, as the membership-inference results noted in SHE-LoRA indicate.
Why not just train with large local batches, since inversion is under-determined? Measured directly: MineGrad recovers 72.6% of patches at batch 128 on CIFAR-100, and UTR exceeds 99% ROUGE-1/2 reconstruction at batch 128. Batch size is not a defense; our geometry analysis will say precisely why, in terms of sketch rank and entanglement.
Why not just run CENSOR on the adapter gradients? Possibly nothing — and that is why it is an ablation rather than an objection. CENSOR explicitly relies on the vast dimensionality of full parameter spaces to find loss-minimizing orthogonal directions; in a rank-64 adapter subspace those directions may not exist. If they do, CENSOR wins and we report it; if they do not, that result is itself one of the paper's findings and motivates the decoy mechanism.
Decisive Checks
Four results decide the proposal. First, the headline audit number: passive patch recovery at batch 32 on the CIFAR-100 adapter setup, compared against PEFTLeak's active 85.9%; if passive recovery lands within roughly ten points of the active ceiling, the "verified backbone is safe" belief is false for vision, and if it collapses, we will have quantified the first real safety margin for adapter federated learning — either outcome is publishable, but only one is good news. Second, the domain-shift shield test: recovery on Fed-ISIC2019 with a natural-image generative prior, reported as patch recovery and PSNR against ground truth, with the decoy-consistency check (PSNR of the attacker's reconstruction against the planted decoy) for the defense arm. Third, the utility ledger: defended balanced accuracy on Fed-ISIC2019/HAM10000 within about 2% of the undefended adapter baseline, i.e., well inside the ~4%-per-decade tax, at matched communication bytes. Fourth, the CENSOR-in-low-rank ablation and the FEDLPA text leg: whether adapter exchange stops the legal-text inversion that FEDLEGAL demonstrated for full fine-tuning, and whether decoy entanglement with a public language-model prior blunts the UTR-style attack that currently exceeds 99% ROUGE. Together these checks either hand silos a cheap, verifiable privacy story or hand the community a measured account of exactly when adapter updates betray them.
References
- MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning
- ReCIT: Reconstructing Full Private Data from Gradient in Parameter-Efficient Fine-Tuning of Large Language Models
- Reconstructing Training Data from Adapter-based Federated Large Language Models
- Shadow defense against gradient inversion attack in federated learning
- Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classification
- Concealing Sensitive Samples against Gradient Leakage in Federated Learning
- GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization
- Gradient Inversion with Generative Image Prior
- Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning
- Dual-Personalizing Adapter for Federated Foundation Models
- FEDLEGAL: The First Real-World Federated Learning Benchmark for Legal NLP
- SHE-LoRA: Selective Homomorphic Encryption for Federated Tuning with Heterogeneous LoRA
- CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Sampling
- Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning
- FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction
- Adapter-Based Selective Knowledge Distillation for Federated Multi-Domain Meeting Summarization