Federated fine-tuning of large pretrained models usually relies on low-rank adaptation (LoRA). LoRA adds a small trainable update $BA$ to each frozen weight matrix: $A$ compresses the layer's input into a few dimensions, and $B$ expands that back out. Clients send only these small matrices, which keeps communication cheap. Two results in the literature now pull this design in opposite directions. First, freezing $A$ to a shared random matrix and training only $B$ makes differential privacy (calibrated noise that formally bounds what any single example can reveal) work well, and it halves communication. Second, an active server that chooses the frozen weights, including that same $A$, can turn small adapter updates into near-perfect copies of client images. This note proposes a simple change. Each client draws its own down-projection $A_i$ from a seed the server never sees, trains it locally, and never transmits it. Only $B$ is shared, noised and averaged.
The idea, put in two sentences: the matrix that decides what part of the input the shared update "sees" should be a private key held by each client, not a public setting chosen by the server. This removes the lever that the strongest known adapter attack pulls, keeps the noise-friendly linear structure of freeze-$A$ training, and turns each client's $A_i$ into a personal input encoder, which helps under non-IID data rather than hurting.
Core Claim
Consider a client that sends the gradient of $B$ for one adapted layer with $A$ fixed:
$$\nabla_B \mathcal{L} = \sum_{j} \delta_j \,(A_i h_j)^\top .$$
Here $h_j$ is the frozen backbone's input to that layer for example $j$, $\delta_j$ is the backpropagated error at the layer's output, and $A_i$ is the client's down-projection. The server sees only the left-hand side. Its view of the private inputs therefore passes entirely through $A_i h_j$.
When $A$ is public and server-chosen, the server can build $A$ as a set of filters that isolate individual patches. That is exactly how PEFTLeak works in Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning. The attack poisons the frozen ViT layers into identity maps, configures adapter down-projections as statistical sieves using public data, and recovers 85.9% of patches on CIFAR-100 at batch size 32 with SSIM 0.88.
When $A_i$ is secret and keeps drifting under local training, the server faces a blind problem. It must recover $h_j$ from projections through an unknown and changing matrix. It also cannot configure that matrix.
Formal privacy is unchanged. Noise and clipping are applied to $B$ exactly as in DP training of $B$ alone. $A_i$ is hidden state that is never released, so the standard DP-SGD accounting for the released $B$ updates still holds. The claim to test is therefore not a better $\epsilon$. It is that, at the same $\epsilon$ and the same upload size, secret local down-projections achieve much lower empirical leakage against active and adaptive attackers, with utility at least as good as freeze-$A$ under heterogeneous data.
This proposal builds most directly on FFA-LoRA from Improving LoRA in Privacy-preserving Federated Learning. FFA-LoRA freezes a shared random $A_0$, trains only $B$, avoids the quadratic noise cross-terms and averaging mismatch that make naive DP-LoRA collapse, and cuts communication by half. Its frozen $A_0$ is common to all clients and, as the file notes, never tested against inversion. We keep its linearity and its payload but make $A$ per-client, secret and learnable.
The design borrows its structure from the "local encoder, shared decoder" decoupling used for cross-device covariate shift in A Client-server Deep Federated Learning for Cross-domain Surgical Image Segmentation. In that work, local encoders translate differing inputs into a common space read by a shared decoder. Here, the down-projection plays the encoder role at every adapted layer. The design differs from the dual-adapter approach in FedDPA, which keeps a whole local LoRA beside a whole global one. That design leaves the global adapter's own $A$ fully exposed and has no DP. We split a single adapter across the privacy boundary.
Research Program
The method has three parts. Each client initializes $A_i$ from a private seed. During local training it updates both $A_i$ and $B$, but it applies per-example clipping and Gaussian noise only to the $B$ update before upload, as in the client-level DP of Fed-CDP from Gradient-Leakage Resilient Federated Learning. The server averages the $B$ updates.
The central utility risk is incoherence. Different clients' $B$-gradients are expressed in coordinates defined by different $A_i$, so early averaging mixes unaligned coordinates. Our bet is that local training of $A_i$ resolves this. Because $B$ is shared, each client's best $A_i$ is the one that maps its inputs onto the latent directions $B$ already decodes. Coherence then lives in output meaning rather than in a shared input map. We will measure this directly through per-layer alignment between clients' effective updates $BA_i$ over rounds. We will also test two cheap aids that do not reveal $A_i$: a short warm-up with a public frozen $A$ before switching to secret ones, and initializing $A_i$ from the public warm-up plus a secret random perturbation.
The second part is an attack suite designed against this specific defense, not just reused from prior work. Three attacks matter.
The first is PEFTLeak itself, with the server free to poison the backbone. We expect it to lose its main lever.
The second is a surrogate-encoder attack. The server trains its own down-projection on public data against the current $B$ and uses it as a stand-in for $A_i$. This attack is our main worry, because it exploits exactly the alignment that makes the method work.
The third is a probing attack. A poisoned backbone emits known vectors at some token positions, so each round reveals $A_i$ applied to known inputs, from which the server solves for $A_i$ over rounds. Countermeasures cost little and stay local: refreshing a secret random component of $A_i$ each round, and bounding how many rounds a probe can accumulate before $A_i$ has drifted.
We also run Gradient Bridge (GDBR) from Building Gradient Bridges. GDBR recovers more than 80% of batch labels from lower-layer gradients even with the head withheld, by tracing backpropagation with known weights. Hiding $A_i$ breaks part of that chain. Whether it breaks enough is an empirical question, because the error signal $\delta_j$ in the equation is still exposed.
Baselines
The naive reference point is federated LoRA with DP noise added to both $A$ and $B$. FFA-LoRA's paper reports it collapsing to 33.80% on MNLI at $\epsilon=1$, and DEeR's paper reports 42.15% on OCT-C8 at $\epsilon=0.1$. It shows what happens with no structural fix. Alongside it we report non-private federated LoRA under PEFTLeak as the upper bound on leakage.
The strongest utility competitors are FFA-LoRA and DEeR. FFA-LoRA (Improving LoRA in Privacy-preserving Federated Learning) reports 75.05% on MNLI at $\epsilon=1$; it is the closest design, with the same payload and linearity but a public $A$. DEeR (DEeR) alternates between updating $A$ and $B$ across rounds and maps noise interactions back into a linear regime. It reports 84.28% accuracy on OCT-C8 at $\epsilon=0.1$, the strongest private-LoRA number in the file. We run DEeR from its released code, CUHK-AIM-Group/DEeR, on its own medical splits. The file has no reported result on Lacuna for FFA-LoRA on OCT-C8, nor for either method on CIFAR-100 or TinyImageNet under DP, so we produce those numbers ourselves.
All methods share the same backbone, rank, client splits, $\epsilon$ and $\delta$ accounting, number of rounds and hyperparameter search budget. Each method gets the same number of tuning trials over learning rate, clipping bound and LoRA scaling.
Beating FFA-LoRA means matching its accuracy within one point at equal $\epsilon$ and bytes uploaded while cutting PEFTLeak patch recovery to near chance. Beating DEeR means matching its OCT-C8 accuracy at $\epsilon=0.1$ without its extra rounds.
The strongest leakage-side competitor is CENSOR (CENSOR). CENSOR replaces each update with a cold Bayesian sample drawn from the subspace orthogonal to the true gradient. It reports about 86% CIFAR-10 accuracy with negligible overhead but gives no formal guarantee. We apply it to the shared LoRA update under the same attacks.
Three ablations isolate the key ingredient. In the first, $A_i$ is trained locally but disclosed to the server; this keeps personalization and removes secrecy. In the second, $A_i$ is secret but frozen; this keeps secrecy and removes alignment, and so tests the incoherence risk. In the third, a single $A$ is kept secret from the server but shared by all clients.
Why not just…?
Why not just use DP-FFA-LoRA, since a formal guarantee already bounds inversion? DP bounds inversion only at budgets where utility usually suffers. The useful regime reported for FFA-LoRA is $\epsilon=1$, and nothing in the file shows that an active server with a poisoned backbone is stopped there. PEFTLeak's own paper reports the attack surviving low-level noise. Our method keeps the identical DP accounting, so it can only add protection. The experiment that settles this is PEFTLeak and GDBR run against FFA-LoRA and our method at matched $\epsilon$ from 0.1 to 8. If FFA-LoRA's leakage is already at chance at $\epsilon=1$, our contribution shrinks to the personalization gain, and we will say so.
Why not just keep one $A$ secret from the server but shared among clients? This is the most serious threat to the idea. It keeps FFA-LoRA's coherent averaging and still denies the server control of $A$. Its weakness is the threat model. In cross-device FL, a server can register its own clients and so learn the group secret. A single shared $A$ also gives a probing attack every client's rounds to accumulate equations against. We include it as an ablation under both an honest-membership and a Sybil server. If it matches our leakage without Sybils and beats our utility, we will recommend it for cross-silo deployments, where membership is controlled.
Why not just verify the server's weights on the client and reject identity-poisoned backbones? The file lists low-overhead verification against subtle architectural poisoning as an open problem. In any case, verification does not address passive inversion or GDBR-style label recovery on unpoisoned models. Our defense needs no detector. We still report a simple check, the distance of the frozen layers from the public checkpoint, to show how easily PEFTLeak-style poisoning is spotted and how a stealthier adaptive variant evades it.
Why not just apply CENSOR or gradient pruning to ordinary federated LoRA? CENSOR has no formal bound. Pruning is refuted in the file: GDBR survives 50–90% pruning, and MineGrad reports PEFTLeak surviving pruning and quantization (MineGrad). CENSOR is a real competitor on empirical leakage, so it appears in the main comparison rather than being dismissed.
Proposed Experiments
The vision experiments follow PEFTLeak's setting: ViT adapters on CIFAR-100 and TinyImageNet, batch sizes 32 and 128, and Dirichlet label splits plus a covariate-shift split. We measure patch recovery ratio, SSIM and LPIPS for reconstruction, and GDBR label recovery accuracy.
The medical experiments use DEeR's OCT-C8 and Kvasir-v2 silos at $\epsilon$ of 0.1 and 1, measuring accuracy, bytes uploaded per round and rounds to reach DEeR's accuracy. The text experiment uses MNLI at $\epsilon=1$ to compare against FFA-LoRA's 75.05%.
Four checks are decisive. The first is whether PEFTLeak patch recovery falls from its reported 85.9% regime to near chance with secret $A_i$, while disclosing $A_i$ restores it; this isolates secrecy as the cause. The second is whether the surrogate-encoder attack stays weak as $A_i$ align with $B$ over training, which is the failure mode we consider most likely. The third is whether secret-frozen $A_i$ underperform secret-trained $A_i$ under DP, which confirms that local training is what fixes incoherence. The fourth is whether accuracy at matched $\epsilon$ and payload is within one point of FFA-LoRA and DEeR, and above them under covariate shift.
If the surrogate attack reconstructs well, or utility trails FFA-LoRA by more than a point, the idea fails as stated. In that case the group-secret variant becomes the recommended design.
References
- Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning
- Improving LoRA in Privacy-preserving Federated Learning
- A Client-Server Deep Federated Learning for Cross-Domain Surgical Image Segmentation
- Dual-Personalizing Adapter for Federated Foundation Models
- Gradient-Leakage Resilient Federated Learning
- Building Gradient Bridges: Label Leakage from Restricted Gradient Sharing in Federated Learning
- DEeR: Deviation Eliminating and Noise Regulating for Privacy-Preserving Federated Low-Rank Adaptation
- CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Sampling
- MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning