When phones, hospitals or vehicles fine-tune a shared foundation model together, each device trains only a small add-on module, called an adapter, and sends that module's update to a central server. This is federated learning. It is cheap enough for edge hardware, but it is not private by default. Recent attacks show that a malicious server can rebuild training images from adapter updates. It does this by tampering with the weights it sends out so that raw inputs flow undistorted into the trainable adapter. Adding differential privacy, meaning calibrated noise with a formal guarantee, to the standard low-rank adapter (LoRA) destroys accuracy. The best-known fix freezes one of LoRA's two matrices. It does restore accuracy under noise, but it hands the server exactly the fixed projection an attacker needs, and it plateaus on hard tasks.
We propose to let the clients, not the server, generate that frozen projection, draw a new one every round from a secret seed that the clients agree on among themselves, and apply the merged update to their own copies of the model. The server only averages small, noised coefficient matrices whose meaning it cannot interpret. It never holds a model it could tamper with. We expect three things from this change, and each will be tested. The privacy noise stays well behaved, as in the frozen-projection method. The accumulated update escapes the rank ceiling that makes the frozen method stall. And the server loses the levers that active reconstruction attacks rely on. Edge cost stays at half of standard LoRA's upload and requires no local matrix factorization or encryption.
Core Claim
The method builds on FFA-LoRA (Improving LoRA in Privacy-preserving Federated Learning). That paper writes the adapter update as $\Delta W = B A_0$, where $A_0$ is a random down-projection frozen for all of training and $B$ is the only trained and uploaded matrix. With only $B$ trained, the update is linear in the trained parameters. Averaging is then exact and privacy noise does not multiply with itself. On MNLI at $\epsilon=1$ this gives 75.05% accuracy, where vanilla LoRA under the same noise collapses to 33.80%.
We make two changes. First, $A_t$ is regenerated each round $t$ from a seed $s_t$ that the clients derive jointly and the server never learns. Second, the server returns only the averaged coefficients $\bar B_t$, and each client merges them locally. The model that results after $T$ rounds is
$$W_T = W_0 + \sum_{t=1}^{T} \bar B_t\, A_t(s_t), \qquad \bar B_t = \frac{1}{K}\sum_{k=1}^{K}\big(B_{t,k} + \xi_{t,k}\big),$$
Here $W_0$ is the public pretrained weight, whose hash every client checks. $K$ is the number of participating clients, $B_{t,k}$ is client $k$'s clipped local coefficient matrix, and $\xi_{t,k}$ is its Gaussian privacy noise.
Within a round the method is FFA-LoRA exactly. Privacy accounting is therefore unchanged, because $A_t$ does not depend on the data and costs no privacy budget. Across rounds the sum can reach rank $rT$ rather than $r$, which is our proposed answer to the plateau that Why Gradient Subspace? proves for FFA-LoRA. That paper reports a ROUGE-L of 0.3047 on MedQuAD for FFA-LoRA versus 0.6994 for its FedFTG method.
The security claim is narrower than a full guarantee, and we state it that way. Differential privacy remains the only formal protection. Keeping the subspace secret is an extra empirical layer that aims to remove the active attack surface. It holds only as long as no client colludes with the server. If one does, the scheme falls back to FFA-LoRA with privacy noise and becomes no weaker than that baseline.
Why the Secret Subspace Should Matter
The PEFTLeak attack (MineGrad; Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning) needs two things from the server. It poisons the frozen backbone into identity maps, and it plants a statistical sieve in the adapter's down-projection weights. Our design takes both away:
- No backbone to poison. Clients hold $W_0$, check its hash, and apply their own merges, so the server sends no backbone.
- No projection to plant. The down-projection is generated on the client from a seed the server does not know.
What the server can still do is send a crafted $\bar B_t$. Without knowing $A_t$, it cannot aim that matrix at any particular input direction, and clients clip the norm of whatever they receive.
Passive attacks are also weakened. For one sample, the gradient with respect to $B$ is $\delta (A_t h)^\top$, where $h$ is the input to the adapter and $\delta$ is the error signal flowing back into its output. When $A_t$ is public, the server obtains a rank-$r$ projection of $h$ each round. With a fresh public $A_t$ each round it would obtain many different projections of the same data. That is why a public resampling schedule could make leakage worse than FFA-LoRA, and why secrecy is the ingredient that has to be isolated.
The attack that remains untouched is ReCIT's supply-chain threat (ReCIT), in which the base model itself was primed by a malicious provider. That is out of scope. We will nevertheless test ReCIT's gradient-only token-filtering stage, which works from the column space of adapter gradients.
Research Program
The build has three parts.
Seed agreement. We will write a client-side protocol that derives each round's seed $s_t$ from pairwise key agreement among authenticated participants, much like the masking keys in secure aggregation. It will also let clients that join late fetch the history of seeds. This limits the scheme to settings where clients are authenticated, such as cross-silo hospitals or managed vehicle fleets. Open cross-device settings are excluded, because there the server could register fake clients and learn the seed.
Client training loop. The loop generates $A_t$ from the seed and trains $B$ with per-sample clipping and Gaussian noise, as FFA-LoRA does. It uploads $B$, receives $\bar B_t$, clips its norm and merges it in place. Optimizer state is kept only for $B$, and the loop needs no singular value decomposition (SVD) and no encryption.
Analysis of the rank gain. We will characterize when the resampled sum actually escapes the rank-$r$ plateau under data that differs across clients, and when fresh noise in each new subspace simply accumulates.
We will also run an attack suite that is new to this setting. It adapts PEFTLeak and generative-prior inversion (Auditing Privacy Defenses via Generative Gradient Leakage) to FFA-LoRA and to our method, under two conditions: with the subspace known to the attacker (simulated collusion) and with it unknown.
Baselines
Do-nothing reference points. Non-private federated LoRA with the server holding the model gives the utility ceiling and the attack's best case. Vanilla LoRA with DP-SGD gives the collapse point, reported at 33.80% on MNLI at $\epsilon=1$ by Improving LoRA.
FFA-LoRA with privacy noise. This is the strongest private competitor for utility. Its reported 75.05% MNLI accuracy at $\epsilon=1$ is the utility number to match. We will rerun it with the same RoBERTa-scale backbone, client split, clipping search, noise accountant and number of rounds as our method. Beating it means equal or better accuracy at $\epsilon\in\{1,3,8\}$ on MNLI and a clear gain on MedQuAD, together with lower reconstruction success under active attack.
FedFTG. From Why Gradient Subspace?, FedFTG is the strongest method for utility on hard federated tasks, reporting ROUGE-L 0.6994 on MedQuAD with TinyLlama. It includes no privacy mechanism. We will run it with privacy noise added to its full up-projection updates, under the same privacy budget and tuning budget. The goal is to close most of the gap between it and FFA-LoRA at matched privacy.
SDFLoRA. SDFLoRA adds privacy noise only to a shared module and keeps a private module on the client. It is the natural private competitor under heterogeneous clients, run with its own published settings.
SHE-LoRA. SHE-LoRA encrypts a small subset of columns. It is the cryptographic competitor, and we will compare it on edge cost as well as attack resistance.
Isolating ablation. "Resampled public seed" keeps everything in our method except secrecy, and "fixed secret seed" keeps secrecy but drops resampling. These two variants separate the utility gain from the security gain.
Why not just…?
Why not just rely on DP-SGD with FFA-LoRA, since noise already bounds reconstruction? This is the most serious alternative. PEFTLeak was never evaluated against formal privacy noise, so we do not know whether noise at a usable budget already stops it. We will run the active attack against FFA-LoRA with noise at $\epsilon\in\{1,3,8\}$. If reconstruction is already negligible at the budget that gives acceptable accuracy, the secrecy component adds little. The contribution then reduces to the per-round resampling, and we would say so.
Why not just use secure aggregation or selective homomorphic encryption? These hide individual client updates. They do not stop a server from poisoning the weights it broadcasts, and PEFTLeak's attack is exactly that poisoning. SHE-LoRA also assumes an honest-but-curious server and needs an SVD on each client. Our method removes the broadcast channel instead, and it can be combined with secure aggregation. The experiments measure both attack success and on-device cost against SHE-LoRA.
Why not just add privacy noise to FedFTG? Its projection of gradients onto a low-rank subspace is computed from the private gradients themselves, so the projection needs its own privacy accounting. Its full-weight averaging also puts noise across the entire up-projection layer. The noised-FedFTG baseline tests this directly. If it matches our utility at the same budget, our utility claim fails.
Why not just resample a public random projection? The ablation tests this. We expect it to match our utility but to leak more under passive attack than FFA-LoRA, because the server gathers many projections of the same inputs. If it does not leak more, secrecy matters only against active attackers.
Proposed Experiments
Utility. We will run GLUE (MNLI, SST-2, QNLI, QQP, RTE) under non-IID client splits with 10 to 50 clients. We will also run MedQuAD and Dolly-15k with TinyLlama, and GSM8K with a 7B model. For GSM8K, FFA-LoRA's reported non-private 17.12% (Improving LoRA) is the reference.
Leakage. We will use a ViT with adapters on CIFAR-100, TinyImageNet and ImageNet under the adapted PEFTLeak attack. The comparison points are the reported 85.9% patch recovery and SSIM 0.88 at batch size 32, and 72.6% recovery at batch size 128. We will measure recovery rate, SSIM and LPIPS for both the known-subspace and unknown-subspace conditions. Generative-prior inversion on CelebA gives a semantic-attribute check, for which no reported result was found on Lacuna. Token recovery on Enron follows ReCIT's filter stage.
Edge cost. We will measure peak on-device memory, uplink bytes per round and client wall-clock time on an edge-class GPU, against SHE-LoRA and SDFLoRA.
Decisive checks. The main utility test compares two numbers at matched $\epsilon$: MedQuAD ROUGE-L for our method against FFA-LoRA with privacy noise. Our method needs to recover at least half of the gap between FFA-LoRA's 0.3047 and FedFTG's 0.6994. The main security test is PEFTLeak patch recovery on CIFAR-100 without collusion. It should drop to near chance for our method while staying high for the public-resampling ablation. Under collusion, recovery should return to the level of FFA-LoRA with noise and go no higher.
References
- Improving LoRA in Privacy-preserving Federated Learning
- Why Gradient Subspace? Identifying and Mitigating LoRA's Bottlenecks in Federated Fine-Tuning of Large Language Models
- MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning
- Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning
- ReCIT: Reconstructing Full Private Data from Gradient in Parameter-Efficient Fine-Tuning of Large Language Models
- Auditing Privacy Defenses in Federated Learning via Generative Gradient Leakage
- SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clients
- SHE-LoRA: Selective Homomorphic Encryption for Federated Tuning with Heterogeneous LoRA