Client-Owned Down-Projections for Private, Heterogeneous Federated LoRA

Federated fine-tuning lets many devices adapt a shared foundation model without pooling their data. In the most common form, LoRA (low-rank adaptation), each client trains two small matrices: a "down-projection" $A$ that compresses each input feature into a few numbers, and an "up-projection" $B$ that maps those numbers back into the model. Each client then sends its updates to a server. Small updates were once assumed to be private, but they are not. A server that is allowed to rewrite the model can turn the frozen backbone into an identity map and set the down-projection to act as a filter that isolates individual image patches. With this attack it recovers most of each client's images even through a rank-8 adapter. The standard fix, differential privacy (clipping each example's influence and adding calibrated noise), wrecks accuracy at strict budgets, and it does so especially badly when clients' data distributions differ sharply.

We propose a single design change that addresses three problems at once. The client, not the server, owns the down-projection. Each client derives $A$ from a public seed that it can verify, adapts $A$ locally, and never sends it. The client checks the frozen backbone against a published hash, and it uploads only a noised, aligned $B$. This removes the server's main lever for active attacks. It lets $A$ adapt to each client's own feature distribution, which matters under strong heterogeneity. It also keeps the upload to one matrix with noise that is linear in what is transmitted. The proposal is to build this protocol, give it a correct privacy accounting, and test it against attacks designed to break it.

Core Claim

The active attack in MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning, called PEFTLeak, needs two capabilities. The server must be able to rewrite frozen backbone layers into identities, and it must be able to author the down-projection weights as band-pass filters. That paper reports 85.9% of CIFAR-100 patches recovered at batch size 32. Our claim is that a client which pins the backbone and owns $A$ removes both capabilities. What remains for the attacker is the uploaded up-projection gradient. Per example, that gradient has the form $\delta\,(A_i x)^\top$, where $x$ is the adapter's input feature, $A_i$ is the client's private down-projection, and $\delta$ is the backpropagated error signal at the adapter output. The attacker therefore sees only rank-$r$ projections of the input through a map it did not choose, and it sees them under differential-privacy noise.

We further claim that this same ownership resolves a known contradiction in the literature. Freezing $A$ makes private training work, but it hurts accuracy under heterogeneity. Adapting $A$ helps heterogeneity, but it reintroduces noise problems.

What prior work establishes and what it leaves open

Two papers motivate the design by exposing the tension.

Improving LoRA in Privacy-preserving Federated Learning (FFA-LoRA) freezes a random $A_0$ and trains only $B$. This removes the "noise-times-noise" cross term that appears when both factors are perturbed, and it lifts MNLI accuracy at privacy budget $\epsilon=1$ from 33.80% to 75.05%. Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients (LoRA-$A^2$) shows that a permanently frozen $A$ costs up to 23% accuracy under severe label skew (Dirichlet concentration 0.01). It restores adaptation by alternating which factor is trained across rounds.

Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models (LA-LoRA) supplies a method we reuse. It alternates the two factors within each local step, which removes the cross term while letting both factors learn, and it reaches 60.68% on Tiny-ImageNet at $\epsilon=1$. However, LA-LoRA still transmits both noised factors, and it is untested against active attacks.

The closest existing design is Decoupled Low-Rank Adaptation for Robust Federated Fine-Tuning (FedDR-LoRA). It already keeps $A$ local and aggregates only $B$, and it argues that $A$ is the more stable factor under client divergence (88.30% vs. 60.11% on noisy CIFAR-100). FedDR-LoRA sends $B$ in plaintext, has no differential privacy, never evaluates inversion, and does not ask whether local copies of $A$ drift into incompatible coordinate frames.

SDFLoRA splits each adapter into a shared module and a private module, and it adds noise only to the shared module. It is a natural comparison, but its private module, which is trained on raw data without noise, raises the accounting issue we address below. It is also untested against attacks.

FedRot-LoRA supplies our alignment tool. Its Orthogonal Procrustes step rotates each client's factors to a common orientation before averaging, raising MNLI from 76.80% to 87.34%. It does so without any privacy noise, and the file flags the interaction between Procrustes alignment and noise as unstudied.

Research Program

Protocol. All clients derive $A_0$ from a public seed and reject any server-sent backbone whose hash differs from the published one. In each round, a client runs local alternation in the style of LA-LoRA. During $B$-steps, $A_i$ is frozen and per-example $B$ gradients are clipped and noised. During $A$-steps, $B$ is frozen and per-example $A$ gradients are clipped and noised with their own noise level.

Because each local $A_i$ drifts, the client's $B_i$ lives in a rotated latent frame. Before upload, the client computes the rotation $R_i$ that best aligns $A_i$ to $A_0$ and sends $B_i R_i^\top$. That matrix is the same update expressed in the shared frame. Since $A_i$ was itself produced under differential privacy, computing $R_i$ is post-processing and adds no privacy cost. The server averages the aligned $B$ matrices and broadcasts the result. Only one $d \times r$ matrix goes up per adapter, matching FFA-LoRA's 50% uplink saving.

Privacy accounting. This is a substantive point, not a formality. A down-projection trained without noise but used to compute the released $B$ gradients would break the guarantee. Changing one example changes $A_i$, and $A_i$ in turn changes every other example's $B$ gradient. The same concern applies to SDFLoRA's unperturbed private module. We therefore account for the $A$-steps and $B$-steps jointly as one composed mechanism per client.

We also study the trade-off of moving budget between the two step types. The question is whether spending most of the budget on $B$, and running few, heavily noised $A$-steps, keeps most of the heterogeneity benefit.

Attack analysis. We pose a precise question: given a verified backbone, the public $A_0$, and control of $B$ only, what can a malicious server recover? We construct adaptive attacks for this setting, and they are the key stress test. One attack crafts $B$ so that $\delta$ concentrates on single examples. Another runs multi-round strategies in the style of PEFTLeak that change $B$ across rounds. A third is a passive inversion that jointly estimates $A_i$ and $x$ under the image prior used in Gradient Inversion with Generative Image Prior. For text we use the DAGER attack referenced by SHE-LoRA.

Baselines

The naive reference is standard federated LoRA, which sends both factors with no privacy mechanism. We run it under PEFTLeak to reproduce the attack's reported CIFAR-100 recovery of 85.9% at batch size 32. It anchors both the accuracy ceiling and the leakage ceiling.

The strongest private-utility competitor for language is FFA-LoRA, which reports 75.05% MNLI accuracy at $\epsilon=1$. For vision it is LA-LoRA, which reports 60.68% on Tiny-ImageNet at $\epsilon=1$ with Swin-B.

Two baselines must be implemented with differential privacy added. The first is FedDR-LoRA, the closest architectural relative; no reported result was found on Lacuna for it under differential privacy or under any inversion attack. The second is SDFLoRA; no reported numeric result was found on Lacuna for it at $\epsilon=1$, only a qualitative claim. LoRA-$A^2$ is the heterogeneity-robust private baseline at Dirichlet 0.01.

All methods share the same backbone, client partitions, rank, number of rounds, privacy accountant, and total $\epsilon$. Each gets the same hyperparameter search budget over learning rate, clipping threshold, and LoRA scaling. Every method runs under every attack. Beating a baseline means two things: matching or exceeding its accuracy at equal $\epsilon$ and equal uplink bytes, and lowering reconstruction quality under the adaptive attack.

Three ablations isolate the key ingredient. The first is "pinned FFA-LoRA", which uses a verified backbone and a frozen public $A_0$. It tests whether client ownership of an adapted $A$ matters beyond verification. The second removes Procrustes alignment, testing whether frame drift harms aggregation. The third accepts a server-supplied, unverified $A$. Its purpose is to confirm that the PEFTLeak protection comes from ownership of $A$ and not from noise.

Why not just…?

Why not just use differential privacy, since it bounds every adversary, active ones included? This is a real threat to the protection claim. A correctly applied client-side mechanism bounds what any server learns, whatever weights the server sends. Our answer is empirical, and the experiment can come out against us. We plot reconstruction quality under PEFTLeak against $\epsilon$ for plain private LoRA and for our protocol. If privacy noise at a budget that preserves accuracy already drives PEFTLeak to chance, then the privacy contribution of ownership shrinks to the weak-privacy regime. In that case the paper's claim becomes the utility gain alone, and we will say so.

Why not just pin the backbone and freeze a public $A_0$ (FFA-LoRA with verification)? This alternative may block filter installation as well as our protocol does. It is therefore our most important ablation, not a strawman. Two things separate them. First, LoRA-$A^2$ reports up to a 23% accuracy loss from a frozen $A$ under Dirichlet 0.01. Second, with a public $A_0$ the server knows $A_0 x$ exactly from each $B$ gradient, which may make passive inversion easier. If pinned FFA-LoRA matches our protocol on both accuracy and leakage at Dirichlet 0.01, the idea fails and we will report that.

Why not just use secure aggregation or selective homomorphic encryption? SHE-LoRA defeats DAGER while encrypting 0.03% of parameters. However, the file states that it assumes an honest-but-curious server, that all clients share one key, and that the remaining 99.97% of parameters stay in plaintext without formal guarantees. None of this addresses a server that rewrites the backbone. We include SHE-LoRA as a passive-attack comparison only.

Why not just add privacy noise to FedDR-LoRA? If that works, it is a strong result for FedDR-LoRA. Our protocol differs from it in three specific places: verified $A_0$ and backbone, privacy-correct accounting for the training of $A$, and alignment before upload. The ablations measure each of these.

Proposed Experiments

The vision experiments follow the PEFTLeak setting: a ViT backbone on CIFAR-10, CIFAR-100, and Tiny-ImageNet, at batch sizes 32 and 128, with Dirichlet client splits at 0.5, 0.1, and 0.01. The language experiments use RoBERTa-Large on MNLI, QNLI, and QQP, and BANKING77 and 20 Newsgroups at Dirichlet 0.01, where LoRA-$A^2$ reports its gains. We test $\epsilon \in \{1, 3, 8, \infty\}$.

The metrics are accuracy, uplink bytes per round, fraction of patches recovered, reconstruction PSNR and SSIM, and DAGER token recovery for text. We also measure frame drift, defined as the Procrustes residual between $A_i$ and $A_0$ over rounds.

Three checks are decisive. The first is whether PEFTLeak recovery falls from roughly 85.9% to near chance once ownership of $A$ is enforced at $\epsilon=\infty$, that is, with no noise. This isolates ownership from privacy noise. The second is whether the best adaptive attack against $B$ alone still recovers recognizable inputs. The third is whether, at $\epsilon=1$ and Dirichlet 0.01, the protocol beats both FFA-LoRA and pinned FFA-LoRA on accuracy at equal uplink cost. A positive answer on all three supports the claim. A failure on the second or third narrows the claim to the specific axis that survives.