In federated learning, many devices train a shared model without sending their raw data to a server. The catch is that the model updates they send can often be inverted back into the training examples. Defenses that stop this inversion without destroying accuracy tend to be expensive on the device. One runs a StyleGAN3 per client, another solves a nested optimization per sample, others add encryption. The cheap defenses, mostly added noise and pruning, cost accuracy. A recent design for fine-tuning large models suggests a different lever. In Fed-SB, clients train only a tiny square "bridge" matrix $R$ sandwiched between two frozen projection matrices ($\Delta W = BRA$). This makes noisy private training far more accurate. Whatever a client uploads then passes through the frozen projection $A$, so the choice of $A$ decides what an attacker can possibly see.
We propose to choose that frozen projection deliberately, once, before training. It should keep directions of the layer input that help the task and drop directions that mainly help an attacker redraw the input. Fed-SB builds $A$ from the top singular directions of proxy activations. Those are the directions that carry the most input energy, which are also the most reconstructable ones. We replace this with a one-time generalized eigenvalue problem. We pair it with a cheap client-side audit that blocks a malicious server from planting a leaky projection. Per-round client cost stays identical to Fed-SB, and the method needs no generator, no inner optimization, and no ciphertexts.
Core Claim
The claim has three parts. First, bridge-only adapters have a structurally narrow leakage channel. Second, the standard way of building them, an SVD of proxy activations, points that channel at exactly the wrong directions. Third, re-choosing the subspace gives a better privacy-utility curve than adding noise or encryption, at no extra per-round cost.
The narrowness follows from the gradient itself. For one adapted layer with input $x_i$ and backpropagated error $\delta_i$, the gradient with respect to $R$ is
$$\nabla_R L = \sum_{i=1}^{b} (B^\top \delta_i)(A x_i)^\top,$$
where $b$ is the local batch size. The input enters only through the $r$-dimensional projection $A x_i$, where $r$ is the bridge rank, and the error enters only through $B^\top \delta_i$. We take $\Sigma_x$ to be the proxy covariance of layer inputs, and $\Sigma_s$ to be a task-signal covariance, here the gradient-weighted between-class scatter of the inputs on the proxy set. We then pick the rows of $A$ to maximize $\mathrm{tr}(A\Sigma_s A^\top)$ while holding the captured input energy $\mathrm{tr}(A\Sigma_x A^\top)$ fixed. This is the familiar linear-discriminant-style problem. Its solution is the top $r$ generalized eigenvectors of $(\Sigma_s, \Sigma_x + \lambda I)$, with $\lambda$ a regularizer that trades utility for privacy.
The best linear reconstruction of $x$ from $Ax$ leaves an error of
$$\mathrm{tr}(\Sigma_x) - \mathrm{tr}\big(\Sigma_x A^\top (A\Sigma_x A^\top)^{-1} A \Sigma_x\big).$$
This error is smallest for the SVD choice and grows as $A$ moves away from the high-variance directions. This bound covers only linear decoders. Attacks with generative image priors or language-model priors may fill in what the projection drops. Testing that empirically is central to the proposal, not a footnote.
Research Program
The work has three pieces.
The first is a measurement study that does not yet exist. Fed-SB (Fed-SB) shows strong utility under differential privacy but was never attacked with gradient inversion. We will attack it with three kinds of attack:
- Active attack: PEFTLeak (MineGrad), in which a server poisons the frozen backbone and adapter down-projections to funnel image patches.
- Passive attacks: gradient matching with a generative prior (Gradient Inversion with Generative Image Prior), and the multi-step attack AGIC (AGIC), which treats several local steps as one giant batch.
- Text attack: LAMP (LAMP) on the language side.
This tells us whether the bridge structure alone already blunts inversion.
The second piece is the leakage-aware subspace described above, computed once per adapted layer from a small public proxy set. Fed-SB itself uses about 0.1% of the data as proxy. We will compare it against the SVD subspace and a random Gaussian subspace of the same rank. The point is to separate "low rank helps" from "the right low-rank directions help."
The third piece is a one-time client audit against active poisoning. PEFTLeak works because the server can set the down-projection and backbone to whatever it likes. Bridge-only training freezes both after initialization, and our subspace is a deterministic function of public proxy data, public code and a public seed. So each client can recompute $A$ and $B$ once, compare hashes of the backbone and projections, and refuse to train on a mismatch. After that, the server's only per-round freedom is the $r \times r$ matrix $R$. We will test whether an adversarial choice of $R$ can by itself recover meaningful leakage, since $R$ still shapes downstream activations. This audit costs one proxy pass at enrollment and nothing per round. That stands in contrast to Shadow Defense (Shadow defense), which runs a StyleGAN3 and Grad-CAM++ on-device, and DCS² (Concealing Sensitive Samples), which optimizes a decoy per sample every iteration.
Prior work plays three roles here:
- Motivating: Fed-SB motivates the design through its structure and the noise-amplification result. PEFTLeak and AGIC motivate it through attacks that defeat large batches, pruning and multiple local epochs.
- Constraining: Huang et al. (Evaluating Gradient Inversion Attacks and Defenses) constrain the evaluation. Their finding that attacks weaken sharply without exact BatchNorm statistics and labels means we must report both strong-adversary and realistic-adversary settings, or the gains could be artifacts of a weak attacker.
- Closest in spirit: CENSOR (CENSOR) also exploits subspace geometry. It replaces each gradient with a sample from the orthogonal complement every round, which has no certified guarantee and was tested only on vision backbones. We instead fix a task-discriminative subspace once and leave the gradient inside it untouched, which keeps aggregation linear and composes cleanly with differentially private training.
Baselines
Undefended standard federated LoRA (FedIT). This is the do-nothing reference point. It shows how much a typical adapter deployment leaks and what accuracy it reaches. We run it with the same backbone, rank budget, clients and rounds as every other method.
Fed-SB with its SVD subspace, with and without differentially private SGD. This is the strongest competitor on the utility side, because it already solves noise amplification. The file reports 70.33% on SNLI at $\epsilon=1$, versus 49.57% for FedIT (Fed-SB). We use identical proxy data, rank, clipping norm and tuning budget for the noise level. Our method beats it if, at matched attack success, it reaches higher accuracy, or if at matched accuracy it shows lower patch reconstruction rate and higher reconstruction LPIPS.
CENSOR ported to bridge or adapter gradients. This is the strongest low-overhead inversion defense in the file. It reports about 86% CIFAR-10 accuracy and 0.00236% added runtime, and no reported result was found on Lacuna for CENSOR on LoRA or other adapters. We give it the same temperature-tuning budget as our $\lambda$.
SHE-LoRA. This is the strongest cryptographic competitor. It defeats the exact text inversion attack DAGER while encrypting 0.03% of parameters (SHE-LoRA). It is run on the text tasks with its released selection procedure, and we report its client encryption time and the size of the extra traffic.
Random-subspace and SVD-subspace ablations at equal rank. These ablations isolate the key ingredient, which is the choice of directions rather than low rank or freezing. We also run the audit-off ablation under PEFTLeak, which isolates the audit's contribution against active attacks.
Why not just…?
Why not just run Fed-SB with differential privacy and stop? This is the most serious threat. If Fed-SB with modest noise already pushes inversion to failure, our subspace adds little. The experiment that settles it is a full attack-success-versus-accuracy curve for both subspaces across noise levels, including zero noise. Our idea survives only if the leakage-aware subspace shifts that curve: either it needs less noise for the same attack failure, or it resists attacks that noise at the useful $\epsilon$ does not stop. If the curves coincide, we will report that as the finding.
Why not just use large batches and many local epochs? PEFTLeak still recovers 72.6% of CIFAR-100 patches at batch size 128 (MineGrad). AGIC turns multiple local steps into an asset for the attacker, reporting up to 50% higher PSNR (AGIC). Large batches also strain device memory. We include batch-size sweeps so readers can see where our gain persists.
Why not just apply CENSOR's orthogonal sampling to adapter gradients? It is nearly free, so it is a real competitor. However, it resamples every round with no fixed structure, which may interact badly with DP clipping and with AGIC's cross-epoch matching. It also does nothing about a server that poisons the frozen layers. The head-to-head under PEFTLeak and AGIC will decide this.
Why not just encrypt the sensitive columns, as in SHE-LoRA? Encryption gives strong protection against passive attackers. It needs a shared key that collusion breaks, and it requires client-side SVD and encryption work. We compare protection at matched client wall-clock time. We expect SHE-LoRA may remain stronger against exact text inversion, and the question is whether it buys that at a cost edge devices can sustain.
Proposed Experiments
Vision. We use a Vision Transformer with adapters on CIFAR-10, CIFAR-100 and TinyImageNet, the PEFTLeak settings. The reference numbers are 82.8%, 85.9% and 81% patch reconstruction at batch size 32. Clients receive non-IID splits.
Text. We fine-tune on SNLI and COMMONSENSE170K, following the Fed-SB setup, and attack with LAMP and a DAGER-style exact attack.
Metrics. Leakage is measured by patch reconstruction rate, PSNR, SSIM and LPIPS for images, and token recovery for text. Utility is test accuracy. Client cost is per-round wall-clock time, peak memory, and the one-time audit cost.
Decisive checks. There are four:
- Bridge-only training alone should drop PEFTLeak success once the audit pins the frozen layers. If an adversarial $R$ restores it, the audit idea fails.
- At equal rank, the leakage-aware subspace should raise reconstruction LPIPS against the generative-prior attack relative to SVD, with accuracy within a margin we fix in advance. If generative priors erase the linear-bound advantage, the core claim is wrong.
- Under DP-SGD at $\epsilon=1$ on SNLI, our method should match or exceed 70.33% while showing lower token recovery.
- Client per-round overhead should be indistinguishable from Fed-SB.
References
- Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
- MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning
- Gradient Inversion with Generative Image Prior
- AGIC: Approximate Gradient Inversion Attack on Federated Learning
- LAMP: Extracting Text from Gradients with Language Model Priors
- Shadow defense against gradient inversion attack in federated learning
- Concealing Sensitive Samples against Gradient Leakage in Federated Learning
- Evaluating Gradient Inversion Attacks and Defenses in Federated Learning
- CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Sampling
- SHE-LoRA: Selective Homomorphic Encryption for Federated Tuning with Heterogeneous LoRA