Exact-in-Product Federated LoRA Under Secure Aggregation via Linear Sketches

Federated fine-tuning of large models usually trains LoRA adapters. LoRA stores each weight update as a product of two thin matrices, $\Delta W = BA$. It creates a quiet conflict between accuracy and privacy. The cryptographic tool that hides individual client updates, secure aggregation (a protocol that lets the server learn only the sum of client vectors, never any single one), can only add things. The update we actually want to average, $\frac{1}{N}\sum_i B_iA_i$, is a sum of products, and averaging the factors separately gives something else. Every method that fixes this mismatch while training both factors does so by inspecting individual factors. FedRot-LoRA aligns them with rotations, FLoRG decomposes Gram matrices, and FedPA-LoRA runs QR and SVD on the server. All three are therefore incompatible with secure aggregation. The methods that are compatible stay linear by freezing part of the adapter, and they pay for it in expressiveness under heterogeneous client data.

We propose that each client send two small random projections ("sketches") of its own product $B_iA_i$ instead of its factors. A sketch is linear in the product, so the server can securely sum the sketches and recover a near-optimal rank-$r$ approximation of the true average product from the sum alone. It does this without ever seeing a client's factors. The result is aggregation with no cross-client error from factor averaging and no dependence on how each client's factors happen to be rotated, it runs inside off-the-shelf secure aggregation, and any differential-privacy noise enters linearly in product space. The price is roughly twice the per-round upload of plain LoRA. The central empirical question is whether that price is repaid in bytes-to-accuracy under non-IID data against the cheap linear alternatives.

Core Claim

With a shared random test matrix $\Omega \in \mathbb{R}^{k\times s}$ and a co-range matrix $\Psi \in \mathbb{R}^{d\times s}$ (sketch width $s$ a small multiple of the rank $r$), client $i$ computes $Y_i = B_i(A_i\Omega) \in \mathbb{R}^{d\times s}$ and $Z_i = (\Psi^\top B_i)A_i \in \mathbb{R}^{s\times k}$. It never forms the $d\times k$ product. Because both maps are linear in $M_i = B_iA_i$, secure aggregation delivers exactly

$$Y = \tfrac{1}{N}\textstyle\sum_i Y_i = \bar M\Omega,\qquad Z = \tfrac{1}{N}\textstyle\sum_i Z_i = \Psi^\top \bar M,\qquad \bar M = \tfrac{1}{N}\textstyle\sum_i B_iA_i .$$

The server then reconstructs $\hat M = Q(\Psi^\top Q)^{+}Z$, where $Q$ is an orthonormal basis for the columns of $Y$ and $(\cdot)^+$ is the pseudo-inverse. This is the standard single-pass randomized low-rank approximation. It truncates $\hat M$ to rank $r$ and broadcasts balanced factors $B = U\Sigma^{1/2}$, $A = \Sigma^{1/2}V^\top$.

The first key property is that no cross terms $B_iA_j$ ever appear. The second is that the sketch is unchanged by any per-client rotation $B_iR$, $R^{-1}A_i$, so the rotational misalignment that FedRot-LoRA corrects never arises. The third is that the only approximation error is the unavoidable rank-$r$ truncation of $\bar M$ plus a sketching term. Randomized-SVD theory says this sketching term is a modest constant times the tail singular values of $\bar M$ once $s$ exceeds $r$ by a small oversampling margin.

The new element is the combination: exact product-space aggregation that both trains both factors and is compatible with blind additive aggregation. No work in the research file does both.

Research Program

The threat model is deliberately the semi-honest one. The server and up to a threshold of colluding clients follow the protocol but try to learn from what they see.

Under this model, secure aggregation already hides individual updates. The contribution is making exact non-linear aggregation possible inside it, not defeating an active server. MineGrad's PEFTLeak shows that a malicious server that rewrites frozen weights can extract image patches from adapter gradients. LIA-SA shows that a single tampered normalization layer disaggregates labels from secure sums. Both attacks are out of scope, and we say so rather than overclaim. One small hardening falls out for free: $\Omega$ and $\Psi$ are derived from a public seed agreed among clients, so the server cannot choose sketch directions adversarially.

The collusion model also differs from SHE-LoRA. SHE-LoRA encrypts a small set of sensitive columns under one key shared by all clients, so a single colluding client can hand the server that key. Pairwise-masked secure aggregation has no such single point of failure.

We will build four pieces. The first is the client-side sketch computation, which costs $O((d+k)rs)$ per adapted layer. The second is quantization of $Y_i$ and $Z_i$ into the finite field that secure aggregation requires, with the resulting error measured. The third is server reconstruction plus rank-$r$ truncation. The fourth is a server-side error-feedback buffer. The part of $\hat M$ discarded by truncation is a function of the aggregate only, so it reveals nothing further and can be carried into the next round.

We also need a policy for local continuity. FedPA-LoRA argues that overwriting clients' factors with a server re-factorization harms optimization. We will test two options: plain restart from the broadcast factors, and FedPA-style continuity, where clients keep local factors and add a product-space pull toward the global $BA$.

For differential privacy, each client clips $\|B_iA_i\|_F$ to a bound $C$, which is a well-defined, gauge-invariant quantity. The sensitivity of its sketch contribution is then at most $C\|\Omega\|_2$, and likewise with $\Psi$. Calibrated noise is added to the securely summed sketch, either through distributed noise shares or by a trusted aggregator at the end. The noise therefore enters $\hat M$ linearly through a fixed projection. This avoids the $N_BN_A$ term that DEeR, FFA-LoRA and PRISM identify as the cause of DP-LoRA's collapse, and it does so without freezing factors. PRISM obtains gauge invariance by projecting onto the tangent space in a centralized setting. We obtain it through product-space clipping in a federated setting.

Baselines

The naive reference is FedIT-style factor averaging under secure aggregation. Clients securely sum $B_i$ and $A_i$ separately and the server multiplies the averages. This is what one gets by bolting secure aggregation onto standard federated LoRA, and it carries the full cross-term error. FedRot-LoRA reports FedIT at 76.80% on MNLI with 50 clients in plaintext.

The strongest secure-aggregation-compatible competitor is alternating minimization under secure aggregation, as in DEeR. All clients share one frozen factor while training the other, and the roles swap each phase, so each round's aggregation is exactly linear. This is the most dangerous baseline because it is exact, linear, and cheaper per round than our method. Its known weakness is that only one factor moves per round, which should double rounds to convergence. We found no reported result on Lacuna for alternating minimization combined with secure aggregation, so we will run it ourselves from the DEeR codebase.

Two frozen-subspace methods also run natively under secure aggregation. FFA-LoRA freezes a random $A$ and reports 75.05% on MNLI at $\epsilon=1$. Fed-SB trains only an $r\times r$ core between public-data SVD bases and reports 70.33% on SNLI at $\epsilon=1$ versus 49.57% for FedIT. We will run Fed-SB in two versions: with its public warm-up slice, and with that slice drawn from a distribution mismatched to the clients.

Plaintext methods serve as an upper reference: they show the accuracy one gives up, if any, by insisting on secure aggregation. FedRot-LoRA reports 87.34% on MNLI with 50 clients. FedPA-LoRA reports 92.26% average GLUE accuracy with heterogeneous ranks. FLoRG reports a 2041× reduction in communication to reach 85% on QNLI versus FedEx-LoRA.

Two ablations isolate the key ingredient. The first replaces the sketch with exact secure summation of the dense product $B_iA_i$. This gives the same aggregate as our method without sketching error, at $d\times k$ cost. The second sweeps sketch width $s \in \{r, 1.5r, 2r, 3r\}$ and toggles the error-feedback buffer.

All methods share the backbone, client split, number of local steps, rank $r$ (for us, the rank after truncation), and a hyperparameter search of equal size. All run inside the same secure-aggregation implementation, so its byte expansion is counted identically.

Beating the baselines means two things. First, higher final accuracy than alternating minimization and Fed-SB at matched cumulative uplink bytes under non-IID splits. Second, accuracy within about one point of plaintext FedRot-LoRA or FedPA-LoRA at the same round budget.

Why not just…?

Why not just run Fed-SB or FFA-LoRA under secure aggregation, since they are already linear and far cheaper? This is the most serious threat, and the proposal lives or dies by it. The evidence against these methods concerns heterogeneity. FedFTG reports FFA-LoRA collapsing to ROUGE-L 0.3047 on MedQuAD with TinyLlama under non-IID splits, versus 0.6994 for FedFTG, and gives a subspace-bottleneck excess-risk argument. Fed-SB depends on a public slice representative of the clients. If Fed-SB matches our method on non-IID MedQuAD and GLUE with no representative public data, the extra bytes are unjustified, and we will report that as the outcome.

Why not just alternate which factor trains, as DEeR does, which is exact and linear? Alternation is exact, but each round explores only the span of the frozen factor. The file records that it roughly doubles rounds to convergence. Our method moves both factors every round and re-bases the global subspace from the aggregate. The accuracy-versus-cumulative-bytes curve, with secure-aggregation overhead included, decides whether halving rounds outweighs doubling the per-round payload. This is not obvious in advance.

Why not just securely sum the full product $B_iA_i$? It is exact and simple, but its upload is $d\times k$ per layer. That discards the whole point of LoRA, and it is the ablation that measures what sketching costs in accuracy.

Why not just use full homomorphic encryption or SHE-LoRA's selective encryption? Neither removes the need for non-linear server work on individual updates. SHE-LoRA shows full encryption costs about 344× more bytes than its selective scheme. Its own single shared key breaks under one colluding client, which pairwise-masked secure aggregation tolerates.

Proposed Experiments

The main experiment fine-tunes RoBERTa-large on MNLI, QNLI and SST-2 with 50 clients under Dirichlet non-IID label splits at concentrations 0.1 and 1.0. This matches the FedRot-LoRA and FLoRG settings. The second setting is TinyLlama on MedQuAD, reproducing the FedFTG split where frozen subspaces failed. Private runs use SNLI and MNLI at $\epsilon=1$ to compare against the Fed-SB and FFA-LoRA numbers. If compute allows, a Llama 3-8B GSM8K run tests scale.

The metrics are accuracy or ROUGE-L, and cumulative uplink bytes to reach a target accuracy counted after secure-aggregation expansion. We also measure $\|\hat M_r - \bar M_r\|_F / \|\bar M\|_F$, the gap between our reconstruction and the exact rank-$r$ truncation computed offline from plaintext, together with server reconstruction time.

A leakage sanity check runs DAGER, the attack used by SHE-LoRA, against both the individual plaintext factors sent under FedRot-LoRA and the aggregate sketch our server sees. This confirms that only aggregate-level information is exposed.

Three results would be decisive. The reconstruction gap should shrink toward the pure truncation error as $s$ grows, matching the randomized-SVD form. Under concentration 0.1 with a mismatched public slice, our method should beat Fed-SB, FFA-LoRA and alternating minimization in bytes-to-target accuracy. And the dense-product ablation should show that sketching costs under half a point of accuracy.

Two findings would falsify the proposal: Fed-SB or alternation matching us on bytes-to-accuracy under strong heterogeneity, or large gaps between our reconstruction and exact truncation at $s=2r$.