Rotating Shared Subspaces for Private Federated Fine-Tuning

When many devices jointly fine-tune a large pretrained model without sharing their data (federated learning), two costs dominate. The first is bandwidth, since each round every device uploads its model change. The second is privacy noise. To give a formal differential-privacy guarantee, which bounds how much any one client's data can change what the server sees, each client's update is clipped to a maximum size and then covered with random noise. The most successful current recipe handles both costs with a single trick. It trains only a small low-rank adapter, and it freezes one of the adapter's two factor matrices at a random value, as in Improving LoRA in Privacy-preserving Federated Learning. This keeps server averaging exact and avoids noise being multiplied by noise. The price is that every update the model will ever receive is confined to one random subspace fixed on day one.

We propose redrawing that random subspace every round from a seed the server broadcasts, so all clients share the same subspace in a given round but the subspace moves over training. Each round, clients train only the small factor, clip it after projection, add noise, and upload it. The server averages, folds the product into the base weights, and draws a new subspace. Because the projection is shared, averaging stays linear and noise enters only once. Because the subspace keeps changing, the accumulated update is no longer capped at rank $r$. Our central claim is a theorem backed by experiments. This scheme converges to a stationary point of the full fine-tuning objective, not the restricted objective that a frozen adapter optimizes. It pays an explicit, measurable factor $k/r$ in the rate, and it does so at the per-round communication and noise cost of a rank-$r$ frozen adapter.

Core Claim

Consider one adapted weight matrix $W \in \mathbb{R}^{m \times k}$. In round $t$ the server broadcasts a seed that determines $A_t \in \mathbb{R}^{r \times k}$ with orthonormal rows drawn uniformly at random. Clients train only $B \in \mathbb{R}^{m \times r}$, initialized to zero, and the server applies

$$W_{t+1} = W_t + \bar B_t A_t, \qquad \mathbb{E}[A_t^\top A_t] = \tfrac{r}{k} I_k,$$

where $\bar B_t$ is the noisy average of the clipped client factors. The second identity is what makes the method work. A one-step update $G A_t^\top A_t$, with $G$ the full gradient of the loss with respect to $W$, is an unbiased estimate of $G$ up to the scale $r/k$. So over rounds the method behaves like randomized subspace descent on the whole matrix, rather than descent restricted to a fixed random kernel.

The privacy step exploits the projection. The quantity a client actually transmits is its update expressed in the $r$ directions of $A_t$. By Johnson–Lindenstrauss concentration, which says random low-dimensional projections roughly preserve lengths, its norm concentrates near $\sqrt{r/k}$ times the full update norm. We therefore clip after projection, at a threshold $C_r \approx \sqrt{r/k}\,C$. Clipping after projection is always a valid sensitivity bound, and it lowers the Gaussian noise needed for a given privacy level $(\epsilon, \delta)$ by the same factor. The only cost is extra clipping bias when a projected norm happens to land far above its typical value.

A back-of-envelope descent calculation shows that, in the noise-dominated regime, per-round progress then roughly matches full-dimensional private averaging while uploading a fraction $r/k$ of the coordinates. Communication compression becomes nearly free in privacy-utility terms, instead of adding noise-selection problems on top of privacy noise. If clipping happens before projection, the advantage disappears, and our ablation targets exactly this.

Relation to Prior Work

The frozen-factor method Improving LoRA in Privacy-preserving Federated Learning supplies our starting mechanism. It freezes $A_0$ to linearize aggregation and eliminate the cross-product noise term, and it proves equivalence to a random-feature kernel. It also motivates the proposal, because its own open problem is the representational ceiling of that static subspace. We keep its linearity and single-noise-entry property and remove the ceiling.

Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models instead keeps both factors trainable by alternating their updates locally, adding a low-pass filter on uploads. It has no general non-convex convergence theorem under heterogeneity. We avoid its per-client, nonshared factors altogether.

FedRot-LoRA shows that averaging independently learned factors causes rotational misalignment and fixes it with Procrustes alignment, but without privacy noise. A shared per-round $A_t$ removes misalignment by construction, with no alignment step on noisy matrices.

Sketched Gaussian Mechanism for Private Federated Learning is the closest theoretical relative. It clips the full gradient, projects it with a dense isotropic Gaussian matrix, adds noise in the sketch, and proves privacy amplification plus convergence logarithmic in dimension. Its authors leave structured, cheaper projections open. Our per-layer seeded projections are such a structured operator, and they cost nothing beyond ordinary adapter training. We do not claim SGM's amplification, which relies on the sketch's randomness. Our gain comes instead from clipping in the projected space.

Two further papers constrain the analysis. Understanding Clipping for Federated Learning supplies the error decomposition into averaging error, noise variance, and clipping bias, and we add a projection-variance term to it. Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compression warns that biased compressors without error correction stall at a constant error. It is also why we choose an unbiased-in-expectation projection rather than magnitude-based selection.

Research Program

The theoretical part proves a non-convex convergence bound for the scheme with $K$ local steps, uniformly sampled partial participation, and client-level privacy. We expect the bound to decompose into four pieces. The first is the usual local-drift term under bounded gradient dissimilarity, an assumption we adopt openly rather than remove. The second is a projection-variance term scaling with $k/r$. The third is a noise term whose dimension dependence is set by the post-projection clipping threshold. The fourth is a clipping-bias term controlled by JL concentration of projected norms.

The contrast result is the key deliverable. The frozen-factor method converges only to stationarity of the restricted objective. The rotating method drives the full gradient norm $\|G\|$ toward zero at rate slowed by $k/r$. We will also state honestly where the bound is loose: drift under extreme label skew, and the interaction between local steps and a subspace that is fixed within a round.

The algorithmic part builds the method on top of standard adapter code. The build includes seeded generation of $A_t$ via a fast orthonormalization, and a rotation period hyperparameter so that rotating every round and never rotating are the two endpoints. It also includes a catch-up protocol: clients that miss rounds replay the broadcast pairs of seed and $\bar B_t$, whose downlink cost we will measure rather than hide.

Baselines

The naive reference point is vanilla private federated adapters, where noise is added to both factors. On MNLI at $\epsilon=1$, Improving LoRA in Privacy-preserving Federated Learning reports 33.80% accuracy for this approach, which is chance level. We also report non-private federated adapters as a ceiling, and full-parameter private averaging with difference clipping, following Understanding Clipping for Federated Learning, as the high-communication reference.

The strongest competitor on our exact question is the frozen-factor method, because it shares our communication cost and noise structure. It reports 75.05% on MNLI at $\epsilon=1$ in Improving LoRA in Privacy-preserving Federated Learning, and 17.12% on GSM8K with LLaMA-7B. We rerun it inside our codebase at the same model, client split, privacy accountant, number of rounds, and hyperparameter search budget. We sweep its rank so that we compare both at equal rank and at equal total uploaded bits.

On vision, the strongest reported private result is alternating local updates. Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models reports 60.68% on Tiny-ImageNet with Swin-B at $\epsilon=1$. We rerun it under our splits with its filter enabled. We include a sketched-mechanism baseline following Sketched Gaussian Mechanism for Private Federated Learning on the BERT-Base SST-2 setting. For that setting the file reports no exact accuracy, only a noise scale of 0.0883 versus 0.8 on EMNIST, so we treat its numbers as ones we regenerate.

Beating the frozen-factor method means higher accuracy at equal $\epsilon$ and equal uploaded bits, with the margin growing with training length and task difficulty. Two ablations isolate the ingredients. The first varies only the rotation period, from rotating every round to never rotating. The second holds everything else fixed and clips before projection versus after projection.

Why not just…?

Why not just raise the rank of the frozen factor? This is the real threat. A frozen rank-$4r$ adapter may match a rotating rank-$r$ adapter on moderate tasks, and it needs no catch-up protocol. Its costs scale with rank, however: upload grows fourfold, and the noise dimension grows too. We settle the question with an equal-bits comparison and a rank sweep. If frozen adapters at matched communication match us across MNLI, GSM8K, and Tiny-ImageNet, the practical case collapses to the theoretical one alone, and we will say so.

Why not just use alternating local updates of both factors? Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models avoids the frozen ceiling, but its factors differ across clients. That reintroduces the misalignment documented by FedRot-LoRA, and the method has no convergence theorem under heterogeneity. The Tiny-ImageNet comparison and increasingly skewed Dirichlet splits will show whether shared rotation holds up better as heterogeneity grows.

Why not just sketch the full gradient as in the sketched Gaussian mechanism? That approach needs a dense $b \times d$ Gaussian matrix per client per round. Its authors flag this as impractical for large models and leave structured sketches open. We report client compute and memory alongside accuracy. If it wins on accuracy at feasible scale, that is informative, and it would tell us that sketch-randomness amplification matters more than post-projection clipping.

Why not just add noise and then prune the top coordinates? After full-dimensional noise, the largest-magnitude coordinates are dominated by noise. Without error correction, biased selection leaves a non-vanishing error floor, as Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compression proves. We include this as a cheap baseline at matched bits.

Proposed Experiments

We use four benchmark settings. For MNLI with RoBERTa, we follow the $\epsilon=1$ protocol of the frozen-factor paper and run 50 clients as in FedRot-LoRA. For Tiny-ImageNet with Swin-B at $\epsilon=1$, we use Dirichlet label splits. We also run GSM8K with a 7B model, and the fed-legal suite on Flower Hub to measure real uplink and downlink volume and round latency. Every run uses $\epsilon \in \{0.5, 1, 4\}$, several participation fractions, and three seeds, and reports accuracy, total transmitted bytes, and client memory.

Three checks decide the proposal. First, we periodically compute the full gradient norm of the fine-tuning loss on held-out client data. The theory predicts it keeps falling under rotation but plateaus under a frozen factor. If both plateau together, the core claim is wrong. Second, the gap between clipping before and after projection should grow roughly like $\sqrt{k/r}$ as rank shrinks. If no gap appears, the privacy argument is not what drives performance. Third, the advantage over equal-bits frozen adapters should widen with rounds and task difficulty, from MNLI to GSM8K. A flat or negative margin would mean the static subspace was never the binding constraint in practice.