Low-rank adaptation (LoRA) fine-tunes a large model by learning a small update $BA$: a tall matrix $B$ times a wide matrix $A$, added to a frozen weight. Differential privacy trains with per-example gradient clipping plus Gaussian noise (DP-SGD), which bounds how much any one training example can influence the model. Combined with LoRA, it works badly. Both factors receive noise, and the noise in one gets multiplied by the other. The literature calls this quadratic noise amplification. The standard remedies either freeze one factor, which caps what the model can learn, or move the noise into the tangent space of low-rank matrices, which is the set of directions in which a rank-$r$ matrix can change. The second remedy roughly doubles the cost of each training step.
We argue that the part of the problem worth fixing is mostly a coordinate problem, not a product problem. The same update $BA$ can be written in infinitely many ways, as $(BM)(M^{-1}A)$ for any invertible $r\times r$ matrix $M$. DP-SGD's clipping and noise depend on which way is chosen. We propose that after every noisy step, the factors be re-expressed in one canonical "balanced" form. This costs a single $r\times r$ decomposition and consumes no privacy budget, because it only post-processes already-privatized parameters. The re-expression makes the privacy mechanism exactly invariant to the choice of factors. It also minimizes the noise that lands in the actual weight update. If the claim holds, private LoRA can keep both factors trainable at essentially vanilla LoRA cost.
Core Claim
Call the freedom to rescale factors without changing their product "gauge freedom." After first-order noise propagation, one DP-SGD step with clip norm $C$, noise multiplier $\sigma$ and learning rate $\eta$ injects $\eta\sigma C(B\Xi_A + \Xi_B A)$ into the $d\times k$ weight update, where $\Xi_A,\Xi_B$ are standard Gaussian matrices. The expected energy of this term depends on the gauge, but only down to a gauge-invariant floor:
$$ \mathbb{E}\,\|B\Xi_A+\Xi_B A\|_F^2 \;=\; k\|B\|_F^2 + d\|A\|_F^2 \;\ge\; 2\sqrt{dk}\,\|BA\|_*, $$
Here $\|\cdot\|_*$ is the nuclear norm, the sum of singular values. The bound follows from the variational identity $\min_{BA=X}\tfrac12(\|B\|_F^2+\|A\|_F^2)=\|X\|_*$ after rescaling $B$ and $A$ by $(k/d)^{\pm1/4}$. Equality holds exactly when the factors are weighted-balanced, meaning $\sqrt{k}\,B^\top B=\sqrt{d}\,AA^\top$.
Once factors are held balanced, the only freedom left is an orthogonal rotation inside the $r$-dimensional space. Per-example gradient norms and isotropic Gaussian noise are both unchanged by such rotations. So the distribution of the DP update becomes a function of $BA$ alone, which is the gauge invariance that PRISM obtains through tangent-space projection.
Without balancing, noise accumulating in $A$ inflates $\|A\|$. That inflation scales up every later noise draw in $B$, and vice versa. We hypothesize that this factor drift, not the second-order term $N_BN_A$, drives most of the collapse reported for vanilla private LoRA. Per step, the second-order term is suppressed by an extra factor of $\eta\sigma C$ relative to the linear terms.
Relation to Prior Work
Two papers motivate the proposal by documenting the failure. FFA-LoRA freezes a random $A$ so that only $B$ is noised. It reports vanilla LoRA collapsing to 33.80% on MNLI at $\epsilon=1$, against 75.05% for the frozen variant. PRISM identifies gauge sensitivity of clipping and noise as a root cause. It fixes this by projecting gradients and noise onto the rank-$r$ tangent space. We share PRISM's diagnosis but replace its roughly 2× per-step overhead with an $O((d+k)r^2)$ re-factorization.
Three papers constrain the design by showing that freezing costs accuracy. FedSVD periodically re-orthonormalizes $A$ on the server and beats FFA-LoRA by +9.63 points on GLUE at $\epsilon=3$. FedASK reports a 46% relative GSM8K gain from updating both factors. Stiefel-LoRA finds that geometric constraints lose their benefit when $A$ is fixed. Our method differs from FedSVD in three ways. It canonicalizes at every step, not every few rounds. It uses the noise-minimizing balanced gauge rather than orthonormal $A$ with all scale in $B$. It needs no server broadcast.
Riemannian Preconditioned LoRA supplies the closest method: cheap $r\times r$ preconditioners that make non-private updates gauge-aware. It never touches clipping or noise, and it acts on the gradient rather than the coordinates in which the gradient is clipped. LA-LoRA avoids same-step noise products by alternating which factor is updated. That addresses the second-order term but not drift in factor scale.
LoRAM and PiSSA matter for initialization. In the balanced gauge, $B=0,A\neq0$ collapses to the saddle $B=A=0$, so the method needs a small nonzero starting product. How large it should be under privacy is an open question that the magnitude results make pointed.
Research Program
We will build the method as a drop-in change to DP-SGD/DP-AdamW for LoRA. After each noisy update, we compute thin QR decompositions of $B$ and $A^\top$, take an SVD of the resulting $r\times r$ core, and redistribute its singular values with the $(k/d)^{1/4}$ weighting. Adam moment buffers are rotated along with the factors so that the optimizer state stays consistent with the new coordinates. We will verify that the privacy accounting is unchanged, since the step is pure post-processing. We will also report wall-clock cost per step against vanilla DP-LoRA and PRISM.
The first analysis is a noise decomposition. During private training we log the realized update error, split into four components: the linear noise terms, the second-order $N_BN_A$ term, clipping bias, and the growth of $\|A\|_F$, $\|B\|_F$ and their imbalance ratio. We do this for vanilla DP-LoRA, FFA-LoRA and our method. This tests directly whether factor drift or the second-order term dominates.
The second analysis is a gauge-invariance test. We apply random invertible $M$ at initialization, with no change to the product, and measure the spread of final accuracy. Theory predicts large spread for vanilla DP-LoRA and zero spread, up to seed noise, for ours. A secondary study transfers the method to the federated setting of FFA-LoRA and FedSVD. There, clients canonicalize locally at every step, and the server aggregates and re-canonicalizes. This asks whether local balancing removes the need for FedSVD's broadcast of a new basis.
Baselines
The naive reference is vanilla DP-LoRA with DP-AdamW, trained with both factors and swept over learning rate and clip norm. PRISM's paper reports it at 0.641 average accuracy on Math-10K at $\epsilon=6$, and FFA-LoRA's paper reports it at 33.80% on MNLI at $\epsilon=1$. Any improvement must first clear this bar under an equal tuning budget.
The strongest existing centralized method is PRISM, which reports 0.690 on Math-10K at $\epsilon=6$ on Gemma models. It is the direct competitor because it targets the same gauge problem. We run it with its released recipe and give our method the same number of hyperparameter trials, the same privacy accounting and the same model. We count "beating" in two ways. On a matched step count, our method should come within 0.5 points of 0.690 or exceed it. On a matched wall-clock budget, our method should exceed PRISM, since PRISM completes roughly half as many steps.
The strongest freezing baseline is FFA-LoRA, at 75.05% on MNLI at $\epsilon=1$. We run it with the same rank, scaling factor and tuning sweep. We must beat it at $\epsilon\in\{1,3,6\}$ to justify training both factors.
We also include LoRA+ under DP, with the ratio $\lambda$ swept rather than fixed at 16. For the federated check, FedSVD is the reference; its GLUE result is reported only as +9.63 points over FFA-LoRA on RoBERTa-large at $\epsilon=3$, and the file gives no absolute score.
Two ablations isolate the key ingredient. The first replaces weighted balancing with FedSVD-style orthonormal $A$ at every step, which is also a canonical gauge and therefore also invariant but not noise-minimizing. This separates the effect of invariance from the effect of minimizing noise. The second canonicalizes only every $T$ steps, with $T\in\{1,10,100\}$, which tests whether drift between resets matters.
Why not just…?
Why not just freeze $A$ as FFA-LoRA does? This is the most serious threat at $\epsilon=1$. Freezing removes all noise in $A$ at no cost, and at very tight budgets a fixed random subspace may be the best available trade. The evidence against it as a general answer is FedSVD's +9.63-point gain and FedASK's 46% relative GSM8K gain, both obtained by letting $A$ move. Our $\epsilon$ sweep will show where, if anywhere, the curves cross. If FFA-LoRA wins at $\epsilon=1$, we will report that as the regime boundary.
Why not just use PRISM? PRISM already delivers gauge invariance, and our balanced gauge might discard something its tangent-space noise captures, such as the exact geometry of the noise. If PRISM beats us by more than a point at matched steps, the cheap route is insufficient. The matched-step and matched-wall-clock comparisons settle this directly.
Why not just tune vanilla LoRA or LoRA+ harder? The unified study of LoRA variants shows that tuned vanilla LoRA rivals complex variants, but only in non-private training. Tuning a learning-rate ratio cannot stop the norms of randomly walking factors from growing. Our gauge-invariance test makes this concrete: if tuned vanilla DP-LoRA shows no spread under random re-gauging and no factor drift in the logs, our mechanism is unnecessary, and the logs will show it.
Why not just apply the Riemannian preconditioner to the noised gradient? That is free post-processing and cheap. But it leaves clipping in gauge-dependent coordinates, and its preconditioned noise $B(B^\top B+\delta I)^{-1}\Xi_A$ still depends on factor scale. We include it as a baseline. If it matches our method, the contribution shrinks to the analysis alone.
Decisive Checks
The main experiments are GLUE (MNLI, SST-2, QNLI, QQP) on RoBERTa-large at $\epsilon\in\{1,3,6\}$, which matches FFA-LoRA and FedSVD, and Math-10K on Gemma-3-4B at $\epsilon=6$, which matches PRISM. Each runs over three seeds, with accuracy and step time reported.
The proposal succeeds if three things hold. First, our method closes most of the gap between vanilla DP-LoRA (0.641) and PRISM (0.690) at under 1.1× vanilla step cost. Second, it beats FFA-LoRA at $\epsilon=3$ and $\epsilon=6$. Third, the decomposition logs show factor-norm drift, not $N_BN_A$, as the dominant error in vanilla training.
The proposal fails informatively in either of two ways. If the orthonormal-$A$ ablation matches weighted balancing, then invariance matters and noise minimization does not. If the second-order term dominates the error logs, then alternating schemes like LA-LoRA are attacking the right problem and ours is not.
References
- PRISM: Gauge-Invariant Tangent-Space Differentially Private LoRA
- Improving LoRA in Privacy-preserving Federated Learning
- FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA
- Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix
- Riemannian Optimization for LoRA on the Stiefel Manifold
- Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models
- Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models
- The Primacy of Magnitude in Low-Rank Adaptation
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
- LoRA+: Efficient Low Rank Adaptation of Large Models
- A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation