Mitigating Quadratic Noise in Private Low Rank Adaptation

This direction is merged into another direction: quadratic noise amplification. Its last version stays readable and citable here.

Version 1 of 113 papers19 conceptsfrozen 2026-10-04

Low-Rank Adaptation (LoRA) has become the standard for fine-tuning large language models because it drastically reduces the number of trainable parameters. By representing a weight update $\Delta W$ as the product of two smaller matrices, $A$ and $B$, LoRA allows practitioners to update models with minimal memory and storage overhead.

However, when LoRA is deployed in a federated learning (FL) setting—where data remains on local devices—it faces a significant hurdle: Differential Privacy (DP). To protect individual user data, noise must be added to the model updates. In the context of LoRA, this noise interacts with the factorized structure of the matrices in a way that can lead to "noise amplification," often causing the training process to collapse.

The Quadratic Noise Problem

The core issue lies in the mathematical structure of the LoRA update. In standard LoRA, the weight change is calculated as $\Delta W = AB$. When we apply differential privacy, we must add noise to the updates of both $A$ and $B$ before they are aggregated by a central server.

If we denote the noise added to each matrix as $\epsilon_A$ and $\epsilon_B$, the resulting update becomes: $$(A + \epsilon_A)(B + \epsilon_B) = AB + A\epsilon_B + \epsilon_A B + \epsilon_A\epsilon_B$$

As noted by Youbang Sun and Zitao Li in Improving LoRA in Privacy-preserving Federated Learning, this expansion reveals three noise terms. The first two ($A\epsilon_B$ and $\epsilon_A B$) scale linearly with the weights. The final term, $\epsilon_A\epsilon_B$, represents a "noise-times-noise" interaction.

This interaction leads to quadratic noise amplification. Under strict privacy budgets where the noise scale $\epsilon$ must be large, this quadratic term grows rapidly, overwhelming the actual learning signal ($AB$). Jin Liu and Yinbin Miao demonstrate in Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models that this effect is the primary cause of training instability and performance degradation in private federated fine-tuning.

Decoupling the Interaction

To solve this, researchers have moved toward "decoupling" the two matrices to prevent the noise terms from multiplying.

One of the most effective strategies is to simply stop training one of the matrices. In FFA-LoRA, Youbang Sun et al. propose freezing the randomly initialized matrix $A$ and only training the zero-initialized matrix $B$. Because $A$ is never updated, no DP noise is added to it ($\epsilon_A = 0$). This eliminates the quadratic $\epsilon_A\epsilon_B$ term entirely, reducing the noise to a linear scale ($A\epsilon_B$). Similarly, Tianqu Kang and Jun Zhang suggest fixing one matrix as a scaled orthonormal matrix to maintain stability in Federated Low-Rank Adaptation with Differential Privacy over Wireless Networks.

Alternating Optimization

Another approach is to update the matrices sequentially rather than simultaneously. Jin Liu et al. introduce LA-LoRA (Local Alternating LoRA), which alternates between updating $A$ and $B$. By ensuring that only one matrix is being "noised" at any given step of the calculation, the interaction between noise terms is minimized.

This concept is also central to the DEeR framework developed by Axiu Mao and Meilu Zhu. In DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptation, the authors use an alternating optimization strategy combined with a "noise regulator" to prevent the noise-to-signal ratio from increasing as the weights grow during training.

Geometric and Orthogonal Stabilization

Beyond just decoupling updates, the geometric properties of the low-rank matrices themselves can be managed to resist noise. Seanie Lee and Sangwoo Park introduce FedSVD (FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA), which periodically re-orients the adapter matrices using Singular Value Decomposition (SVD).

By keeping the matrices orthogonal, FedSVD prevents the weights from becoming poorly conditioned. This is crucial because, as Meilu Zhu points out, noise is scaled by the magnitude of the weights during matrix multiplication. If the weights grow unchecked or become skewed, the noise is amplified even further. Orthogonalization acts as a stabilizer, ensuring that the injected DP noise does not blow up the gradient signal.

Why It Works

These methods succeed because they transform the noise profile of the model update from a quadratic function of the noise scale to a linear one.

  1. Noise Reduction: By freezing one matrix or alternating updates, the $\epsilon \times \epsilon$ term is removed.
  2. Signal Preservation: By regulating the magnitude or orthogonality of the matrices (as in DEeR or FedSVD), the $Weight \times \epsilon$ terms are kept within a range that the optimizer can handle.
  3. Efficiency: These techniques often reduce communication costs—since only half the parameters are being sent to the server in any given round—while simultaneously improving the model's utility under strict privacy constraints.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mitigating_quadratic_noise_in_private_lo_2ade5dde,
  title = {Mitigating Quadratic Noise in Private Low Rank Adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-quadratic-noise-in-private-low-rank-adaptation/rdr_2ade5dde9dc14c9e8e2cb346ea478066},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_mitigating_quadratic_noise_in_private_lo_2ade5dde_v1,
  title = {Mitigating Quadratic Noise in Private Low Rank Adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-quadratic-noise-in-private-low-rank-adaptation/rdr_2ade5dde9dc14c9e8e2cb346ea478066/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}