Carrying Adam's Memory Through LoRA Rebalancing

Low-rank adaptation (LoRA) fine-tunes a large model by learning a small update written as a product of two thin matrices. The same update can be written in infinitely many ways, because one factor can be scaled or rotated as long as the other is inversely compensated. Recent methods exploit this freedom. After each optimizer step they reshuffle the two factors into a "balanced" form that leaves the update unchanged but makes the loss surface better conditioned. These methods are run with Adam, which keeps a running memory of past gradients for every individual parameter. The reshuffling changes what each parameter means, but Adam's memory is left as it was. Our claim is that this stale memory is a hidden source of error in balanced LoRA. We propose to fix it by moving Adam's memory along with the factors, and by choosing, among all balanced forms, the one that disturbs that memory least.

The idea fits in two sentences. If you re-express the factors in new coordinates, Adam's stored momentum has to be re-expressed in the same coordinates, or the next step points in a distorted direction. We derive the exact rule for the momentum, an approximate rule for Adam's per-parameter scale estimates, and a choice of balanced form that keeps the coordinate change as close to the identity as possible. We then test whether this makes balanced LoRA faster and less sensitive to the learning rate.

Background and the gap

Classical work on factorized matrix problems established that factor imbalance is harmful. Adding the penalty $\tfrac14\|U^\top U - V^\top V\|_F^2$ yields linear convergence in Finding Low-rank Solutions to Matrix Problems, Efficiently and Provably. The same penalty removes spurious local minima in No Spurious Local Minima in Nonconvex Low Rank Problems, and it makes convergence rates independent of the condition number in Procrustes Flow. All of these results concern plain gradient methods on linear recovery problems. They motivate balancing but say nothing about adaptive optimizers.

Three recent papers carry balancing into LoRA, and they supply the methods we build on.

Balanced LoRA (BaLoRA) works with an update $\Delta W = AB$, where $A \in \mathbb{R}^{d\times r}$ and $B \in \mathbb{R}^{r\times k}$. After every step it projects the factors with an inexpensive singular value decomposition so that $A^\top A = BB^\top$, preserving $AB$ exactly. It proves that, under plain gradient descent, this is a Riemannian method under the Bures metric. Its authors also state openly that how the projection interacts with Adam's moment estimates "is still an open question."

RefLoRA instead solves for an optimal symmetric positive definite refactoring at each step. It also offers a cheap scalar version, RefLoRA-S, which keeps 98.7% of vanilla LoRA's throughput.

LORO performs Riemannian updates on the set of rank-$r$ matrices, but only for pretraining without a frozen base weight.

None of these three transforms the optimizer state when it transforms the parameters. That omission is the gap we target, and it is a different contribution from all three. We do not propose a new balanced manifold or a new refactoring objective. We keep BaLoRA's projection target and change what happens to the optimizer around it.

Core claim

Let a rebalancing step replace $(A,B)$ by $(AP,\,P^{-1}B)$ for some invertible $r\times r$ matrix $P$. The gradients in the new coordinates are $G_A P^{-\top}$ and $P^\top G_B$, where $G_A$ and $G_B$ are the old gradients with respect to $A$ and $B$. Because Adam's first moment is a running average of gradients, it is exactly transported by

$$m_A \leftarrow m_A P^{-\top}, \qquad m_B \leftarrow P^\top m_B .$$

Adam's second moment is different. It is an elementwise average of squared gradients, so it cannot be transported exactly under a general $P$; Adam is not invariant to rotations of the factor coordinates.

There is one case where transport is exact. If $P = \mathrm{diag}(p_1,\dots,p_r)$, column $j$ of $m_A$ scales by $1/p_j$, row $j$ of $m_B$ scales by $p_j$, and the second moments scale by the squares of those factors.

This leads to the design. The set of balanced representations is still an orbit under orthogonal $r\times r$ matrices: if $(A,B)$ is balanced, so is $(AQ, Q^\top B)$ for any orthogonal $Q$. BaLoRA's singular-value gauge picks one member of that orbit. When two singular values cross during training, that choice can swap or rotate coordinates abruptly, which scrambles Adam's per-coordinate scale estimates. We instead pick the balanced representation closest to the current iterate. Finding it is an orthogonal Procrustes problem in $r\times r$ dimensions, so the total transform $P$ stays near the identity.

We then split $P$ into a diagonal part and a small residual rotation. We transport the first moment exactly through all of $P$, the second moment exactly through the diagonal part, and the second moment approximately through the rotation by applying it to the square root of $v$. The extra cost is $O((d+k)r^2)$, the same order as BaLoRA's projection, and no new memory beyond the existing moment buffers.

We predict three things relative to BaLoRA with untouched moments. Transported balancing should reach a target loss in fewer steps. It should tolerate a wider range of learning rates before diverging. Its advantage should grow with rank, because BaLoRA's gains concentrate at $r=64$ to $128$, and because LoRA Training Provably Converges… Or It Fails Loudly shows that conditioning worsens as rank grows.

A further consequence is specific to our approach. Once moments are transported, balancing need not happen every step. If transport removes most of the harm from stale moments, rebalancing every $K$ steps should match per-step rebalancing. LORO's lazy schedule suggests this kind of savings is attainable.

Research Program

The first step is measurement before any method change. We instrument BaLoRA under AdamW on LLaMA-3.2-3B. At each step we record the transform $P$ it applies, split into a scaling part and a rotation part. We also record how far the stale momentum direction lies from the transported one, measured as a cosine.

This is a go or no-go check. If the rotation part is negligible and the scaling part is close to 1 after the first few hundred steps, stale moments cannot matter much and the project should stop there. If instead we see large early rescalings, or rotation spikes at singular-value crossings, the mechanism is real.

The second step is the method: Procrustes-gauged balancing with moment transport, as described above.

The third step addresses initialization. Standard LoRA starts with $B=0$. Balancing a zero product returns two zero factors, which would stop training. So the balancing target is undefined at the start, a problem that RefLoRA also flags for its own method. Our default is to begin projection once $\|B\|_F$ passes a small threshold, transporting moments from that point onward.

The fourth step is a controlled study of LoRA+. LoRA+ makes the learning rate of $B$ 16 times that of $A$ ($\eta_B = 16\,\eta_A$), so it deliberately treats the factors asymmetrically. Balancing pushes them toward symmetry. These two prescriptions appear to conflict, and the literature leaves the conflict unresolved. Before choosing any combination of the two, we will measure the imbalance $\|A^\top A - BB^\top\|_F$ under LoRA+ in our setup.

We will begin with synthetic asymmetric matrix sensing trained with Adam. There the condition number and the exact gauge are observable, and the transport rule can be checked against an exact Riemannian reference.

Baselines

The naive reference is vanilla LoRA with AdamW: $B=0$ initialization, independent factors, and standard weight decay. That last choice matters because the Fails Loudly paper shows that weight decay on the factors acts as a nuclear-norm regularizer. Any gain must first beat this baseline.

The strongest direct competitor is BaLoRA itself, run with the authors' per-step projection and unmodified AdamW moments. It is exactly the method our proposal claims to improve. The file reports that it achieves lower test loss than LoRA, DoRA and OLoRA on GSM8K and Wikitext-2, but no reported result was found on Lacuna for BaLoRA's numerical GSM8K accuracy or Wikitext-2 perplexity. We therefore rerun it ourselves on identical data, model, rank and token budget. Each method gets the same learning-rate sweep of the same size, and each is run over three seeds. Beating BaLoRA means two things. First, reaching BaLoRA's final training loss in clearly fewer steps, beyond seed noise, at $r\ge 64$. Second, either matching or improving final GSM8K accuracy, or widening the range of learning rates that do not diverge.

The cheap-refactoring competitor is RefLoRA-S, run under the same protocol.

The zero-cost competitor is LoRA+ with $\lambda=16$. Its reported gains are 1–2% over LoRA on GLUE and +1.3% on MMLU for LLaMA-7B, according to LoRA+.

As an external reference point on accuracy, we include LoRA-Pro. It reports 57.57% GSM8K accuracy at $r=8$ against 42.08% for LoRA, and a GLUE average of 88.44 against 87.91 for full fine-tuning, according to LoRA-Pro. It also runs at about 60% of LoRA's throughput. We report throughput alongside accuracy so that its accuracy is weighed against its cost.

Ablations isolate the key ingredient. We test BaLoRA with Procrustes gauge but no transport, BaLoRA with first-moment transport only, BaLoRA with full transport, and BaLoRA with moments reset to zero after each projection. Of these, the reset ablation is the most informative crude alternative.

Why not just…?

Why not just reset Adam's moments after each rebalancing? Resetting throws away all of Adam's gradient memory every step, which should behave like a much weaker optimizer. It is cheap, though, and we run it as an ablation. If reset matches transport, the transport rules add nothing, and we would report that.

Why not just use LoRA+ and skip balancing? This is the real threat for low ranks. LoRA+ costs nothing and reliably helps. Two pieces of counter-evidence exist. SingLoRA reports 91.3% on MNLI with LLaMA-7B against 90.2% for LoRA+. BaLoRA reports that unbalanced LoRA plateaus or diverges at $r=64$ to $128$. Neither result tests LoRA+ at high rank. Our rank sweep from 8 to 128 does: if LoRA+ matches transported balancing at all ranks, the method is unnecessary.

Why not just add the classical Gram penalty or rely on weight decay? The penalty adds a sensitive weight that must be tuned. Weight decay limits the total size of the factors but not their relative scale. BaLoRA reports that regularized LoRA still exhibits severe conditioning disparities. We include a tuned Gram penalty at equal tuning budget to check this claim directly.

Why not just rebalance less often? That is part of our hypothesis rather than an alternative to it. Without transport, rare rebalancing should cause occasional large jolts to the optimizer. With transport, rare rebalancing should approach per-step rebalancing. The $K$-sweep below tests this.

Proposed Experiments

On synthetic matrix sensing with Adam, we compare loss trajectories and the observed Hessian condition number against an exact Riemannian reference.

On LLaMA-3.2-3B and Qwen-2.5-3B, we fine-tune on MetaMathQA and evaluate GSM8K accuracy, and we fine-tune and evaluate on Wikitext-2 perplexity. Ranks are 8, 32, 64 and 128. We measure steps to target loss, final accuracy, and the divergence rate over the learning-rate sweep.

On DeBERTaV3-base, we report the GLUE average, matching RefLoRA's setting.

Three decisive checks determine the outcome. The measured stale-momentum cosine must be meaningfully below 1 for the mechanism to exist. The full-transport method must beat BaLoRA and moment reset at $r\ge64$ for it to matter. Rebalancing every $K$ steps with transport, for $K$ from 1 to 100, must stay near the per-step result where untransported BaLoRA degrades for the efficiency claim to hold.