Low-rank adaptation (LoRA) fine-tunes a large frozen model by training a small add-on update written as the product of two thin matrices. Newer variants change the size of that add-on during training. They prune some rank-one pieces, reactivate others, or move rank budget between layers. The optimizer underneath is almost always Adam, which keeps two running averages per weight: a mean of recent gradients and a mean of their squares. Adam then rescales both by a "bias correction" factor computed from one global step counter. That factor is only correct for weights that have been receiving gradients since step one. A rank component that is born or revived at step 3,000 has empty or stale averages, yet the global counter tells Adam they are fully warmed up. For dynamic sparse pre-training, Memory-Efficient LLM Training with Dynamic Sparsity showed that this mismatch makes the first update of a regrown weight about 3.16× too large and produces sawtooth loss spikes. No dynamic-rank LoRA method tracks this.
This proposal claims that in parameter-efficient fine-tuning, Adam's bias correction barely matters at the start of training but matters exactly when the adapter's structure changes. A global warmup schedule cannot help at that point. The proposed fix is to give each rank-one component its own optimizer clock. Steps where a component is inactive are treated as missing observations, not as zero gradients. The fix costs two scalars per component and contains standard Adam and the reset used by dynamic sparse training as special cases. It also handles a case neither covers: a component that comes back after a long dormancy with decayed, stale statistics.
Core Claim
Write a LoRA update as a sum of rank-one components. Component $k$ is the pair formed by column $k$ of the output factor $B$ and row $k$ of the input factor $A$, plus a singular value in the factorized form used by AdaLoRA. Let $a^{(k)}_t \in \{0,1\}$ indicate whether component $k$ is active at step $t$, and let $g_t$ be its gradient. The proposed estimator keeps Adam's running mean $m_t$ (decay $\beta_1$) unchanged in form. It adds a running "observed mass" $c_t$ that accumulates only on active steps:
$$m_t = \beta_1 m_{t-1} + (1-\beta_1)\,a_t g_t,\qquad c_t = \beta_1 c_{t-1} + (1-\beta_1)\,a_t,\qquad \hat m_t = m_t / c_t .$$
The squared-gradient average $v_t$ (decay $\beta_2$) gets the same treatment with its own mass $d_t$, giving $\hat v_t = v_t/d_t$. If a component is always active, then $c_t = 1-\beta_1^t$, which is exactly Adam's correction. If a component is freshly created with zero state, $c$ starts from zero, which reproduces the per-weight timestep reset of SMET. If a component was dormant for $k$ steps, both $v$ and $d$ have decayed by $\beta_2^k$. Their ratio therefore still estimates the old gradient variance instead of collapsing toward zero, and the estimator neither discards that information nor treats it as current.
The size of the error being corrected is easy to work out. A fresh component under the global counter receives an update inflated by the inverse of SMET's scaling factor, evaluated at the component's local age $s$:
$$\frac{1}{p(s)} = \frac{1-\beta_1^{s}}{\sqrt{1-\beta_2^{s}}}.$$
With the defaults $\beta_1=0.9$ and $\beta_2=0.999$, this is 3.16 at $s=1$, the figure SMET reports. Plugging in later values of $s$ shows it is not the worst case. The inflation rises to roughly 6.5× around the tenth to fifteenth step, is still about 3× at step 100, and stays above 1.2× at step 1,000. The overshoot therefore lasts hundreds of steps, which is comparable to the interval between budget updates in dynamic-rank schedules.
With $\beta_1=\beta_2=0.95$, the setting Simplifying Adam: Bias Correction Debunked recommends, the same formula gives about 0.22 at $s=1$. The error flips sign: new components are under-stepped rather than over-stepped. This sign flip is a sharp, falsifiable prediction. The same structural event should cause spikes under default β and sluggish learning of the new capacity under tied β, and the per-component clock should fix both.
Research Program
The work has three parts. The first part asks whether bias correction matters in ordinary, fixed-rank LoRA. The two sides of the literature disagree. Few-sample full fine-tuning of BERT collapsed without it: Revisiting Few-sample BERT Fine-tuning reports 48% failed RTE runs, and On the Stability of Fine-tuning BERT attributes the fix to an implicit warmup. Large-scale pre-training with an explicit warmup does not care about it at all, according to Simplifying Adam. Neither side tested LoRA. We cross bias correction (on/off) with explicit warmup (on/off) and two β settings on few-sample GLUE tasks.
Our hypothesis is that, with warmup, LoRA behaves like the "debunked" regime. That is the premise for the rest of the work: if bias correction is irrelevant at the start, any remaining effect must come from mid-training structural events. If instead LoRA without warmup fails like BERTAdam did, that is a useful finding in its own right, and it raises the stakes for the dynamic case.
The second part is instrumentation. We log a per-component quantity at every structural event in three settings: AdaLoRA (pruning and possible unmasking of singular triplets), DoRA's dynamic rank distribution (DoRA, which prunes rank-one triplets under a cubic schedule), and PRILoRA (PRILoRA, which periodically prunes entries of $A$). The logged quantity is the ratio of the component's actual update norm to the update norm a steady-state Adam would give it.
It is genuinely uncertain how often these methods revive components, and whether masked weights receive zero gradients (so their state decays) or no gradients (so their state freezes). The answer depends on implementation. This audit determines which failure mode is real in practice. We expect AdaLoRA-style reactivation to produce stale, under-scaled restarts and a momentum tail that keeps moving masked weights.
Because the existing methods may mostly prune, we add a controlled rank-growth testbed. LoRA starts at rank 2 and adds components at fixed steps, following standard LoRA initialization of the new slice (zero column of $B$, Gaussian row of $A$), up to a target rank. This is the cleanest analogue of SMET's regrowth and the setting where the 1/p(s) curve should appear most directly.
The third part implements the per-component clock as a drop-in Adam variant. We apply it to the growth testbed and to the three dynamic-rank methods, and we compare it against the alternatives below under matched tuning.
A minor side question is the one-step lag of $A$ in standard LoRA. Its gradient is exactly zero at step one because $B=0$. Working through the arithmetic, the global counter then under-scales $A$'s first real step only modestly, to about 0.74 of the local-clock value. We expect this to be negligible and will report it, but we do not build on it. Similarly, the observation by The Impact of Initialization on LoRA Finetuning Dynamics that $A$ and $B$ train at very different speeds is a separate issue. Its fix lives in learning-rate scaling, not in the optimizer's averages.
Baselines
The do-nothing reference is each dynamic-rank method run exactly as published, with global-counter Adam, bias correction on, and the authors' warmup schedule. On DeBERTaV3-base the file reports a GLUE average of 90.01 for PRILoRA, 89.46 for AdaLoRA, and 88.50 for fixed-rank LoRA (PRILoRA). AdaLoRA alone reports 87.36% on RTE with 0.32M parameters (AdaLoRA). On RoBERTa-base, DoRA reports 83.53 F1 on SQuAD v2.0 against 82.34 for LoRA (DoRA).
PRILoRA's 90.01 GLUE average is the strongest reported number and the target to beat. Our reproductions of all these methods will be rerun under our seeds before any comparison. We will not compare against the quoted numbers directly.
The strongest mechanistic competitor is SMET's optimizer-state warm-up transplanted to rank components (Memory-Efficient LLM Training with Dynamic Sparsity). It hard-resets a component's moments and local timestep whenever the component is created or revived. On fresh components it should match our method exactly, since the two are mathematically identical there. It differs only on revived components, where it discards the stored variance estimate. This comparison is therefore the ablation that isolates our key ingredient, namely treating dormant steps as missing data. If the hard reset matches us on revival-heavy schedules, the stale-state handling is unnecessary, and the contribution reduces to applying SMET's fix to structured rank.
The third baseline is a per-component learning-rate warmup. New or revived components get a linear ramp of tuned length on their own learning rate, with the global optimizer left unchanged. We also include fixed-rank LoRA at matched final parameter count, plus LoRA+ (decoupled learning rates for $A$ and $B$, as discussed in The Impact of Initialization on LoRA Finetuning Dynamics), to confirm that any gain is not just a learning-rate effect.
Every method gets the same data splits, the same number of steps, the same structural schedule, and the same budget of learning-rate and warmup-length search. That budget is a fixed grid of equal size for every method, including the per-component warmup length. Each configuration is run with ten seeds.
Beating a baseline means meeting three conditions together. The per-component clock must give equal or better mean score. It must give a statistically lower seed variance or failure rate on the small GLUE tasks. And it must remove the loss spike at structural events, measured as the peak training-loss jump within 50 steps of an event. It must do all this without tuning anything the baseline did not also get to tune.
Why not just…?
Why not just rely on the explicit warmup everyone already uses? A global warmup ends long before AdaLoRA or DoRA make most of their budget decisions. So the learning rate is at its peak exactly when a new component's averages are emptiest. This is the reason SMET's spikes appeared under a standard schedule. Part one of the program tests the start-of-training version of this argument directly. We expect warmup to make bias correction irrelevant there, which concedes the reviewer's point for static LoRA and isolates the mid-training case.
Why not just give new components their own short warmup? This is a real threat. For fresh components, a ramp proportional to $p(s)$ reproduces bias correction exactly, and a tuned linear ramp may come close. We include it as a baseline with its length tuned on the same budget. The method should differ in three situations. First, at tied β the error is under-stepping, so a ramp makes it worse. Second, a ramp has a length that must be tuned per schedule. Third, a ramp does nothing about stale variance on revived components. If the tuned ramp matches us everywhere, we will report that the problem is real but a schedule-level fix is enough.
Why not just switch to β₁ = β₂ = 0.95 so state refills quickly? The formula above predicts that this setting turns overshoot into undershoot. With β₂ = 0.95, fresh components move at about a fifth of the intended step and refill within roughly 20 steps. That may well remove the spikes, and Simplifying Adam found tied β fine in pre-training. We test it as a full arm crossed with every method. The question is whether the end score suffers from slower uptake of new capacity, or from noisier variance estimates throughout training.
Why not just skip dynamic rank and use well-tuned fixed-rank LoRA? Reported gaps over fixed-rank LoRA are about one to one-and-a-half GLUE points (88.50 versus 89.46 and 90.01 in PRILoRA). Part of that gap could be eaten by optimizer noise, or hidden by it. If the per-component clock widens the gap, dynamic allocation looks more valuable. If fixed-rank LoRA at matched parameters still ties after the fix, that is worth knowing too.
Proposed Experiments and Decisive Checks
The main testbed is GLUE (GLUE) on DeBERTaV3-base and RoBERTa-base. Results are reported separately for the low-data tasks RTE, CoLA, MRPC, and STS-B, where earlier work found instability concentrated. The metric for each task is its standard one: accuracy, Matthews correlation, F1, or correlation. We also report failure rate, using the criterion from Revisiting Few-sample BERT Fine-tuning (RTE accuracy below 55%), and seed standard deviation across ten seeds. SQuAD v2.0 F1 on RoBERTa-base and XSum ROUGE on BART-base extend the test to span extraction and generation. The file has no reported ROUGE numbers for these methods, so those comparisons rest entirely on our own reruns.
A scale check fine-tunes a 7B Llama model on GSM8k and MMLU with the rank-growth schedule. These benchmarks were used by The Impact of Initialization on LoRA Finetuning Dynamics, but no reported accuracy for them was found on Lacuna, so we will rely on our own baselines there too.
The first decisive check is mechanistic. In the growth testbed with default β, the measured update-norm ratio of new components should trace the 1/p(s) curve: about 3.2 at the first step, a peak near 6.5 around step ten, and slow decay afterward. Under the per-component clock the ratio should stay flat near 1. With tied β = 0.95, the uncorrected ratio should instead start near 0.22. If neither curve appears, the premise fails, and we stop.
The second check is causal. Loss spikes at structural events should disappear under the per-component clock and under the hard reset alike. The stale-state ablation should separate the two methods only on schedules that revive components. We will plot the separation against the measured revival rate from the audit, so that a null result on revive-free methods counts as a prediction confirmed, not as a failure.
The third check is the end-task payoff. On the low-data GLUE tasks, at matched tuning budget, the per-component clock should reduce failure rate and seed variance for the dynamic-rank methods. Their gap over fixed-rank LoRA should hold or grow. If the tuned per-component warmup does equally well on all three checks, we will report the diagnosis as the contribution and recommend the simpler fix.
References
- Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
- Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
- Simplifying Adam: Bias Correction Debunked
- Revisiting Few-sample BERT Fine-tuning
- On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
- DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution
- PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation
- The Impact of Initialization on LoRA Finetuning Dynamics
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding