Reversible Rank Budgets from Synaptic Resource Competition

Low-rank adaptation (LoRA) fine-tunes a large pretrained network by adding a small, trainable update to each weight matrix. The update is a product of two thin matrices, and its "rank" is the number of independent directions it can change. Methods that allocate rank dynamically, such as AdaLoRA and a rank-pruning method also named DoRA, give more rank to layers that need it. They do this by scoring components and then pruning the weak ones on a fixed schedule. That works when fine-tuning on a single, stationary task. It is a poor fit when the data keeps changing, because capacity spent early cannot easily be recovered, and the network may later need it somewhere else. Neuroscience has a mechanism for exactly this problem. Synapses on the same dendritic branch compete for a finite, conserved pool of molecular building blocks. When one synapse strengthens, it drains the pool and its neighbours weaken. When a synapse weakens, its building blocks return to the pool for others to use.

We propose to build LoRA whose rank budget obeys this conservation law. Each rank-one component holds a magnitude drawn from a per-layer pool of fixed size. Magnitude flows back into the pool when a component stops being useful. A slower pool trades budget between layers. Components that hold no magnitude keep their learned directions and continue to learn slowly, much like biological "silent synapses," so they can be re-recruited when the data shifts. The claim is that reversible allocation of this kind retains the ability to learn new data (plasticity) better than scheduled pruning in streaming and continual fine-tuning, and matches it on single-task fine-tuning.

Core Claim

Three findings in the literature motivate the proposal. First, Factor Imbalance and Plasticity Loss in Low-Rank Factorized Networks shows that factorized networks lose the ability to learn new tasks faster than full-rank ones. The cause is that the two factors drift apart in scale, measured by the Gram mismatch $\|U^\top U - V^\top V\|_F$ between the factors $U$ and $V$. Higher rank can make this worse. Their fix is a balance penalty at fixed rank, and they report that it recovers plasticity only partially at low rank.

Second, the dynamic-rank methods Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning and DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution reallocate rank using sensitivity or component-norm scores on a cubic pruning schedule. That schedule assumes budget only ever shrinks. DoRA also needs an extra stabilizer against the gradient shocks that pruning causes.

Third, Competition for synaptic building blocks shapes synaptic plasticity proves that competition for a conserved pool produces two things without any added normalization step. One is multiplicative scaling, meaning each change is proportional to the synapse's current strength ($\Delta w_i \propto w_i$). The other is transient depression of neighbouring synapses.

The new element is not "budgeted rank" as such. It is the combination of three properties. The budget is conserved, so it cannot be spent twice. Released budget is returned rather than destroyed. The components are parameterized so that factor imbalance cannot arise and dormant components stay reactivatable. No paper in the file combines these. Energy-Structured Low-Rank Adaptation for Continual Learning reallocates low-energy ranks, but it does so offline, with a decomposition of feature drift computed after each task, and it needs task boundaries. PLAN: Proactive Low-Rank Allocation for Continual Learning reserves orthogonal subspaces for future tasks. That also requires boundaries and forbids reusing a subspace. Our mechanism runs online and needs no task labels.

Research Program

Each adapted matrix receives $\Delta W_\ell = \sum_i \lambda_{\ell,i}\, p_{\ell,i} q_{\ell,i}^\top$. Here $p_{\ell,i}$ and $q_{\ell,i}$ are unit-norm directions for component $i$ in layer $\ell$, and $\lambda_{\ell,i} \ge 0$ is its magnitude. Because the directions are always renormalized, all scale lives in $\lambda$, and the Gram mismatch of Oh and Kim is zero by construction. This subsumes their balance penalty rather than competing with it.

Magnitudes obey a conservation law with a free pool $F_\ell$:

$$\sum_i \lambda_{\ell,i} + F_\ell = M_\ell, \qquad \lambda_{\ell,i} \leftarrow \lambda_{\ell,i} + \eta\,\big(a_{\ell,i}\, F_\ell - \beta\, \lambda_{\ell,i}\big).$$

The symbols are defined as follows. $M_\ell$ is the layer's total budget. $a_{\ell,i} = \mathrm{relu}(-\bar g_{\ell,i})$ is the component's "demand," computed from a running average $\bar g_{\ell,i}$ of the loss gradient with respect to $\lambda_{\ell,i}$. $\beta$ is a constant release rate, and $\eta$ is a step size. The update follows the binding and unbinding kinetics of Triesch et al. Binding scales with the free pool, so strong growth in one component automatically slows growth in the others. Release is proportional to current magnitude, so disuse returns budget to the pool.

Across layers, the budgets $M_\ell$ are exchanged through a second, slower pool of the same form. The demand for this pool is layer-level: unmet demand summed over the layer's components. This lets capacity migrate toward whichever layers the current data stresses.

Components with $\lambda \approx 0$ would normally receive no gradient on their directions. We therefore compute their direction gradients as if $\lambda = \varepsilon$, a small floor, which lets them keep tracking useful directions while contributing nothing to the output. These are the analogue of silent synapses, which have receptor slots but no receptors.

The analyses follow the mechanism's predictions. We will track four quantities: how budget moves between layers over a stream, how often components are reactivated, the Gram mismatch in a non-normalized variant, and the new-task learning speed of the adapter relative to a freshly initialized one. We deliberately leave out a neuromodulatory "surprise" signal of the kind in Surprise-based learning: neuromodulation by surprise in multi-factor learning rules. A surprise signal could modulate $\beta$, but adding it before the conservation mechanism is shown to work would confound the result.

Baselines

The naive reference is fixed-rank LoRA from LoRA: Low-Rank Adaptation of Large Language Models. It is trained sequentially over the stream with the same parameter count as the largest configuration of our adapter. It shows what happens with no allocation and no plasticity treatment.

The strongest single-task competitor is AdaLoRA, run with its published cubic schedule and orthogonality penalty. On GLUE RTE it reports 87.36% accuracy at 0.32M parameters (Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning). We also run Mao et al.'s DoRA, which reports SQuAD v2.0 F1 of 83.53 at 0.34M parameters against 82.34 for LoRA (DoRA). These methods are the most direct competitors because they solve the same allocation problem.

For streaming, the strongest plasticity baseline is fixed-rank LoRA with Oh and Kim's balance regularization. The strongest boundary-aware continual methods are PLAN and E²-LoRA. On ImageNet-R with 20 tasks, PLAN reports 71.06% average accuracy and 77.93% average anytime accuracy; the same paper reports InfLoRA at 69.83% and 76.83%. E²-LoRA reports 82.77% last accuracy on 10 tasks, against 82.76% for joint training.

PLAN and E²-LoRA are given task boundaries; our method is not. Our method therefore counts as beating them if it matches their accuracy without boundaries, not only if it exceeds them.

Fairness rules apply to every run. All methods use the same backbone (DeBERTa-v3-base for GLUE and SQuAD, ViT-B/16 for vision), the same trainable-parameter ceiling, and the same number of training steps. Each method receives the same hyperparameter search budget of 20 trials, chosen on held-out validation streams. Wall-clock overhead is reported against AdaLoRA's stated 11–16% overhead over LoRA.

Three ablations isolate the key ingredients. The first removes the free pool by forcing $F_\ell = 0$, which turns the method into a pure zero-sum reallocation. The second sets $\varepsilon = 0$, so dormant components get no direction learning. The third removes the cross-layer pool, fixing each layer's budget $M_\ell$.

Why not just…?

Why not just add balance regularization to fixed-rank LoRA? It addresses factor drift, which our parameterization also removes, but it cannot move capacity between layers. Oh and Kim also report incomplete recovery at low rank (Factor Imbalance and Plasticity Loss). If balance-regularized LoRA matches us on the domain-streaming benchmark at equal parameters, then reallocation adds nothing in that regime, and we will report it that way.

Why not just re-run AdaLoRA's importance scoring periodically without permanent pruning? This "reversible AdaLoRA" is a real threat, because it shares our reversibility. We implement it directly: the singular-value mask is recomputed every $k$ steps from the same sensitivity score, and $k$ is tuned. What it lacks is conservation. Nothing stops scores from rising everywhere at once, and nothing gives released budget a defined destination. If it matches us, the contribution reduces to "do not prune permanently," a smaller but still useful finding.

Why not just project the magnitudes onto a fixed total budget each step? Projecting the magnitudes onto a simplex is the minimal version of conservation. It could plausibly match the full binding kinetics, so we treat it as a cheaper implementation of the same idea rather than as a rival. The ablation that separates the two is the free pool. Simplex projection always spends the whole budget, while our dynamics can leave budget unspent when demand is low and hold it in reserve for a shift. If projection performs equally well, we will recommend it and credit conservation plus silent-direction learning, not the kinetics.

Proposed Experiments

The first test is a single-task sanity check. On GLUE and SQuAD v2.0, at AdaLoRA's parameter budgets, the method should match AdaLoRA and DoRA within noise over five seeds. The goal is to show that conservation costs nothing when the data is stationary. Where layer budgets end up is compared with AdaLoRA's reported concentration of rank in feed-forward and upper layers.

The main test is streaming. The first stream is the CIFAR-10-C domain-streaming protocol of Oh and Kim, which cycles through corruptions such as fog, snow and brightness. The second is ImageNet-R with the 20-task split, presented as a boundary-free stream. Metrics are final accuracy, average anytime accuracy, and a plasticity ratio: accuracy reached on each new segment within a fixed number of steps, divided by that of a freshly initialized adapter. The file reports no exact CIFAR-10-C numbers for Oh and Kim, so we rerun their method rather than quoting it.

A recurring-domain variant of CIFAR-10-C, in which fog returns after other corruptions, is the decisive check. If reversibility matters, previously dormant components should reactivate quickly when fog returns. Our method should then show faster recovery than both AdaLoRA and reversible AdaLoRA, and the reactivation counts should rise at the moment of return. If recovery speed is the same with $\varepsilon = 0$, the silent-synapse ingredient is unnecessary. If the no-free-pool ablation matches the full method, the free pool is unnecessary.

A final diagnostic reruns the method with unnormalized factors to check Gap 3 directly. It tests whether reallocating rank itself drives Gram mismatch in dormant and reactivated components, a question none of the cited dynamic-rank methods have measured.