Orthogonal Rank Revival for Low-Rank Adapters

Low-rank adapters fine-tune a large model by adding a small update $\Delta W = BA$, where $B$ is tall and thin and $A$ is short and wide. How many columns $B$ gets, its rank, is usually fixed in advance for every layer. The two best recent attempts to improve this pull in opposite directions. One keeps the columns of $B$ exactly orthonormal, which stops them from collapsing onto each other, but never changes the rank. The other grows and shrinks rank per layer as training proceeds, but in ordinary unconstrained space, so directions added late can duplicate directions the adapter already has. This note proposes an adapter that does both. Its rank dimensions can be switched off and later switched back on ("revived"), and any dimension that is switched off is kept parked in the part of space the active dimensions do not cover. When it comes back, it cannot duplicate them.

The mechanism is cheap because it uses projections in the surrounding matrix space rather than full Riemannian geometry (the mathematics of optimizing on curved surfaces). Gradients are projected onto the set of orthonormal matrices at the current point, the tangent space, and the updated matrix is snapped back onto that set by a polar projection. This works in the regime where dynamic revival has been memory-hungry, and it removes that cost. In a low-rank adapter the "shadow" copy of the pruned dimensions is only a few extra columns of $B$, not a full dense copy of the network.

Core Claim

We claim that when rank dimensions of an adapter are pruned and revived during training, revival should happen inside the orthogonal complement of the active subspace. Doing so buys three things over two alternatives: reviving in unconstrained space, as LoSA does, and fixing an orthonormal rank from the start, as Stiefel-LoRA does. First, the adapter reaches higher task accuracy at the same average rank. Second, revived directions carry genuinely new information, measured as effective rank equal to active rank. Third, rank allocations across layers are more stable. The part of this that is new is how a revived direction is re-seeded, projected against the active subspace, and how that keeps the whole update orthonormal through discontinuous changes to the mask (the on/off pattern over rank dimensions). Gating rank on and off is not new by itself.

Where This Sits in Prior Work

The method draws on two groups of papers. The first supplies the optimizer. Decentralized Riemannian Gradient Descent on the Stiefel Manifold shows that projecting ambient vectors onto the local tangent space removes the need to transport optimizer history across the curved set of orthonormal matrices. A general framework of Riemannian adaptive optimization methods with a convergence analysis proves that this projection is enough for Adam-style adaptive updates to converge. Finite-Time Analysis of Stochastic Nonconvex Nonsmooth Optimization on the Riemannian Manifolds bounds the distortion projection introduces relative to exact transport. Distributed methods for synchronization of orthogonal matrices over graphs and Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data support taking a Euclidean step and then snapping back onto the constraint set. None of these papers change which coordinates or dimensions are active during optimization. The finite-time analysis also explicitly assumes a continuous objective, whereas our mask is discontinuous.

The second group supplies the revival principle. Dynamic Model Pruning with Feedback (DPF) runs the forward pass through a masked model while accumulating gradients into a dense shadow copy. It also shows that the resulting sparse mask, retrained from scratch, loses accuracy, so the path by which weights leave and re-enter matters. PDP: Parameter-free Differentiable Pruning is All You Need gets revival from soft magnitude masks. Both treat parameters as independent Euclidean scalars. The research file notes this is exactly what breaks under an orthogonality constraint, since zeroing entries of $B$ destroys $B^\top B = I$.

The two closest papers define the gap. Riemannian Optimization for LoRA on the Stiefel Manifold keeps $B$ orthonormal using tangent projection plus QR retraction. It reports that its adapters use all 16 of 16 rank dimensions while AdamW-trained adapters use about 12, but its rank is static and it names adaptive rank allocation as future work. Dynamic Low-Rank Sparse Adaptation for Large Language Models (LoSA) grows per-layer rank using reconstruction error and a mutual-information criterion, and masks the adapter to match a pruned backbone. It does this with unconstrained Adam, so newly expanded dimensions are not kept orthogonal to converged ones.

We also rule out one design deliberately. Elementwise sparsity of $B$ itself is a poor target, because orthonormal matrices are rarely sparse. Proximal Gradient Method for Nonsmooth Optimization over the Stiefel Manifold (ManPG) handles an $\ell_1$ penalty on orthonormal matrices only by solving a Newton-type inner problem at every step, which is too costly for adapters. We therefore prune and revive at the level of whole rank dimensions (columns of $B$). This is exactly the granularity at which orthogonality and on/off switching are compatible. Manifold Regularized Dynamic Network Pruning is only superficially related: its "manifold" is a manifold of input images, not a constraint on weights.

Research Program

Each adapted layer holds $B \in \mathbb{R}^{d \times r_{\max}}$ with $B^\top B = I$, a coefficient matrix $A \in \mathbb{R}^{r_{\max} \times k}$, and a binary mask $m \in \{0,1\}^{r_{\max}}$ over rank dimensions. The forward update is

$$\Delta W = P \odot \big(B\,\mathrm{diag}(m)\,A\big),$$

where $P$ is an optional fixed elementwise mask matching a pruned backbone, as in LoSA's zero-latency merge. Applying $P$ to the product rather than to $B$ leaves orthogonality untouched.

Active columns of $B$ are trained by projected Adam. The Euclidean gradient and the moment buffers are projected onto the tangent space at $B$, and after each step $B$ is snapped back by the polar projection $B \leftarrow B(B^\top B)^{-1/2}$. That projection costs one $r_{\max} \times r_{\max}$ eigendecomposition, which is negligible at adapter sizes.

Inactive columns are the shadow. Their rows of $A$ are frozen, and their columns of $B$ are kept orthonormal to the active ones. Every $T$ steps, the method takes the running gradient with respect to the unmasked update, $\bar G$ (the dense gradient DPF-style feedback would accumulate). It projects $\bar G$ onto the orthogonal complement of the active columns, $(I - B_{\mathrm{act}}B_{\mathrm{act}}^\top)\bar G$. It then re-seeds the parked columns with the leading left singular vectors of that residual, snapped jointly with the active columns. A mask update follows. It scores each dimension by the size of its contribution, the norm of the corresponding row of $A$ for active dimensions and the projected residual gradient for parked ones. Given a per-layer or global rank budget following a schedule, it switches off the weakest active dimensions and revives the strongest parked ones.

A revived dimension therefore enters pointing at the largest loss-reducing direction that the adapter does not already span. This is the property we expect to prevent the redundancy LoSA's growth suffers from. It also makes every mask flip an orthogonal decomposition of the update, which is what the analysis will use.

The analysis aims for a modest result: a convergence statement for projected Adam with periodic mask flips. Because parked columns are orthogonal to active ones, a flip perturbs $\Delta W$ by exactly the energy in the gated components. That lets DPF's error-feedback argument be combined with the curvature-distortion bound for tangent projection from the finite-time analysis paper. Gap 2 of the research file identifies this combination as missing. We would count a bound on stationarity that degrades gracefully with flip frequency as a success. We do not promise matching rates.

Baselines

The naive reference points are two standard-LoRA runs trained with AdamW. One uses a fixed rank equal to our average rank budget. The other uses $r_{\max}$ and receives no rank reduction; it spends more parameters and marks the ceiling. In the sparse-backbone setting we also include the pruned model with no adaptation.

The strongest fixed-rank competitor is Stiefel-LoRA, reimplemented with the same projected Adam and polar snapping we use, rather than its QR retraction, so that only rank dynamics differ. It is run at the same average rank as our method. It is the right competitor because it already captures the anti-collapse benefit of orthogonality, and our claim is precisely that dynamic rank adds something on top.

The strongest dynamic-rank competitor is LoSA, run with its own rank-allocation criteria, the same backbone sparsity and cubic schedule, the same Alpaca-GPT4 data, and the same final rank budget. On LLaMA-2-7B at 70% sparsity, LoSA reports that it "reduced perplexity by 68.73 and improved zero-shot accuracy by 16.32% compared to the pruned baseline" (Dynamic Low-Rank Sparse Adaptation for Large Language Models). We will reproduce LoSA in our pipeline and compare against the reproduced number rather than the reported one. For Stiefel-LoRA's commonsense and math suites, the file contains only effective-rank figures; no reported task-accuracy result was found on Lacuna for Stiefel-LoRA on those benchmarks, so we will produce our own numbers.

Two ablations isolate the key ingredient. The first keeps the orthonormal active set and the mask schedule but re-seeds revived columns with the raw gradient direction, orthogonalized only by the final snap, without projecting against the active subspace first. The second disables revival: dimensions can be switched off but never back on. All methods get the same number of training tokens, the same parameter budget measured as average active rank times layer width, and an equal-size learning-rate and schedule sweep of eight configurations each on a held-out split. Results are averaged over three seeds.

Beating a baseline means a mean improvement larger than two seed standard deviations. This must hold on the average of the zero-shot suite and on WikiText-2 perplexity. It must also come at no more than 10% wall-clock overhead over Stiefel-LoRA.

Why not just…?

Why not just train Stiefel-LoRA at the maximum rank and truncate by singular value decomposition at the end? This is the most serious threat. If the useful directions are all findable at full rank, pruning after the fact loses little. Against it, DPF shows that masks chosen after training underperform masks that evolve during training. However, that evidence comes from elementwise pruning of whole networks, not from adapters, so it may not transfer. We will run this baseline explicitly, at matched final rank and matched training compute. If it comes within noise of our method, we will conclude that during-training revival is unnecessary for adapters and report that result.

Why not just use LoSA and orthogonalize each newly grown dimension once, at the moment it is added? This is cheaper than keeping a parked orthonormal shadow. It is essentially our first ablation with a one-shot projection. The difference we predict is that a single projection at birth does not stop later drift back toward the active subspace under unconstrained Adam. Stiefel-LoRA's observation that unconstrained adapters use about 12 of 16 dimensions suggests such drift is real. We will track the effective rank of revived dimensions over time in both variants. If the one-shot version holds full effective rank, the continuous constraint is not justified.

Why not just add a soft penalty on $\|B^\top B - I\|$ to LoSA? A penalty does not guarantee orthogonality at the moment a mask flips, which is exactly when redundancy enters. Orthogonal Weight Normalization also reports that penalty- and retraction-style alternatives stall or diverge at learning rates where projection-style normalization stays stable. That was measured on convolutional networks, not adapters. We will include LoSA plus a penalty, with the penalty weight tuned inside the same eight-configuration budget, and report stability across learning rates as well as accuracy.

Proposed Experiments

The first setting reproduces LoSA's regime: LLaMA-2-7B and LLaMA-3-8B pruned to 50% and 70% sparsity, adapted on 10K Alpaca-GPT4 examples. We evaluate WikiText-2 perplexity and the seven zero-shot tasks: HellaSwag, Winogrande, BoolQ, OpenBookQA, PIQA, ARC-Easy and ARC-Challenge. The second setting reproduces Stiefel-LoRA's dense fine-tuning regime on LLaMA-3.2-1B and 3B, with commonsense (BoolQ, PIQA, SIQA, HellaSwag, ARC, OBQA) and math (GSM8K, MATH) evaluations, at average ranks of 8 and 16 with $r_{\max}$ set to twice the average. Beyond accuracy, we measure three things: effective rank per layer, computed as the exponentiated entropy of the singular values of $\Delta W$; the principal-angle overlap between each revived direction and the active subspace at the time of revival and 500 steps later; and how often each dimension switches.

Three results would decide whether the idea holds up. The idea survives only if our method beats both SVD truncation from full-rank Stiefel-LoRA and LoSA-with-one-shot-orthogonalization at matched budget, since those are the two cheap alternatives that could make it redundant. The mechanism is supported only if revived directions in our method keep near-zero overlap with the active subspace, while those in the first ablation drift. That drift must also coincide with the accuracy gap. The analysis is practically relevant only if accuracy degrades smoothly, not sharply, as the mask interval $T$ shrinks. That behavior is what the error-feedback-plus-projection bound predicts. If the first check fails, we will report that orthonormal adapters do not benefit from rank revival, which is itself a useful answer to the adaptive-rank question Stiefel-LoRA left open.