Low-rank adaptation (LoRA) fine-tunes a large pre-trained network by adding a small trainable correction to each frozen weight matrix. The correction is constrained to be "thin," meaning it can only change the layer's output along $r$ directions. Practitioners pick $r$ by habit (4, 8, 16) or by a costly sweep, and theory gives no usable guidance. The known optimization guarantee needs $r$ of order $\sqrt{KN}$, where $K$ is the number of classes and $N$ the number of samples. That is far above what works in practice. This proposal argues that, for classification, the rank an adapter needs is set by how far the pre-trained features are from the geometry that fine-tuning drives them toward, and that this distance can be measured before training begins. The relevant geometry is neural collapse: late in training, each class's features shrink onto a single point, and the class points spread into a maximally separated, symmetric configuration called a simplex. If the pre-trained backbone already separates most classes well, the correction needed to reach that configuration has only a few significant directions. An adapter of that rank should be enough. Adding rank beyond it should buy little.
The practical result would be a rank chosen from one forward pass over the training set, with no sweep. The scientific result would be a test between three competing accounts of what limits a low-rank adapter. The first is the number of classes, which the symmetric-simplex picture suggests. The second is roughly $\sqrt{K}$, which the weight-decay low-rank-bias theory suggests. The third is a data- and backbone-dependent "geometric deficit," which is what we propose. A second part of the work asks whether a shortfall below that rank comes from too little capacity or from the optimizer starting in a bad direction. We separate the two by initializing the adapter along the deficit directions.
Core Claim
Let $M_0 \in \mathbb{R}^{d \times K}$ hold the centered, normalized class means of the frozen backbone's penultimate features on the downstream training set. Let $E$ be a simplex arrangement of $K$ points at the same scale. We define the correction matrix as
$$\Delta = Q^\star E - M_0, \qquad Q^\star = \arg\min_{Q^\top Q = I} \lVert QE - M_0 \rVert_F .$$
Here $Q^\star$ is the rotation that brings the ideal simplex closest to where the classes already sit. The rotation step matters because the simplex is defined only up to rotation, so we should not charge the adapter for rotating it. We then extend $\Delta$ with the leading directions of within-class spread that overlap the between-class directions. These are the directions along which classes are currently confused. The geometric deficit rank $r^\star(\varepsilon)$ is the number of singular values of this combined matrix that carry more than a $1-\varepsilon$ share of its energy.
The claim has two parts. First, downstream accuracy and downstream within-class spread, plotted against LoRA rank, will show a knee near $r^\star$, not near $K-1$ or $\sqrt{K}$. Second, the knee will move with the backbone and dataset in the way $r^\star$ predicts. For example, $r^\star$ should be lower for a CLIP backbone on Pets than for a ResNet on FGVC-Aircraft, even when $K$ is similar.
Where This Sits
Three papers motivate the proposal.
Understanding and Improving Transfer Learning of Deep Models via Neural Collapse shows that within-class spread on downstream data correlates near-linearly with transfer accuracy across ResNet, ViT and CLIP backbones. The same measure computed on the source data does not. That finding is why we measure geometry on downstream data. Its proposed fix, Skip Connection Layer fine-tuning, updates one intermediate layer at full rank and adds a skip connection to the head. It never asks how a rank constraint limits that collapse.
The Expressive Power of Low-Rank Adaptation proves a depth-for-rank trade-off: rank needed ≈ width × target depth / frozen depth. The proof uses a linearization in which ReLUs are forced to stay active. It bounds the existence of weights matching an arbitrary target network. That target is far larger than what classification requires, which is why its rank bounds are loose here.
LoRA Training in the NTK Regime has No Spurious Local Minima gives the $r(r+1)/2 > KN$ condition. It operates in the lazy regime, where features do not move, and it attributes the gap between that condition and practical ranks to an uncharacterized "low intrinsic dimension." Our deficit rank is a concrete candidate for that intrinsic dimension, and it applies in a regime where features do move.
Two theory papers constrain the hypotheses rather than supply methods. Neural collapse vs. low-rank bias proves that under weight decay, intermediate layers prefer rank near $\sqrt{K}$ rather than full collapse to rank $K$. The Persistence of Neural Collapse Despite Low-Rank Bias explains why full collapse still appears in practice: in wide layers, the set of collapsed solutions is much larger, so gradient descent finds it more easily. Both analyses treat features as free variables trained from scratch. A rank-$r$ adapter on a frozen wide model is a different situation. It may shrink that large solution set. That gives the $\sqrt{K}$ hypothesis a real chance, and we test it directly instead of assuming our account wins. Neural Rank Collapse bounds a weight matrix's distance from rank $K$ by the previous layer's within-class spread. This suggests deficit directions should be measured where the adapter's output lands, which we do per layer in the analysis.
The method for the second part comes from When Is Rank-1 Enough?. That paper shows rank-1 LoRA fails across seeds because a random start is nearly orthogonal to the needed shift. Its fix, Gap-Init, initializes one factor to the gap between image and text features. We generalize this from one cross-modal vector to the $r^\star$-dimensional multi-class correction subspace. We also use it differently: as a diagnostic that separates capacity limits from starting-direction limits, not as a performance trick.
LoRI freezes one LoRA factor as a random projection and still performs well. It is the natural counter-hypothesis that direction does not matter.
Research Program
We first compute $r^\star(\varepsilon)$ for every backbone–dataset pair from one forward pass, including the within-class confusion directions. We also compute per-layer versions at the outputs of the attention and MLP projections that LoRA modifies. Next, we train LoRA at ranks $\{1, 2, 4, 8, 16, 32, 64\}$, either on all attention projections or on a single block, with several seeds per rank. At convergence we record test accuracy, downstream within-class spread, distance of the class means from a simplex, and the effective rank of each learned correction $BA$.
We then fit a knee to each accuracy-versus-rank curve and regress the knee location on three predictors: $K-1$, $\sqrt{K}$, and $r^\star$. If our account is right, $r^\star$ should explain most of the variation in knee location, and the other two predictors should add little. A secondary check tests the depth trade-off. If correcting all layers spreads the deficit, the knee for all-layer LoRA should sit below the knee for single-block LoRA by a factor that shrinks with the number of adapted layers.
Finally, we run the causal intervention. For ranks below the observed knee, we initialize the output factor $B$ of the final adapted blocks to the top deficit directions, the analog of Gap-Init. If performance below the knee recovers, the knee was an optimization artifact of random starts. If it does not recover, the knee reflects true capacity.
Baselines
The naive reference is linear probing: freeze the backbone and train only the classifier. It sets the floor and shows how much of the deficit the head alone can absorb. It runs with the same data, the same number of epochs, and its own learning-rate search. Our account is useful only if the knee in LoRA rank sits clearly above this floor.
The strongest existing competitor is standard LoRA with a full rank sweep (LoRA). It is the strongest competitor because, given enough compute, a sweep always finds the empirically best rank. It gets the full sweep with a matched per-rank tuning budget. Our selector beats it if the predicted rank lands within the sweep's top-performing ranks (within seed noise) on most pairs, at the cost of one sweep point instead of seven. On the reference number: no reported result found on Lacuna for LoRA accuracy on CIFAR-100, FGVC-Aircraft, DTD or Pets. The closest reported comparison in the file is from a different task. When Is Rank-1 Enough? reports rank-1 Gap-Init at 140.59 CIDEr against standard rank-8 LoRA at 138.49 CIDEr on COCO captioning, with seed standard deviation dropping from 7.10 to 1.44. We will reproduce our own LoRA numbers and not rely on that cross-task figure.
Skip Connection Layer fine-tuning (Understanding and Improving Transfer Learning via Neural Collapse) is the strongest method built on neural collapse. It updates about 8% of parameters (2M vs 23M on ResNet-50), and the file gives no per-dataset accuracies for it. We reimplement it on the same backbones and splits. We compare at matched trainable-parameter counts and also at its native size.
Full fine-tuning with weight decay is the upper reference. It runs under the same data and epochs.
LoRI-style frozen random $A$ (LoRI) at each rank is the direction-agnostic control.
The ablation that isolates our key ingredient is deficit-directed initialization versus random initialization at the same rank. A shuffled-subspace control initializes with the deficit subspace computed after permuting class labels. That control keeps the spectrum but destroys the meaning of the directions.
Why not just…?
Why not just sweep the rank? A sweep is the honest default. If accuracy is flat across ranks 1 to 64 on every pair, then rank selection is moot, and we will report that as the finding. The case for a predictor rests on regimes where the curve is not flat. Large $K$ (CIFAR-100, Tiny ImageNet) and the 10-shot setting are where the transfer–collapse paper found full fine-tuning overfits. In those regimes, too high a rank should hurt as well as cost compute. The sweep experiment itself settles this question.
Why not just use $\sqrt{K}$ or $K-1$ as the rule? These are cheaper rules with some theoretical backing, from Neural collapse vs. low-rank bias and the simplex picture respectively. They are real threats. They also ignore the backbone. For example, they predict the same rank for CLIP and ResNet on Pets. The regression of knee location on the three predictors decides between them. If $\sqrt{K}$ explains the knee as well as $r^\star$, we will recommend $\sqrt{K}$.
Why not just tune the head or use the Skip Connection Layer method? The Expressive Power of Low-Rank Adaptation proves that tuning only the final layer is strictly weaker than spreading updates across layers. The transfer–collapse paper finds that linear probing lags behind. The Skip Connection Layer method works, but it changes the network graph and updates a full dense layer. LoRA, by contrast, can be merged back into the frozen weights. Our matched-parameter comparison will show whether the skip connection's advantage persists at equal trainable-parameter budgets.
Why not just freeze a random projection, as LoRI does? If random frozen $A$ matches deficit-directed initialization at every rank below the knee, then direction does not matter in this regime. In that case the second part of the claim fails, though the rank predictor may still hold. Rank-1 evidence from When Is Rank-1 Enough? suggests direction does matter at the extreme. Whether that extends to $1 < r < K$ is exactly what the ablation measures.
Proposed Experiments
The backbones follow the transfer–collapse paper: ResNet-50, a ViT, and CLIP's image encoder. They are evaluated on CIFAR-10, CIFAR-100, FGVC-Aircraft, DTD, Oxford-IIIT Pet and Tiny ImageNet, with full data and 10-shot splits and at least five seeds per configuration. Metrics are test accuracy, downstream within-class spread, distance from a simplex, effective rank of the learned corrections, and seed variance.
Three checks decide the outcome. First, $r^\star$ must predict knee location better than $K-1$ and $\sqrt{K}$ across the backbone–dataset grid, measured by held-out regression error. Second, the predicted rank must perform within seed noise of the best swept rank on most pairs. Third, deficit-directed initialization must close a significant part of the below-knee gap where both random initialization and the shuffled-subspace control fail; that result would show the gap is about starting direction rather than capacity. A negative result on the third check, with success on the first, would still be informative: it would mean low-rank adapters on classification are limited by capacity, and the deficit measures how much capacity is needed. If the first check fails, the geometric account of adapter rank is wrong for classification, and we will say so.
References
- Understanding and Improving Transfer Learning of Deep Models via Neural Collapse
- The Expressive Power of Low-Rank Adaptation
- LoRA Training in the NTK Regime has No Spurious Local Minima
- Neural collapse vs. low-rank bias: Is deep neural collapse really optimal?
- The Persistence of Neural Collapse Despite Low-Rank Bias
- Provable Emergence of Deep Neural Collapse and Low-Rank Bias in $L^2$-Regularized Nonlinear Networks
- When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning
- LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
- LoRA: Low-Rank Adaptation of Large Language Models