Upstream Adapter Libraries as Priors for Low-Rank Fine-Tuning Initialization

Low-rank adaptation (LoRA) fine-tunes a large pretrained model by learning a small add-on to each weight matrix: the product of two thin matrices, so that only a tiny fraction of parameters are trained. Where those thin matrices start turns out to matter a great deal. The default start is random, and a recent line of work shows that a better starting point improves accuracy. One option is to compute a single gradient of the full model on a batch of target data and initialize the adapter along that gradient's strongest directions. When target data is scarce, though, that gradient is noisy. Meanwhile, practitioners already hold large libraries of adapters trained on hundreds of earlier tasks, and these are almost never used as starting points for new fine-tuning runs. They are either discarded or frozen and mixed together.

This note proposes treating such a library as a prior over which directions a new adapter should start in. The method computes the noisy target gradient, as the gradient-based methods do, and then shrinks it toward the subspace spanned by the upstream adapters. The amount of shrinkage is set by how much two halves of the target batch disagree about the gradient. The claim is that this matters most where a direction estimate from target data alone breaks down, namely with tens to a few hundred target examples. The claim also makes a falsifiable prediction: as target data grows, the library's contribution should fade to zero.

Core Claim and Positioning

The closest prior work falls into two camps that never meet. The first camp initializes fresh adapters from the target data or from the base model. LoRA-One takes the top singular directions of one full-model gradient step and proves that adapters drift toward this subspace anyway. LoRA-DA adds a Fisher-information correction and explicitly models a variance term from small batches. EVA and CorDA use target activations. PiSSA and MiLoRA use the singular vectors of the frozen weights without looking at any data. These papers supply our method, and LoRA-DA's variance term in particular motivates us: it names the problem we target. None of them, however, draws on knowledge from other tasks.

The second camp reuses trained adapters but never fine-tunes them further. LoraHub mixes about 200 FLAN-task adapters using scalar weights found by gradient-free search. Hochlehnert et al. find that naively copying an adapter between related base models is a surprisingly strong zero-shot baseline, but one that degrades on generative tasks.

Our proposal differs from both camps. Unlike the first, the starting subspace is informed by upstream adapters. Unlike the second, the library is used only to start gradient-based low-rank fine-tuning, and the choice of directions is guided by the target gradient rather than by black-box search over mixing weights.

The concrete estimator works as follows. For a given layer, let $G$ be the target gradient of the full weight matrix. Let $P_U$ and $P_V$ be orthogonal projectors onto the span of the library adapters' output-side matrices $B_i$ and input-side matrices $A_i$, respectively. We form

$$\tilde G_\lambda = (1-\lambda)\,G + \lambda\, P_U\, G\, P_V,$$

and initialize $B$ and $A$ from the top-$r$ singular vectors of $\tilde G_\lambda$, exactly as LoRA-One does with $G$. The shrinkage weight $\lambda \in [0,1]$ is chosen per layer from split-half agreement. We compute $G$ separately on two disjoint halves of the target batch and measure the overlap of their top-$r$ subspaces. Low overlap means the target estimate is noisy, which calls for a larger $\lambda$. The form is standard shrinkage toward a structured target: it trades the variance of $G$ for the bias of restricting to library directions, and the split-half overlap is a cheap proxy for that variance.

A second question rides along with the first. Li et al. show that starting with a nonzero adapter product makes training more robust to the learning rate, but they use random values. Hayou et al. show that which matrix is zero changes the optimization dynamics. So we also test whether to keep the product at zero, as in LoRA-One, or to start from a nonzero, library-informed adapter value obtained by projecting a composed library adapter onto the chosen subspace.

Research Program

We first build the estimator and the split-half rule. We then run a controlled study on a single question: does an upstream library help initialization beyond what the target data already provides, and when?

The primary testbed reuses LoraHub's setup, with FLAN-T5-large, its roughly 200 upstream FLAN-task adapters, and Big-Bench Hard (BBH) as the target suite. This choice has two practical benefits. Computing one full gradient for a model of this size is cheap, and the library and the target tasks are already established.

The main sweep varies the number of target training examples from 16 to the full available amount, and varies library size by randomly subsampling from 200 adapters down to 10. A control replaces the library with random adapters of matched rank and norm. This tests whether any gains come from task knowledge or merely from restricting to some subspace of the right dimension. Diagnostic measurements track the overlap between each initialization's subspace and the subspace of a long-trained reference adapter on the full target data. The diagnostics also record the learned $\lambda$ per layer, to see whether shrinkage concentrates in particular layers.

A secondary testbed checks generality on generative tasks with LLaMA 2-7B. We train our own library of math and code adapters on disjoint source data, then target GSM8K and HumanEval. This makes contact with LoRA-One's reported numbers and with Hochlehnert et al.'s warning that generative tasks transfer poorly.

Baselines

Plain LoRA. The naive reference is standard LoRA, with a random input-side matrix and a zero output-side matrix, tuned over the same learning-rate grid as every other method.

LoRA-One. The strongest single-task competitor is LoRA-One, run from its codebase. Its gradient is computed on exactly the same target examples our method sees, and it receives the same number of fine-tuning steps and the same tuning budget. It is the right competitor because our method reduces to it at $\lambda=0$, so any gain is attributable to the library. On LLaMA 2-7B, LoRA-One reports 28.66 HumanEval pass@1 against 25.85 for standard LoRA. On GSM8K it reports a gain of about 2.5 points over standard LoRA, but no absolute accuracy figure appears on Lacuna. No reported result was found on Lacuna for LoRA-One on BBH.

LoRA-DA. LoRA-DA addresses variance through Fisher information rather than through a prior. It is run with the same target subset and its K-FAC estimate.

Library reuse. On the library side there are three baselines. The first is LoraHub's composition without further training, for which BBH average accuracy is reported at 34.7%, against 37.3% for few-shot in-context learning; both figures come from LoraHub. The second is LoraHub's composition used as a warm start and then fine-tuned. The third is the single library adapter whose subspace best aligns with the target gradient, copied and then fine-tuned. All three use the same examples and steps.

Ablations. Three ablations isolate the key ingredient. The first fixes $\lambda$ at 1, using pure projection. The second fixes $\lambda$ at 0.5. The third replaces the library with random adapters.

What counts as success. Beating the baselines means a statistically reliable gain in BBH average accuracy over both LoRA-One and fine-tuned LoraHub at 16 to 128 examples, measured over at least five seeds and example draws. It also requires matching LoRA-One, within noise, at full data. A gain that persists at full data would actually be suspicious, and we would examine it as a possible tuning artifact.

Why not just…?

Why not just warm-start from the LoraHub composition and fine-tune? This is the most serious threat. It may well capture most of the benefit, because it also injects library knowledge as a nonzero starting point. Our method differs in two ways. It chooses directions by alignment with the target gradient rather than by a gradient-free search over about 200 scalar weights, which LoraHub itself reports to be high-variance across runs. It also falls back automatically to target-only directions when the library is irrelevant. The fine-tuned LoraHub baseline is run head to head. If it matches our method across the data sweep, we will report that simple warm-started composition suffices, and we will drop the projection machinery.

Why not just average LoRA-One's gradient over more batches? In the large-data regime this would indeed remove the noise problem, which is why we predict no gain there. In the regime we target, there are no more batches to average: with 32 examples, the total target data is the batch. The data sweep tests this directly, giving LoRA-One every available example at each budget.

Why not just use PiSSA or MiLoRA, which need no extra data or library? Both decompose only the frozen base weights, and the two papers disagree about whether the largest or smallest singular directions are the right ones. That disagreement suggests neither choice is tied to the task. We include PiSSA as a baseline. The random-library control separately tests whether "any structured subspace" is enough.

Why not just copy the most similar upstream adapter? Hochlehnert et al. show that copying works well for multiple-choice tasks but loses about 15% COMET on translation. A single adapter is also unlikely to cover a BBH task that resembles none of the FLAN tasks. Copy-then-fine-tune is included as a baseline.

Proposed Experiments and Decisive Checks

On FLAN-T5-large and BBH, we report average accuracy against the number of target examples for all baselines, with five seeds and resampled example sets. At 16 to 64 examples, the 37.3% in-context learning figure serves as a reference ceiling that fine-tuning methods have not cleared in LoraHub's setting.

Four outcomes would be decisive. The first is that our method beats both LoRA-One and fine-tuned LoraHub in the low-data regime while the random-library control does not. This would show the gain comes from task knowledge rather than from the subspace restriction. The second is that the learned $\lambda$ decreases as target data increases, confirming the variance-shrinkage interpretation. The third is that the overlap with the reference adapter's subspace at step zero is higher for our initialization and predicts final accuracy. The fourth concerns the nonzero-start variant: either it improves robustness to the learning rate in the way Li et al. report for random values, or it shows that zero-start, library-informed directions are enough.

On LLaMA 2-7B, we check whether the gain survives on HumanEval and GSM8K, where LoRA-One's 28.66 pass@1 is the number to beat at matched data. We also check whether multi-step generation erodes library benefits, as Hochlehnert et al.'s findings on translation would predict. A null result on the generative tasks paired with a positive result on BBH would itself be informative. It would locate the boundary of when upstream adapters are useful priors.