When one language model is fine-tuned on many tasks at once with small add-on weight matrices, the tasks that have only a few dozen examples tend to get worse, not better. This proposal explains one likely cause and tests a fix. The add-on is LoRA, low-rank adaptation, which learns a weight change $\Delta W = BA$. Here $A$ compresses each input into a small space and $B$ expands it back out. Recent multi-task variants give each task its own $B$ but keep one shared $A$. Our hypothesis is that this shared $A$ is shaped almost entirely by the high-resource tasks. It ends up pointing at the input directions those tasks care about. A low-resource task whose inputs live elsewhere then passes through a compressor that cannot "see" its signal, and no task-specific $B$ can recover information that $A$ has already discarded.
The fix starts by measuring the problem directly. For each task and layer, we compute what fraction of the task's input activation energy falls inside the subspace spanned by the shared $A$. We call this the task's coverage. Then we give each task a small private compressor whose size is set by how much coverage it lacks. Coverage is a single averaged ratio, not a full spectrum, so it can be estimated from the handful of examples a low-resource task actually has. If the idea is right, it turns "negative transfer for scarce tasks" from something observed after training into something predicted before it. It also spends extra parameters only where the shared subspace fails.
Core Claim
Under strong data imbalance across tasks, the harm a low-resource task suffers from joint low-rank fine-tuning is governed largely by its coverage deficit in the shared down-projection. For task $t$ at a given layer, let $\Sigma_t$ be the second-moment matrix of that task's inputs to the adapted weight, and let $P_A$ be the orthogonal projector onto the row space of the shared $A$. Coverage is
$$c_t = \frac{\operatorname{tr}(P_A \Sigma_t)}{\operatorname{tr}(\Sigma_t)},$$
the fraction of the task's input energy that the shared compressor retains. Because both traces are averages of per-example squared norms, $c_t$ is estimated like a sample mean. Its error shrinks with the number of examples times tokens per example, rather than requiring more examples than dimensions. That is exactly the regime in which ThanoRA's spectral-entropy rank allocation is acknowledged to become noisy.
The method keeps a trainable shared $A$, so that scarce tasks still borrow the input structure learned from abundant tasks. Each task additionally gets a private down-projection $A_t$ of rank $r_t$:
$$\Delta W_t x = B_t\left(\Lambda_t A x + A_t x\right).$$
Here $B_t$ is the task's up-projection and $\Lambda_t$ is a task-specific diagonal scaling, both as in MTL-LoRA. $A_t$ is initialized from the top directions of the task's inputs that lie outside the shared subspace. The ranks $r_t$ are drawn from a fixed total budget in proportion to $1 - c_t$, and $c_t$ is recomputed periodically as $A$ moves.
Two things would be new. The first is the diagnostic: coverage as a cheap, few-shot-estimable predictor of which tasks will be hurt. The second is the allocation rule, which targets the shared-$A$ coupling that the mtLoRA authors explicitly leave open as a limitation.
Relation to Prior Work
The existing multi-task LoRA designs share the down-projection and differ in how they specialize the rest. MTL-LoRA inserts a task-specific diagonal $\Lambda_t$ and a mixture of up-projections. mtLoRA orthogonalizes only the low-singular-value parts of task updates and routes within subspaces. LoRI freezes $A$ as a random Gaussian matrix and sparsifies task $B$s. These papers supply our base architecture, and they motivate the gap: none of them can restore input directions that their single $A$ ignores. A random $A$ captures little energy from strongly anisotropic activations. A learned $A$ captures mostly the dominant tasks' directions.
ThanoRA is the closest relative because it also allocates rank per task and layer. However, it estimates full activation spectra from calibration batches and enforces pairwise orthogonality across task factors. Its allocation degrades at very small sample sizes, and its penalty conflicts with mtLoRA's finding that 89.4% of shared knowledge sits in the top 10% of singular components. Our rule asks a narrower question: what the shared subspace misses. That question is answerable with less data, and it imposes no orthogonality on the shared part.
The data-efficiency papers constrain the design. MeTA-LoRA shows that 100 examples per task with a first-order meta-learned shared adapter can beat full-data HydraLoRA, but it uses one uniform rank and one shared factorization for every task. CA-MTL shows that entropy-based task sampling rescues small tasks, and it is the main competing explanation for our results. RaSA shows that pooling rank across layers acts as a data multiplier, which supports the premise that how rank is spent matters most when data is scarce.
The theory in Improving Few-Shot Learning Through Multi-task Representation Learning Theory requires task predictors to span the shared representation. Coverage is the input-side analogue of that diversity condition. A task with low coverage violates it by construction.
Research Program
The work has three stages.
Stage 1: test the diagnostic before building the method. We train standard shared-$A$ multi-task LoRA and MTL-LoRA on mixtures with deliberate imbalance. We measure each scarce task's coverage at every adapted layer and correlate it with that task's negative transfer, defined as joint-training score minus single-task LoRA score. We also check estimation stability by recomputing coverage from 5, 16, 50, and full-size subsets and comparing the results.
Stage 2: build coverage-guided private down-projections. This involves three parts: initialization from the out-of-subspace directions of each task's inputs, budgeted rank allocation proportional to coverage deficit, and periodic re-estimation. Total trainable parameters are matched to the baselines.
Stage 3: analyze when the method stops helping. We expect it to stop mattering when tasks share input structure, which should show up as high coverage everywhere. We also test whether coverage deficits concentrate in particular layers or in attention versus feed-forward weights. This connects to mtLoRA's observation that softmax attention amplifies conflict.
Baselines
The naive reference is single-task LoRA trained separately on each scarce task with the same per-task data. This is the bar any multi-task method must clear to claim positive transfer. Next to it sits standard multi-task LoRA with proportional mixing, which is the naive joint approach.
The strongest architectural competitor is MTL-LoRA. It shares our backbone design and reports an 8-task commonsense average of 82.1, against 77.6 for LoRA and 80.5 for DoRA. It is the obvious method to beat because our proposal adds exactly one ingredient to it. No reported result found on Lacuna for MTL-LoRA with fewer than 100 examples per task, so we rerun it ourselves.
The strongest data-efficient competitor is MeTA-LoRA. It reports a Big-Bench Hard score of 46.32 on LLaMA-2-13B using 100 examples per task from the 43-task Flanv2 subset, versus 44.7 for full-data HydraLoRA. We rerun it under our splits.
We also include LoRI, with its frozen random $A$, and ThanoRA, the closest rank-allocation method. A sampling baseline completes the competitor set: MTL-LoRA trained with CA-MTL-style entropy sampling and with temperature sampling.
Two ablations isolate the key ingredient. The first gives every task an equal-rank private $A_t$ under the same total budget, which tests whether coverage-based allocation matters or private capacity alone does. The second allocates the same budget at random or by task size, which tests whether coverage is the right signal.
Fairness rules apply to all methods. They use the same base model, the same per-task data subsets drawn with three seeds, the same total trainable parameter count, and the same number of optimizer steps. Each gets an equal hyperparameter search of twelve configurations per method, with selection on a held-out split of the scarce tasks' own data, never on test.
Beating a baseline means two things. First, a statistically reliable gain on the average over scarce tasks, using paired bootstrap over seeds and examples. Second, no loss larger than half a point on the high-resource tasks.
Why not just…?
Why not just oversample the small tasks, as CA-MTL does? This is the real threat. Entropy sampling in CA-MTL cut required data by 34% and lifted small tasks. Oversampling changes how much the scarce task pulls on the shared $A$. With 16 to 100 examples, however, that pull comes from the same few inputs repeated, which risks memorization rather than a better subspace. Our experiments include oversampled MTL-LoRA as a baseline and also stack sampling on top of our method. If oversampling alone closes the coverage deficit and matches our scarce-task scores, we will report that the shared-$A$ problem is a scheduling problem, not an architectural one. In that case, coverage survives only as a diagnostic.
Why not just give each task its own full LoRA, or freeze $A$ as LoRI does? Separate adapters discard what scarce tasks can borrow from abundant ones. CA-MTL's 24-task model beats single-task fine-tuning by 0.7–1.0%, and MeTA-LoRA's few-shot gains depend on sharing. A frozen random $A$, as in LoRI, keeps every task on an equal footing but captures only a small fraction of energy from anisotropic activations. We will measure that coverage directly. Both options are baselines, and the coverage measurements should explain where each falls short.
Why not just use ThanoRA's spectral rank allocation? It answers a different question, namely how complex each task is at each layer, and it needs reliable spectra. Its authors note that these become noisy with very few samples. We compare both allocation signals at 5, 16, 50, and 100 examples per task under matched budgets. If ThanoRA's allocation matches ours even at 16 examples, our advantage reduces to cheaper estimation.
Proposed Experiments
The base model is a 7B LLaMA-family model, consistent with the MeTA-LoRA setting. There are three testbeds.
The first uses GLUE as defined in GLUE. It has natural imbalance: MNLI with about 392k examples alongside CoLA with 8.5k, RTE, and WNLI with 634. We further subsample CoLA, RTE, MRPC, and STS-B to 16, 50, and 100 examples. Metrics are each task's official metric, including Matthews correlation for CoLA.
The second uses the eight-task commonsense suite. Four tasks are kept at full size and four are reduced to 16–100 examples. We report per-task accuracy and the scarce-task average against MTL-LoRA's reported 82.1 full-data average as context.
The third uses the 43-task Flanv2 mixture at 100 examples per task, evaluated on Big-Bench Hard, where MeTA-LoRA's reported 46.32 is the reference. This setting tests uniformly scarce data rather than imbalanced data. We expect smaller gains here, and saying so in advance is part of the test.
Three decisive checks determine whether the idea holds up. First, in Stage 1, coverage deficit must correlate with negative transfer across scarce tasks and layers. If the rank correlation is weak (below about 0.4), the mechanism is wrong and we stop before building the method. Second, coverage estimated from 16 examples must agree closely with coverage estimated from the full task data. Otherwise the few-shot argument fails. Third, at a matched parameter budget, coverage-guided allocation must beat equal-rank private projections and oversampled MTL-LoRA on the scarce-task average, while keeping the high-resource tasks within half a point. If only the private capacity matters, or only the sampling, the result is still informative, but the core claim is refuted and we will report it that way.
References
- MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
- Scalable Multi-Task Low-Rank Model Adaptation
- LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
- ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation
- MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Models
- Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data
- RaSA: Rank-Sharing Low-Rank Adaptation
- Improving Few-Shot Learning Through Multi-task Representation Learning Theory
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding