Open libraries now hold hundreds of small add-on weight modules for the same base language model. Each is a LoRA (low-rank adaptation) adapter: a pair of thin matrices trained for one task and added to a frozen model. A user with a new task and a handful of labeled examples has to decide which few of these adapters to load and combine. The automated ways of choosing fail badly. The LoRALib benchmark found that ranking adapters by perplexity, meaning how fluent the model finds the text, systematically prefers generic logic adapters over the math adapter on math problems, and that human hand-picking beats it. Other systems choose randomly before tuning the combination weights. We propose to treat adapter choice as a team-selection problem from the ensemble-pruning literature. Each candidate adapter is scored by which of the target examples it answers correctly. Then, one at a time, we add the adapter that covers the most examples the current team still gets wrong. This objective has diminishing returns, so a simple greedy procedure carries a known approximation guarantee.
In two sentences: the expert-selection failure is a failure of the signal, because fluency is not correctness, and a failure of the search, because adapters are ranked one by one with no regard for redundancy. We replace both with a correctness-coverage objective over a few labeled target examples, optimized greedily. We test whether this closes the gap to manual selection across three ways of combining the chosen adapters.
Core Claim
Given $n$ labeled target examples and a pool $\mathcal{M}$ of adapters on a shared backbone, let $q_i(x)$ be the probability that adapter $i$ assigns to the correct answer of example $x$. For generative tasks this is the likelihood of the reference answer, or the exact-match rate over a few samples. We choose the subset $S$ of at most $k$ adapters that maximizes
$$f(S) = \frac{1}{n}\sum_{x} \max_{i \in S} q_i(x),$$
which is the score an oracle would get if it could hand each example to the best adapter in the team. This function, a soft coverage known as facility location, is monotone and has diminishing returns. Greedy forward selection is therefore within a factor $(1 - 1/e)$ of the best size-$k$ team, the same style of guarantee that OSBORN used for choosing source models. We claim three things. First, $f$ chooses better adapters than perplexity ranking, random pre-filtering, or ranking by individual accuracy. Second, the advantage over individual-accuracy ranking comes specifically from the coverage term, which rewards adapters whose correct answers do not overlap. Third, a small greedy team ($k \le 5$) matches or beats manual selection on LoRALib.
Relation to Prior Work
Two papers motivate this work by exposing the gap. LoRALib built a pool of 680 adapters over 40 tasks and 17 backbones. It showed that perplexity-based selection loses to manual selection and proposed no scalable fix. LoraHub merges adapters with scalar weights tuned by a gradient-free optimizer on about five examples. It pre-filters the pool randomly, and its own loss-based filtering helped only marginally. We keep LoraHub's few-shot setting but change the step before its weight search.
Two lines of ensemble work supply the method. Hierarchical pruning with focal diversity showed that small teams chosen for non-overlapping failures beat full ensembles, for example 4 models at 97.15% versus 10 models at 96.33% on CIFAR-10. SLM-MUX chose small-model teams by "union accuracy", which counts a question as solved if any member solves it, using exhaustive search. Our objective is a soft, greedily optimizable form of union accuracy. Because all members share one backbone, a team costs a few megabytes of adapter weights rather than several full model copies.
Three instance-level methods are neighbours rather than precedents. LoGo runs every adapter on each input and keeps the top-$k$ by activation norm, with no notion of redundancy, and its authors flag its unreliability on unfamiliar inputs. ELREA trains its own adapters on clusters of training gradients and routes test inputs using test-time gradients, at a 2.3x slowdown. LoRAMoE trains its experts jointly with a router, so it cannot draw from an arbitrary library. We do selection offline, once per task, from adapters we did not train.
LoRA-Ensemble shows that adapters on a shared backbone can disagree usefully. That supports our premise that coverage has room to help, although it never prunes its members.
One constraint is direct. The objective is exact only for an aggregator that picks the right member per example. Averaging outputs, and especially merging weights, can break diminishing returns. The direction overview flags this as an open problem. We therefore test the claim under three aggregators rather than assuming it transfers.
Research Program
We will build a selector that scores every pool adapter on the $n$ target examples in one batched offline pass. This costs $|\mathcal{M}| \cdot n$ adapter forward passes, which shared-backbone batching makes cheap. The selector then runs greedy maximization of $f$ up to size $k$.
The chosen team is combined in three ways:
- averaging the members' output probabilities;
- confidence-gated selection, where the member with the highest maximum probability answers, following the cascade logic of Wisdom of Committees;
- LoraHub's weighted merge into a single adapter, which serves at single-adapter cost.
The analysis asks four questions:
- Does $f$ on the selection examples rank candidate teams in the same order as held-out accuracy (Kendall's $\tau$ across sampled teams), compared with summed perplexity and summed individual accuracy?
- On quantitative tasks, does $f$ actually pick math adapters over LogiQA-type adapters?
- How does performance degrade as $n$ falls from 50 to 5, since coverage over five examples is noisy?
- Does the coverage benefit survive weight merging, or only output-level combination?
Throughout, the target task's own adapter is removed from the pool. Otherwise selection would be trivial.
Baselines
The naive reference points are the base model with no adapter and a uniform average of the whole pool. The second tests whether selection matters at all. Adaptive LoRA Merge reports that unselected merging suffers interference, with LoraHub at 28.13 ROUGE-L versus 33.12 for its pruned merge on Japanese Bloomberg news.
The perplexity selector is LoRALib's automated method. We run it with the same $n$ examples and the same $k$. The file reports only that manual selection consistently beat it; no reported number on Lacuna was found for the gap size.
Manual selection is the target to match. We use LoRALib's manual choices where it supplies them. Its strongest reported system score on Llama-3.1-8B-Instruct is 61.87 for LoRAMoE, against 56.96 for Rocket and 51.37 for X-LoRA. We report our methods on the same backbone and average, but LoRAMoE trains a router, so that number is a ceiling for context rather than a like-for-like comparison.
LoraHub is the strongest existing training-free composition method. Its reported average is 34.7% on Big-Bench Hard (BBH, a suite of 27 reasoning tasks) with FLAN-T5-large, against 37.3% for in-context learning (putting the examples in the prompt). We rerun it in two configurations, with its random pre-filter and with our greedy team as the pre-filter. Both get the same five-example set and the same budget of optimizer forward passes. Beating it means a higher BBH average than the rerun LoraHub on matched seeds. A further target is closing a meaningful part of the 2.6-point gap to in-context learning while keeping its roughly 5x shorter inputs (111.6 versus 597.8 tokens).
LoGo serves as the instance-level reference. It reports gains of up to 3.6% over training-based baselines on BBH but no absolute score in the file, so we rerun it on the same pool.
The key ablation is greedy top-$k$ by individual mean $q_i$. This is the same correctness signal with the coverage term removed, and it isolates whether team diversity, not just a better signal, does the work.
Why not just…?
Why not just rank adapters by their few-shot accuracy and take the top few? This is the real threat. It already fixes the perplexity blind spot, and if one adapter dominates a task, coverage adds nothing. The individual-accuracy ablation above is designed to settle it. We expect coverage to help most on mixed tasks such as BBH and on combined math sets, where different adapters solve different subproblems, and to tie on narrow tasks. If it ties everywhere, the contribution reduces to "use correctness, not fluency." We would report that plainly, since that alone would still close a documented gap.
Why not just let LoraHub's weight optimizer, with its sparsity penalty, choose from the entire pool? Gradient-free optimizers degrade as the number of weights grows into the hundreds, and LoraHub itself chose to pre-filter rather than search the full pool. We test this directly by running LoraHub on progressively larger random pools at a fixed query budget and comparing it with LoraHub seeded by our team.
Why not just fine-tune one fresh LoRA on the target examples? A Unified Study of LoRA Variants shows that a well-tuned plain LoRA is hard to beat, for example 75.51% on GSM8K with Llama-3.1-8B-Base, but that result used full training sets. With 5 to 50 examples, training from scratch is fragile. We include this baseline with a learning-rate sweep, at $n = 50$ to match Adaptive LoRA Merge's regime, and report where the crossover lies.
Proposed Experiments
The primary benchmark is LoRALib, with the target adapter held out per task. We use Llama-3.1-8B-Instruct as the main backbone plus one small and one 9B-class backbone. ELREA observed that gains from multiple adapters shrink at larger scale, so we check whether coverage selection also fades.
The secondary benchmark is BBH with LoraHub's FLAN-T5-large adapter pool, which allows a direct comparison with the 34.7% figure. MATH-Combined serves as a probe where specialist adapters should matter. ELREA reports 20.41% versus 18.61% for a single LoRA on Gemma-2b-it, though ELREA trains its own adapters and so is context rather than a matched baseline.
Metrics are task accuracy, or the benchmark's native score, and the gap closed to manual selection. We also report rank correlation between selection score and held-out score, and inference cost per aggregator.
Two results would be decisive. The first is whether greedy coverage teams match manual selection on LoRALib's quantitative tasks, where perplexity is known to fail. The second is whether the coverage-versus-individual-accuracy ablation shows a consistent gain on mixed tasks under at least the output-level aggregators. If the gain appears only under confidence-gated selection and vanishes under merging, that too is a clear finding. It would mean diminishing-returns selection suits serving several adapters side by side, not merging them into one.
References
- Building a Winning Team: Selecting Source Model Ensembles using a Submodular Transferability Estimation Approach
- LoRALib: A Standardized Benchmark for Evaluating LoRA-MoE Methods
- LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
- Hierarchical Pruning of Deep Ensembles with Focal Diversity
- SLM-MUX: Orchestrating Small Language Models for Reasoning
- LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging
- Ensembles of Low-Rank Expert Adapters
- LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin
- LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks
- Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
- Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generation
- A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation