A large model becomes noticeably more accurate and better calibrated when its predictions are averaged with those of a small model from a different architecture family. The small model, a "sidekick," costs only about a tenth of the large model's compute. Nobody knows how to find a good sidekick, though, except by trying candidates by hand. Meanwhile public hubs hold more than a million models, and nearly 60% of them have empty or missing documentation. This note proposes a shared index that records how each hub model behaves on one fixed set of probe inputs. From that index, a user with a single GPU can predict which small models will make the most complementary partners for their own model. The index never needs a README, and the user never has to download and test thousands of candidates.
The two-sentence version is this. Every model in the hub is run once, centrally, on the same unlabeled probe images, and its internal features are stored. A user runs only their own model on those probes, ranks the hub by how differently each candidate sees the probes combined with how competent it is on a handful of the user's labeled examples, and then tests only the top few. The claim worth testing is that this label-free difference in behavior on generic probes predicts how much error reduction a pairing will actually deliver on the user's task. If it does, the expensive search step moves from the poorest party (the user) to a one-time shared cost.
Where this sits in prior work
Three bodies of work motivate the proposal without covering it.
Asymmetric Duos supplies the method we reuse. It pairs a large base model with a small sidekick from another architecture family and learns one weight per model on a validation set. The authors report 10–15% gains in accuracy and uncertainty metrics for 10–20% more compute, matching a 5-model deep ensemble at 4x less compute. They pick sidekicks manually from standard model suites and say there is no recipe beyond empirical testing.
Wisdom of Committees builds confidence-based cascades of off-the-shelf models. It reaches a 5.4x FLOP speedup over EfficientNet-B7, but it assumes the candidate pool is known and notes that choosing members from a large pool is expensive.
Hub-indexing work makes undocumented models searchable by behavior, but only to answer "find me one model that does X":
- ProbeLog fingerprints each output unit by its responses to 1,000 fixed COCO probe images. It reaches over 40% top-1 text-to-model retrieval on INet-Hub, against 0.1% for random search.
- ProbeX infers training attributes from a single hidden layer.
- CDDM maps datasets to domains to choose a single feature extractor.
We borrow ProbeLog's idea of a fixed shared probe set, including its finding that probing a random subset and filling the gaps by matrix factorization cuts encoding cost 3x. We change the query from "which model matches this description" to "which model fails differently from mine."
Two results constrain the design. Ensembling Tabular Foundation Models found that models from different origins can still form a near-redundant pool, where averaging falls into a calibration trap. Architectural difference is therefore not enough, and functional difference has to be measured. ProbeX reports negative transfer across "model trees," meaning families of models fine-tuned from a common ancestor. That warns that features learned inside one family will not compare cleanly across families. So we use a comparison that is independent of architecture and needs no training: similarity between the two models' feature geometries on the probe set.
The Appeal and Reality of Recycling LoRAs is a further warning about reusing hub models naively. Merging about 1,000 wild LoRAs scored about 0.67 against about 0.66 for training one new LoRA on 100 samples, and random LoRAs did as well as real ones. Their bottleneck was the missing metadata. We keep models separate rather than merging them, and we explicitly test whether the chosen partner adds real information or merely acts as a regularizer.
Core Claim
Representational dissimilarity on a generic, task-agnostic probe set, measured with no labels, combined with cheap competence estimates from a few labeled target examples, predicts realized ensemble gain on the target task. The prediction should be good enough that selecting from an undocumented hub nearly matches exhaustive search. It should also clearly beat selection by architecture family, lineage, or model-card text. Under this claim the user's compute cost becomes one probe pass of their own model plus evaluation of $k$ candidates, instead of evaluation of all $N$.
Research Program
The central index. The index stores, for each hub backbone $m$, its penultimate-layer features $Z_m$ on a fixed probe set. The probe set is unlabeled natural images, following ProbeLog's COCO set. The index also records each model's FLOP cost and, as optional descriptive metadata, its lineage recovered by MoTHer.
The selection score. For a user's base model $b$ with probe features $Z_b$, and a FLOP budget $\beta$ (default 10% of $b$'s cost, from the Duos "10% rule"), candidates are scored as
$$s(m) = \big(1 - \mathrm{CKA}(Z_b, Z_m)\big) \cdot \widehat{\mathrm{acc}}_m \quad \text{for } \mathrm{FLOPs}(m) \le \beta,$$
The terms are defined as follows. CKA, centered kernel alignment, is a standard similarity between two sets of features that is invariant to rotation and scaling, so it compares models of any architecture. $\widehat{\mathrm{acc}}_m$ is the accuracy of a linear head trained on $m$'s frozen features using the user's few labeled examples. Training such a head costs seconds per candidate, but only after a first stage has filtered the pool by CKA alone. The product form encodes the Duos observation that a sidekick must be both different and not useless. We will compare it against a learned combination fitted on held-out tasks.
The final pairing. For the top-$k$ candidates, the user builds the actual Duo with Zhou et al.'s learned weighted average and keeps whichever pairing is best on validation. A second study asks whether the same score also orders members for a Wisdom-of-Committees cascade. That study is secondary and is reported only if the Duo result holds.
Baselines
Do nothing. The reference point is the base model alone, fine-tuned or linear-probed on the target data. Every method shares its validation split and calibration procedure.
Hand-picked Duo. This is the strongest existing method: Asymmetric Duos with Zhou et al.'s own recipe. We take a sidekick from a different architecture family at about 10% of the base FLOPs, using their base/sidekick pairs where they are named, such as a ViT-B or ConvNeXt base with a ResNet-34 sidekick. Their reported figure is "10–15% improvements in accuracy and uncertainty metrics with only 10–20% more computation." The file has no per-dataset numbers, so we re-run their pairs under our protocol rather than quote one. Beating this baseline means a probe-selected sidekick from the hub gives a statistically better Duo on a majority of base models and datasets at equal FLOPs. Matching it while discovering the partner automatically from undocumented models is the weaker, still useful, outcome.
Exhaustive oracle. Every candidate under the budget is evaluated on the user's labeled data and the best Duo is kept. This is the upper bound, and the main metric is regret against it at a fixed user compute budget.
Heuristic selectors. Three cheap rules choose among the same pool with the same FLOP cap and the same $k$ final evaluations:
- Architecture-family difference, read from configuration files, which usually survive even when READMEs are empty.
- Different MoTHer-recovered tree.
- Model-card text retrieval, applied where cards exist.
A random sidekick under the budget is also included.
Deep ensemble. A 5-member homogeneous deep ensemble is included for reference on accuracy per FLOP.
Ablations. Two ablations isolate the key ingredient: CKA-only, and competence-only (the most accurate small model, ignoring difference). The competence-only ablation is the decisive one. If it matches the full score, functional difference adds nothing and the claim fails.
Regularizer control. Following Liu et al., we pair the base with a randomly initialized backbone of the same architecture plus a trained head. This tests whether gains come from real complementary information.
Why not just…?
Why not just read architecture from config files and pick a different family at 10% FLOPs? This is the real threat. The Duos paper suggests architectural difference drives most of the gain, and config files are usually present even in undocumented repositories. The weakness is that hubs hold many same-architecture models with different training data and fine-tuning histories, which this rule cannot tell apart. The tabular result shows that different origin does not guarantee different errors. The heuristic-selector baseline settles the question directly. If its regret against the oracle is statistically indistinguishable from ours, we will report that the probe index is unnecessary for sidekick choice and that the architecture rule should become the default recipe.
Why not just evaluate every candidate on your own validation set? For a small pool and a well-resourced user, this is correct, and it is our oracle. The cost grows with pool size. Each user must download and run every candidate under the budget, which is exactly the cost resource-poor groups cannot pay. With a shared index the cost is one probe pass plus $k$ evaluations. We report user-side GPU-hours and regret as a curve over $k$. The proposal is justified only where small $k$ gets close to oracle quality.
Why not just search model cards with a language model? With about 60% of hub models lacking any README (Kahana et al.), the text cannot be searched. The text that does exist describes what a model does, not where it fails, and Zainyte et al. show that static documentation drifts from the weights it describes. We include text retrieval as a baseline on the subset of models with cards, so the comparison is made where text exists at all.
Why not just fine-tune the base model better, or merge adapters? Duos reports that the pairing gives gains that tuning the base alone did not, and Liu et al. show that merging hub adapters gives roughly what a single new adapter gives. The do-nothing baseline uses the same tuning budget as the pairing, so this question is answered in our own numbers rather than assumed.
Proposed Experiments
Candidate pool. The pool combines public ImageNet-pretrained backbones from the hub, spanning CNN, ViT, and hybrid families with many fine-tuned variants of each, with the 1,500 undocumented INet-Hub models from ProbeLog. INet-Hub is useful precisely because its models differ in training data rather than architecture. Metadata is stripped from every model before indexing.
Targets and base models. Targets are ImageNet, ImageNet V2, iWildCam, and Caltech 256, matching the Duos evaluation. For the transfer targets, candidates are frozen backbones with user-trained linear heads. Base models are ViT-B and ConvNeXt variants.
Metrics. Metrics follow Duos: top-1 accuracy, expected calibration error, and selective-classification coverage at 98% accuracy. They are reported against total FLOPs and against user-side GPU-hours.
The decisive checks. The first check is the Spearman rank correlation between $s(m)$ and realized Duo gain across all budget-feasible candidates for each base/target pair. The second is regret against the exhaustive oracle at $k \in \{1, 5, 20\}$, compared with the heuristic selectors and the competence-only ablation. The third is the iWildCam out-of-distribution split, where correlated failures should matter most. If the probe score beats the architecture rule there and not in-distribution, that is an informative, narrower result.
Probe cost. A secondary study varies probe set size and uses ProbeLog's collaborative probing, to measure how much of the shared indexing cost can be cut before ranking quality drops.
References
- Asymmetric Duos: Sidekicks Improve Uncertainty
- Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
- Can this Model Also Recognize Dogs? Zero-Shot Model Search from Weights
- Learning on Model Weights using Tree Experts
- AutoML Meets Hugging Face: Domain-Aware Pretrained Model Selection for Text Classification
- Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap
- The Appeal and Reality of Recycling LoRAs with Adaptive Merging
- Unsupervised Model Tree Heritage Recovery
- Challenges and Future Directions for Accountable Machine Learning