Shared-Neighbor Anchors: Cold-Start Benchmark Compression from a Handful of Cheap Models

Evaluating a large language model on a full benchmark suite is expensive. A complete HELM run can cost more than $10,000 in API fees or 4,000 GPU hours (tinyBenchmarks). The best compression methods pick a few dozen "anchor" questions whose results predict the full score. They depend on a large table of past results: how many earlier models got each question right, and how confident they were. For a new benchmark that table does not exist. We propose to build it cheaply by running a small panel of inexpensive open models once. The difficulty is that with only a handful of models, the direct correlation between two questions' results is too noisy to show which questions are redundant. Our fix borrows an old idea from word semantics. Call two questions similar if they correlate with the same other questions, not only if they correlate with each other. We compute this "shared-neighbor" similarity with hashing, which avoids comparing every pair of items.

The analogy is close. In distributional semantics, about 84% of related word pairs never appear together in text (Quantifying Context Overlap). Such pairs are only detectable through the contexts they share, which is what second-order similarity means. A sparse panel of models plays the role of a sparse corpus. Two questions that test the same skill may have unreliable direct correlation across five models, yet have heavily overlapping sets of correlated neighbors across the whole benchmark. The claim to test is specific. When the model panel is small, choosing anchors with second-order similarity recovers held-out models' full-benchmark accuracy and ranking better than the first-order correlation used by Anchor Points, and at least as well as the low-rank model fit by tinyBenchmarks. Hashing makes this possible on pooled item banks with hundreds of thousands of questions.

Where this sits in prior work

Three compression methods motivate the work and supply its baselines. Anchor Points groups items with K-medoids, using one minus the correlation of logit-transformed confidences across source models as the distance. It reports that 30 anchors predict a held-out Falcon-7B's results on the remaining 842 SST-2 items with 92% agreement and mean absolute error 0.09. tinyBenchmarks fits an item response theory model, a low-dimensional latent model of item difficulty and model ability. With 100 items it predicts MMLU accuracy within 1.9% on average. EssenceBench uses genetic search with learned predictors of the full score, called surrogates. It keeps 95% of HellaSwag model rankings with 50 items. All three assume a dense history of prior model runs. We keep their goal and drop that assumption. We replace the history with a deliberately small panel and ask which notion of item similarity holds up when that panel is thin.

EssenceBench also shows that questions with no shared words can have identical ranking behavior. This is a vocabulary-mismatch problem inside the benchmark itself. It is why we do not use text similarity to find redundancy, and it gives us a diagnostic: how many of the redundancies we find cross zero lexical overlap. The lexical-gap literature enters in one more way. Retrieval-style benchmarks such as WikiQA, where 20.3% of correct answers share no content words with the question, are exactly the benchmarks on which text-based subset selection should fail. We include them for that reason.

One prior result constrains the proposal directly. Schlechtweg et al. show that low-rank methods such as SVD and skip-gram already capture second-order paths. Explicitly adding second-order links hurt SVD by 0.17 Spearman on WordSim353, while the same addition helped count-based PPMI. Fitting item response theory is a low-rank factorization, so it may already capture what we propose to add. We treat this as the main threat and design the experiments to settle it. The opposite finding, that adding explicit context overlap helped GloVe (Zhuang et al.), suggests the answer depends on regime. Our hypothesis is that the explicit version wins when too few models are available to fit latent dimensions reliably.

The hashing method comes from coreset work. HaCore uses binary hash signatures and an inverted index to cut K-medoids cost from $O(n^2)$ to $O(n)$. Balanced LSH shows that random hyperplanes put most points into a few buckets when data are clustered, and fixes this with thresholds adapted to the data. Both hash raw features or embeddings. Neither hashes relations between items, such as correlations or shared-neighbor sets. NeuroCartography uses MinHash, a set-similarity hash, on neuron activation sets. We apply the same tool to the sets of items each item correlates with.

Core mechanism

The panel consists of $m$ cheap models. For each benchmark item $i$ we record their logit-transformed confidences on the correct answer, then standardize these values to mean zero and unit variance across the panel, giving a profile $z_i \in \mathbb{R}^m$. After standardization, the cosine between two profiles equals their Pearson correlation. Random-hyperplane hashing, called locality-sensitive hashing (LSH), then gives an unbiased estimate of the first-order similarity. Let $r$ be a random Gaussian vector in $\mathbb{R}^m$ and let $\rho_{ij}$ be the Pearson correlation between items $i$ and $j$. Then

$$\Pr_r[\operatorname{sign}(r^\top z_i)=\operatorname{sign}(r^\top z_j)] = 1-\frac{\arccos \rho_{ij}}{\pi}.$$

This follows from the standard angle identity for random hyperplanes (DeepCAM). With multiple hash tables, each item gets an approximate set $N_k(i)$ of its $k$ most-correlated items without computing all $D^2$ pairs, where $D$ is the number of items.

The second-order similarity measures how much two items' neighbor sets overlap:

$$s_2(i,j) = \frac{|N_k(i)\cap N_k(j)|}{|N_k(i)\cup N_k(j)|}.$$

MinHash signatures of the neighbor sets estimate this overlap. A second round of banded hashing proposes candidate pairs, and K-medoids runs over the resulting sparse similarity graph, as in HaCore's inverted-index approach. We then estimate the full-benchmark score from the anchors the same way Anchor Points does, so the only thing that differs from that baseline is the similarity used to choose anchors. There is one design choice we expect to matter. Profiles of very easy or very hard items are nearly constant across models, and after standardization they fall into a few buckets. We will compare balanced thresholds from Balanced LSH with plain sign thresholds.

Research program

The first stage builds the test bed from existing score matrices. We use the public per-item results released with tinyBenchmarks (Open LLM Leaderboard style, covering MMLU, HellaSwag, GSM8K, ARC, and WinoGrande) and EssenceBench's released code and data (EssenceBench repository). To simulate a cold start, we split the models into a "cheap panel" pool of the smallest models and a "target" pool of larger or later-released models. Anchors are chosen using only panel models. All scoring is done on target models.

The second stage compares similarity measures while varying panel size $m$ (3, 5, 10, and 20 models) and anchor budget (10 to 100 items). The measures are first-order correlation, IRT or low-rank latent embeddings fit to the same panel, exact second-order overlap, and hashed second-order overlap. This stage answers the main question: whether second-order structure adds anything beyond low rank, and in which regime.

The third stage is about scale. We pool every item across the benchmarks into one bank of roughly 100k or more items. At that size an exact $D\times D$ correlation matrix, and especially its second-order product, becomes a memory and time problem. We measure wall-clock time and fidelity loss of the hashed version against the exact one as a function of the number of tables and bits per signature. We also count how many anchor-to-member assignments link items with zero content-word overlap, which quantifies the lexical-gap redundancies that text methods cannot see.

The fourth stage transfers the method to retrieval-style evaluation. For WikiQA, the panel consists of cheap answer-selection or retrieval models, and the targets are stronger rerankers. We test whether anchors chosen from the panel preserve the targets' mean average precision (MAP) ranking.

Baselines

The naive reference is uniform random sampling at each anchor budget, averaged over many draws. tinyBenchmarks and Anchor Points both report that it has high variance and loses at small budgets. Any method that fails to beat it at matched budget has no value.

The most direct competitor is Anchor Points with first-order correlation distance (Anchor Points). It is the same pipeline with the same K-medoids step and estimator, fed first-order rather than second-order similarity, so the comparison isolates the key ingredient. Its reported result is 92% agreement and MAE 0.09 using 30 anchors on SST-2, with a dense source-model history. We run it on the identical panel with identical budgets. Beating it means lower target-model accuracy error and higher Spearman rank correlation at small $m$, with confidence intervals from resampling panels.

The strongest existing method on standard benchmarks is tinyBenchmarks' item response theory estimator. It reports 1.9% average MMLU error with 100 items and Spearman above 0.95, with a full history. We refit it on each restricted panel and give its latent dimension and regularization the same validation-tuning budget as our neighbor count $k$ and hash settings. EssenceBench is the strongest reported compressor on HellaSwag, retaining 95% of rankings with 50 items. We run its released code on the same panels at an equal selection-compute budget. A dense text-embedding K-medoids baseline tests whether model behavior is needed at all. Finally, the exact-versus-hashed second-order comparison is the ablation that separates the effect of the similarity measure from the effect of the approximation. The claim succeeds only if second-order similarity beats both first-order correlation and IRT at some panel size, and hashing preserves most of that gain.

Why not just…?

Why not just randomly subsample? At budgets of 30 to 100 items, prior work consistently finds random subsets too noisy. Anchor Points beat random sampling in 29 of 30 settings, and tinyBenchmarks documents lucky and unlucky draws. We still report random sampling at every budget. If our method's advantage over it disappears at small panel sizes, that is a negative result we would publish.

Why not just run Anchor Points with the small panel? This is the natural cheap option. Our hypothesis is specifically that first-order correlations become unreliable when $m$ is small, because a correlation across five points is very noisy. The panel-size sweep tests this directly. If first-order and second-order similarity converge at every $m$, then the second-order ingredient is unnecessary.

Why not just fit a low-rank model such as item response theory, which already captures shared structure? This is the most serious threat. Schlechtweg et al. found that low-rank factorizations already encode second-order paths, and that explicitly adding them can hurt. If IRT fit to the panel matches exact second-order similarity at all panel sizes, the accuracy claim is wrong. What would remain is only the scaling contribution: hashing a pooled bank faster than refitting a latent model. We expect a crossover in which second-order overlap helps when $m$ is too small to identify latent dimensions and loses its advantage as $m$ grows. Stage two is designed to locate that crossover, or to show there is none.

Why not just cluster items by their text embeddings, which is free? EssenceBench shows that textual redundancy and ranking redundancy come apart, and Anchor Points reports embedding baselines losing to confidence-based anchors. On WikiQA-style items, where correct answers often share no words with the question, text similarity should fail most clearly. The text-embedding baseline, together with the count of zero-overlap redundancies, will show how much of the gain comes from behavioral information.

Why not just compute everything exactly? For a single benchmark of 14k items, exact computation is affordable. Hashing earns its place only on pooled banks and in setups where the benchmark keeps growing. We report both regimes and do not claim a speedup where one is not needed.

Decisive checks

Four results decide the project. The first is how target-model score error and Spearman correlation vary with panel size $m$ for first-order, IRT, and second-order similarity. The proposal survives only if second-order similarity wins clearly at small $m$ on at least MMLU and HellaSwag. The second is whether the hashed version stays close to exact second-order fidelity on the pooled bank while running substantially faster. The third is whether the advantage holds on target models released after the panel models, a test of the drift concern raised for all history-based methods. The fourth is whether panel-chosen WikiQA anchors preserve the targets' MAP ranking better than text-based selection. No reported result was found on Lacuna for cold-start compression of any of these benchmarks, so all of these comparisons will be run fresh.