Many benchmarks now rank systems with a learned "meta-metric": a small model, such as a regression or boosted trees, that combines a dozen cheap automatic scores (BLEU, BERTScore, factual-consistency checkers, and so on) into one number fitted to match human ratings. When such a leaderboard names a winner, the ranking is uncertain for two reasons. The first is that the test items are a finite sample, and existing ranking-uncertainty tools already handle this. The second is that the combination weights were themselves fitted on a small, expensive set of human ratings, and a different draw of those ratings would give different weights and possibly a different winner. Current confidence procedures ignore this second source. We propose a paired bootstrap that resamples both the human-rating set used to fit the meta-metric and the test set used to score systems. It refits the whole combination pipeline, including its metric pruning step, inside every replicate, and outputs a set of systems that contains the true best one with a stated probability.
The procedure answers a question the fusion literature has not asked: once a learned combination pays for its own estimation noise, does it still separate systems better than a single strong metric or a fixed, untrained average? If fusion reduces noise at the level of individual items more than fitting the weights adds noise at the level of the leaderboard, honest confidence sets should shrink. If it does not, the extra correlation that fusion papers report is not buying decisions. Both outcomes change how leaderboards built on meta-metrics should be read.
Core Claim
Treating a fitted meta-metric as fixed and applying post-selection ranking inference to it will under-cover. The shortfall should be worst when the rating set used for fitting is small, which is the usual regime for expert annotation. A two-level paired bootstrap that refits the aggregator in each replicate restores the nominal coverage. The same machinery lets us measure, for the first time, whether learned fusion narrows or widens the set of plausible winners compared with its components.
The starting point is The Leaderboard Lottery. That paper shows that declaring a winner is a selection among all systems at once. It introduces paired-bootstrap Multiple Comparisons with the Best (MCB), which resamples test prompts jointly across models and applies a Holm step-down correction. It reports at least 95% coverage of the true best model, while marginal intervals crowned the true best only 0–58% of the time, and 0% on GPQA. That work assumes a single fixed scalar score per benchmark, and that assumption is exactly what we relax. The fusion methods we study come from three sources:
- MetaMetrics, which fits linear weights by Bayesian optimization and fits XGBoost with iterative metric pruning.
- EnsembEval in Surveying the Landscape of Image Captioning Evaluation, which is sparse linear regression with feature selection. It reaches 58.8 human correlation on Polaris against 57.8 for the best single metric, Polos.
- Fusion-Eval, which reports 0.962 system-level Kendall-Tau on SummEval.
All three report point correlations with no resampling of the fitting step. The EnsembEval authors themselves note that significance testing is rarely reported in this area. A one-point gain like 58.8 against 57.8 is precisely the kind of difference whose stability is unknown.
Research Program
The data structure is shared across benchmarks. Each benchmark has a set of source items, such as documents, source sentences or images. Every competing system produces one output per item. A subset of items carries human ratings. We split items into calibration items, used to fit the meta-metric, and evaluation items, used to score systems, with no item in both. This grouping matters: SummEval-style datasets rate the same outputs one would later rank, so fitting and scoring on overlapping items leaks information.
Each bootstrap replicate $b$ runs the following steps:
- Draw calibration items with replacement and refit the full aggregator on them, including the pruning or feature-selection step.
- Draw evaluation items with replacement, using the same draw for every system so the paired structure is preserved.
- Score every system with the refitted aggregator.
- Record, for each system, its score gap to the best system in that replicate.
Let $\hat\theta_j$ be system $j$'s mean meta-score on the full evaluation set, and let $\hat\theta^{(b)}_j$ be the same quantity in replicate $b$. The confidence set is
$$S_\alpha = \{\, j : \max_k \hat\theta_k - \hat\theta_j \le c_j(\alpha) \,\},$$
where $c_j(\alpha)$ is the Holm-adjusted bootstrap quantile of the centred gap $(\hat\theta^{(b)}_k - \hat\theta^{(b)}_j) - (\hat\theta_k - \hat\theta_j)$, following Jain's construction. The only change from Jain is where $\hat\theta^{(b)}$ comes from: it is produced by an aggregator refitted on resampled calibration data rather than a frozen one.
The base metrics are computed once per output and cached. Each replicate therefore only refits XGBoost or a regression on a feature table, which removes the computational objection that heavy neural metrics make 1,000 replicates impractical.
We must be precise about the target. MCB guarantees coverage of the best system under the population meta-metric, meaning the aggregator one would fit with unlimited ratings. It does not cover the best system under human judgment. We report both kinds of coverage and keep them separate, because a meta-metric can be stably wrong. The gap between the two is a finding in itself, not something to hide.
As a by-product, each replicate records which metrics survive pruning. This gives selection frequencies for the committees chosen by MetaMetrics and EnsembEval, and it tests directly whether those committees are artifacts of a particular sample. Asymptotics of the Bootstrap via Stability is the theoretical caution here: bootstrap validity after model selection depends on how stable the selection is. We therefore expect coverage to degrade most for greedy pruning. We will test this rather than assume it.
Baselines
The naive reference point is Jain-style paired MCB applied to a single strong metric with no fusion. We use COMET-family or chrF-style segment-averaged scores on WMT23. For WMT23 we follow Sentence-level Aggregation of Lexical Metrics, which found that segment-level m-chrF reaches 0.806 correlation with MQM against 0.392 for corpus-level chrF, so the single-metric baseline will be segment-averaged. Its confidence sets have no fitting variance and show what resolving power we have without fusion.
The strongest competitor is what a careful practitioner would do today. They fit a MetaMetrics-style aggregator, both the Bayesian-optimization linear version and XGBoost with pruning, once on the calibration items, freeze it, and run paired-bootstrap MCB over evaluation items only. This is exactly Jain's method applied to a learned score. Jain reports at least 95% coverage of the true best for fixed scalar metrics on Open LLM Leaderboard v2 tasks, with sets of 8 to 35 models. There is no reported result on Lacuna for MCB coverage when the score is a learned meta-metric, and closing that gap is the point of this work. The competitor gets the same cached features, the same calibration and evaluation split, the same number of replicates, and the same aggregator hyperparameter budget. It differs only in not refitting the aggregator. Our method beats it if frozen-aggregator MCB falls measurably below 95% coverage of the population meta-metric best at small calibration sizes while ours stays at or above 95%. We also require that our sets not grow so much that they become uninformative, which we check by comparing set sizes at matched coverage.
The ablation that isolates the key ingredient is a middle variant. It resamples calibration items and refits the aggregator weights, but keeps the metric subset chosen on the full data. The gap between this variant and full refitting measures how much of the missing uncertainty comes from selection instability rather than weight estimation. A second reference is a fixed-weight fusion: an unweighted average of standardized metrics with nothing fitted. We discuss it below because it is the most serious alternative.
Why not just…?
Why not just freeze the fitted meta-metric and run Jain's MCB? It is cheaper and already has a guarantee. That guarantee, however, assumes the score function is fixed and known. A meta-metric fitted on 20 to 100 rated documents is a random function, and its variance is the component we suspect dominates at small calibration sizes. The main experiment measures this directly. If frozen MCB already covers at every calibration size we test, we will report that the refitting step is unnecessary, and that outcome is a legitimate one.
Why not just use a fixed, untrained average of standardized metrics? This is a real threat. An untrained average has no fitting variance, so its MCB sets are honest by construction, and it may keep most of the gain in item-level correlation. MetaMetrics and EnsembEval show that learned weights beat single metrics in point correlation, with 58.8 against 57.8 for EnsembEval on Polaris. Neither compares against an untrained average under honest uncertainty. We run it through the same MCB and compare set sizes. If the untrained average gives sets as small as the learned aggregator's honestly bootstrapped ones, the practical recommendation becomes to use untrained fusion for leaderboards, and we will state that plainly.
Why not just cross-validate the aggregator? Cross-validation estimates how well the aggregator correlates with humans on held-out data. It does not produce a confidence set over which system is best, and it does not propagate weight uncertainty into the ranking. Held-out correlation is also what MetaMetrics already reports. We include k-fold refitting as an alternative resampling scheme inside MCB, drawing weights from fold fits instead of bootstrap fits. We expect it to under-cover because it yields only k weight draws, but this is an empirical comparison.
Why not just ask a strong LLM judge? A monolithic judge also has hidden variance from prompt wording, which Fusion-Eval documents. It is not the object studied here. Our procedure applies to any score whose parameters are fitted on ratings, and we leave LLM judges for future work.
Proposed Experiments
We use SummEval through the Yale-LILY/SummEval resource, with 100 documents and outputs from many systems, rated by experts and crowds. The WMT 2023 Metrics task data provides MQM for multiple language pairs, each with many systems. Polaris provides captioning, with outputs from several vision-language models.
To obtain ground truth for coverage, we treat the full rated pool of each benchmark as the population. We define the population meta-metric best by fitting the aggregator on all items and scoring on all items, and the human best by mean expert rating. We then repeatedly draw smaller calibration and evaluation subsets, run each method, and record whether $S_\alpha$ contains each target.
We vary the calibration size from about 20 to 50 items on SummEval, and more widely on WMT23. We report coverage, mean set size, and the frequency with which the empirical winner is the true best. The agreement between expert and crowd ratings on SummEval, Krippendorff's alpha of 0.71 against 0.49 (SummEval), lets us repeat the analysis with noisier crowd calibration labels. This tests whether label noise widens honest sets as expected.
We will treat the proposal as settled by four checks:
- Frozen-aggregator MCB drops below nominal coverage at small calibration sizes, and our refitting bootstrap stays nominal.
- Metric selection frequencies show whether the pruned committees reported by MetaMetrics and EnsembEval recur across replicates or change from draw to draw.
- Honest set sizes show whether learned fusion, the untrained average, or the best single metric resolves the leaderboard most sharply.
- The difference between population-meta-metric coverage and human-best coverage shows how often a stable meta-metric points confidently at the wrong system.
References
- The Leaderboard Lottery: Why Top LLM Rankings Are Often Coin Flips, and How to Fix Them
- MetaMetrics: Calibrating Metrics for Generation Tasks Using Human Preferences
- Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis
- Fusion-Eval: Integrating Evaluators with LLMs
- Asymptotics of the Bootstrap via Stability with Applications to Inference with Model Selection
- Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation
- SummEval: Re-evaluating Summarization Evaluation