Would a Second Set of Annotators Reorder the Leaderboard?

Benchmark leaderboards rank models on test labels written by a handful of people, often three per item. On subjective or ambiguous tasks those people disagree a lot. Had a different three people labeled the same test set, the "correct" answers would change, and so might the ranking. Current practice never measures this. The confidence intervals researchers do report come from resampling test items. They treat the labels as fixed, so they ignore the fact that the labels are themselves a random draw from a population of possible annotators. This proposal measures how often rankings flip under a fresh draw of annotators. It then builds a cheap estimator that predicts that flip rate from the three labels a typical benchmark already has, and it compares the size of this "annotator lottery" to the better-known sources of benchmark inconsistency, such as random seeds, preprocessing choices, and selective checkpoint reporting.

The core claim is simple. Near the human-agreement ceiling, a large share of the rank differences between models comes from which annotators happened to label the test set, and the usual item-level bootstrap systematically underestimates that uncertainty. If this holds, many published "wins" on subjective benchmarks are not reproducible even in principle. A leaderboard should then report, for each pair of models, the probability that their order would survive re-annotation.

Where this sits

Two lines of evidence motivate the question without answering it. The first concerns ceilings. Work on agreement ceilings shows that human disagreement caps achievable scores. Label Convergence scores one annotator's boxes against another's and bounds LVIS performance between 62.64 and 67.52 mAP, with leading detectors already at 66.8. ISLES 2016 and 2017 reports neuroradiologist Dice of only 0.58. The TUPAC16 relabeling study shows that a detector's F1 moved from 0.549 to 0.735 just by changing the label set, with no change to the model. That last result is the strongest hint that the identity of the labelers can swamp architectural differences. All three papers produce a single ceiling or a single relabeling. None asks which pairwise model comparisons survive a change of labelers.

The second line concerns benchmarking inconsistency. That literature shows that pipeline choices act as confounders. SentEval freezes the downstream classifier so that encoders can be compared fairly. The Leaderboard Illusion shows that privately testing 10 variants and publishing the best inflates Arena scores by about 100 points, which is an extreme-value effect driven by evaluation noise. These papers treat the evaluation labels as either correct or simply noisy. They never separate out the specific noise that comes from annotator sampling.

Several papers constrain the design. Majority Vote Silences Minority Values finds that on HateXplain disagreement concentrates at the hate/offensive boundary (42.6% of disagreements), and accuracy there drops from about 80% to about 58%. Those are exactly the items where an annotator redraw would flip the label. Crowd-Calibrator notes that crowd distributions estimated with confidence need about 100 annotations per item, which standard benchmarks lack. Our estimator must therefore be validated on dense data and then shown to work on sparse data. NUTMEG and POPQUORN show that disagreement is partly structured by demographics. In POPQUORN, the original Ruddit labels correlate with White annotators at r=0.66 and with Black or Asian annotators at about 0.4. Which "annotator population" we resample from is therefore a substantive choice, not a technicality.

Core Claim

Take a paired comparison of models $A$ and $B$ with metric gap $\Delta$ on a test set. Under a frozen evaluation harness, the variance of $\Delta$ splits into three parts:

$$ \mathrm{Var}(\Delta) \approx \sigma^2_{\text{items}} + \sigma^2_{\text{annotators}} + \sigma^2_{\text{pipeline}} $$

Here $\sigma^2_{\text{items}}$ is the variation from which test items were sampled, which is what the standard bootstrap captures. $\sigma^2_{\text{annotators}}$ is the variation from which people labeled those items. $\sigma^2_{\text{pipeline}}$ is the variation from seeds, hyperparameter draws, and preprocessing. We predict that $\sigma^2_{\text{annotators}}$ grows as agreement falls, measured by Krippendorff's α, and that for top-ranked model pairs on low-agreement benchmarks it is comparable to or larger than $\sigma^2_{\text{items}}$. If so, item-bootstrap intervals are too narrow precisely where leaderboards are most crowded. A second prediction links to Leaderboard Illusion. If one picks the best of $N$ checkpoints on one annotator draw, that checkpoint should regress measurably when scored on an independent draw, even with the test items held fixed.

Research Program

The first component is ground truth for re-annotation. We use datasets dense enough to simulate many independent "benchmark versions." ChaosNLI (SNLI, MNLI, and aNLI items with roughly 100 labels each, as used by Crowd-Calibrator) is the main testbed. POPQUORN offensiveness and politeness serve as dense data with demographic labels. The LVIS consistency subset (5,000 doubly-annotated images) is the vision case. The New Yorker caption ranking data (resource), with over 250 million ratings, is the preference case. From each dense dataset we draw many disjoint three-annotator label sets and majority-vote each one, which gives many realistic test sets that share items but not labelers. We rank a fixed pool of models on each version and record the empirical pairwise flip rate. That rate is the quantity that matters, and it is measured directly, not modeled.

The second component is a sparse-label estimator. It must predict that flip rate using only the three labels per item that a normal benchmark provides. We fit an annotator model that pools strength across items to get a posterior over each item's label distribution. We test two versions: a Dawid–Skene-style confusion model, and a subpopulation-aware variant in the spirit of NUTMEG when demographics exist. We then draw synthetic annotator triples from that posterior and recompute rankings. Running this estimator on three-label subsamples of the dense datasets lets us check its predicted flip probabilities against the observed ones.

The third component is variance attribution under a frozen harness. Following SentEval's practice, we fix the downstream classifier and tuning grid. We train each model family under several seeds and preprocessing variants, and we estimate the three variance terms by crossing items, annotator draws, and pipeline draws in a factorial design. We then apply the estimator to sparse benchmarks with no dense ground truth, namely HateXplain (three raters, α=0.46) and MHSC. We report which published-style gaps are resolvable.

Baselines

The naive reference point is the current default: a point-estimate leaderboard with no uncertainty, where any positive gap counts as a win. On HateXplain this is how the BERT-HateXplain result is presented, with Macro F1 of 0.687 and AUROC of 0.851 as reported in HateXplain. We will measure how many of the point-estimate orderings in our model pool fail to survive re-annotation.

The strongest existing competitor for "which differences are real" is the paired item bootstrap (or an equivalent paired significance test) applied to fixed labels. This is the standard tool, and it is honest about item sampling. We run it on exactly the same predictions, test sets, and number of resamples as our estimator, so the only difference is whether labels are resampled. For this procedure on these benchmarks, there is no reported result on Lacuna. Our estimator beats it if its predicted flip probabilities are better calibrated against the empirically observed ChaosNLI and POPQUORN flip rates, measured by Brier score and reliability curves on model pairs. It also needs to identify a meaningful set of pairs that the bootstrap calls significant but which flip under re-annotation more than 5% of the time.

The closest existing ceiling method is Label Convergence, which reports a ceiling interval of 62.64–67.52 mAP on LVIS against a best detector at 66.8 mAP. We use its annotator-versus-annotator scoring to place a ceiling on each benchmark and test a simple rule: call a pair indistinguishable whenever both models sit inside the ceiling interval. This is a strong, cheap rival. Our method beats it if pairwise resolvability varies substantially among models that are all inside, or all outside, the interval. That would show a single ceiling is too coarse.

Two ablations isolate the key ingredient. In the first, we resample annotators using raw empirical vote proportions from the three labels, with no pooling across items. This tests whether the posterior annotator model matters or whether naive label resampling is enough. In the second, we drop item resampling and keep only annotator resampling. This separates the two variance sources directly.

Why not just…?

Why not just report the item bootstrap, which is already standard? Because it conditions on the labels. On items where three raters split 2–1, the bootstrap treats the majority label as certain. The HateXplain boundary results suggest these are exactly the items that separate competitive models. This is an empirical claim, though, and the ChaosNLI experiment settles it directly. If observed flip rates match bootstrap predictions, the proposal's main claim is false, and we will report that.

Why not just compute the human ceiling and stop? Label Convergence already does this well for detection. A ceiling answers "how high can scores go." It does not answer "is A really better than B." Two models both inside the ceiling can still be reliably ordered if they differ on high-agreement items, and two models outside it can be unresolvable if their differences sit on contested items. We test the ceiling rule head to head, as described above.

Why not just evaluate on the high-agreement subset only? It removes annotator noise by construction. But it changes what is measured: it discards the contested items on which subjective benchmarks are supposed to test models, and it rewards majority-aligned behavior, which the HateXplain analysis already finds to be excessive (about 70% majority alignment). We include high-agreement-only ranking as a reported variant and measure how far its ordering diverges from the full-set ordering under re-annotation.

Why not just score against soft labels instead of majority votes? Soft-label metrics such as cross-entropy against vote proportions are reasonable. With three votes per item, though, the soft target is itself a noisy draw, so the uncertainty problem moves rather than disappears. Crowd-Calibrator's observation that reliable distributions need about 100 labels makes the same point. We run the full study with both majority-vote accuracy and soft-label divergence as the metric and report flip rates for each. If soft metrics alone remove most annotator-driven flips, that is a cheaper fix, and we will recommend it.

Proposed Experiments

The decisive experiment uses ChaosNLI. We take a pool of roughly two dozen NLI classifiers trained under a frozen harness with multiple seeds. We build 200 disjoint three-annotator test versions, rank the models on each, and measure pairwise flip rates. We then ask whether item-bootstrap intervals computed on any single version predict those flips, and whether our sparse-label estimator predicts them better. We replicate on POPQUORN. There we also compare resampling from the full annotator pool with resampling within demographic groups, to see whether the leaderboard order depends on whose labels are used. On LVIS, with only two annotators, we use the annotator swap itself as the redraw and compare against the Label Convergence interval.

The variance decomposition experiment crosses items, annotator draws, and pipeline draws on ChaosNLI and POPQUORN, and plots $\sigma^2_{\text{annotators}}$ against Krippendorff's α across task subsets. A clear upward trend as agreement falls supports the claim. A flat, negligible term refutes it.

The winner's-curse experiment mirrors Leaderboard Illusion's best-of-$N$ finding, with labels as the only source of noise. We select the best of $N$ checkpoints, for $N$ from 1 to 20, on one annotator draw and score it on an independent draw with identical items. We measure the drop as a function of $N$ and of agreement. Finally, we apply the validated estimator to HateXplain and MHSC and report, for each model pair, the probability that its order survives re-annotation. That turns the method into a reporting standard that sparse benchmarks can adopt without collecting new labels.