One-Step Estimates of True Accuracy on Crowd-Labeled Benchmarks

Many benchmarks are built by asking several non-expert workers to label each item and then merging their votes into one "ground truth" label. Models are then ranked by how often they agree with those merged labels. That merged label is itself a noisy estimate, so a leaderboard score measures agreement with a guess, not accuracy against the truth. We propose to estimate the quantity people actually want, a model's accuracy against the unobserved true labels, together with a valid confidence interval. The estimate is computed directly from the raw worker votes, uses no expert labels, and costs one Newton-style correction step on top of a cheap starting estimate.

The tool is the one-step estimator, a classical idea from statistics. Start from a quick estimate that is roughly right, meaning its error shrinks like $1/\sqrt{n}$ with $n$ items. Take a single second-order correction step. The remaining error then shrinks like $1/n$, which is small enough that the result is as precise as the full maximum-likelihood fit. Our claim is specific. When items differ in difficulty, which Reliable Crowdsourcing under the Generalized Dawid-Skene Model argues is the normal case in real crowd data, the standard practice of scoring a model against aggregated labels is biased. A one-step correction aimed at the single number "true accuracy of model $f$" removes most of that bias while the model of worker behavior stays cheap and imperfect. When difficulty is uniform, we expect no gain, and the experiments are designed to show both cases.

Core Claim

Existing work applies one-step refinement to recover the labels themselves. Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing seeds the Dawid-Skene model with a spectral moment estimate and shows that one or two expectation-maximization (EM) iterations reach the minimax-optimal label error. The Dawid-Skene model treats true labels as hidden and gives each worker a fixed confusion matrix, which records how often that worker reports each class given the true class. That paper, and its evaluations on Bird, RTE, TREC, Dog and Web, never asks what the aggregation error does to a downstream score. We treat the benchmark score itself as the target.

Let $y_i$ be the hidden true label of item $i$, $f(x_i)$ the prediction of the model under evaluation, and $\psi = \frac{1}{n}\sum_i \Pr(f(x_i)=y_i)$ its true accuracy. Let $\eta$ collect the nuisance parameters: worker confusion matrices plus a low-dimensional description of item difficulty. The estimator is

$$\hat\psi_1 = \hat\psi_0 + \frac{1}{n}\sum_{i=1}^n \varphi(L_i, f(x_i); \hat\eta_0),$$

where each symbol is defined as follows. $L_i$ is the vector of worker votes on item $i$. $\hat\eta_0$ is the cheap spectral seed. $\hat\psi_0$ is the plug-in accuracy: the average posterior probability that $f$ is correct under $\hat\eta_0$. $\varphi$ is the efficient influence function of $\psi$ under the difficulty-aware model. This is the one-step form that targeted maximum likelihood estimation and high-dimensional debiasing use. Because $\varphi$ is built to be insensitive to small errors in $\eta$, the seed's $O(1/\sqrt{n})$ error enters $\hat\psi_1$ only at second order.

The difference from prior work is the choice of target. We do not claim better labels. We claim a better and properly uncertain estimate of a model's score, and of the score gap between two models, which is what a leaderboard reports.

The reason this is not trivial deserves stating plainly. If the homogeneous Dawid-Skene model were correct and EM reached the global optimum, the plug-in posterior accuracy would already be efficient, and a one-step correction would add nothing. The gain has to come from difficulty heterogeneity. If each item gets its own difficulty parameter, the number of parameters grows with $n$, full maximum likelihood becomes non-convex and statistically unreliable, and plug-in scores inherit bias concentrated on the hardest items. The generalized Dawid-Skene analysis shows that the error of non-adaptive methods is governed by the worst-case difficulty $\lambda_{\min}$ rather than the typical $\lambda$. Our estimator describes difficulty with a few latent bins or a function of item features. It profiles those out through the efficient score and corrects only in the direction of $\psi$.

Research Program

We will first derive $\varphi$ for accuracy and for pairwise accuracy differences under a generalized Dawid-Skene model with binned item difficulty. The derivation follows the orthogonal-score construction that SIHR: Statistical Inference in High-Dimensional Linear and Logistic Regression Models uses for linear functionals of penalized regressions. That work supplies the method: correct only the functional you care about, not the whole parameter. The seed $\hat\eta_0$ reuses the tensor-based spectral initializer of Opt-D&S, fitted to the homogeneous model, and the difficulty bins are initialized from vote-disagreement quantiles. Worker-level cross-fitting keeps $\hat\eta_0$ and the correction from reusing the same votes.

Second, we will address a known failure mode of single correction steps. Targeted Maximum Likelihood Learning: An Optimization Perspective proves that one-step targeted updates can overshoot the root of the estimating equation in finite samples. That work constrains our design: the probability of overshooting decays with $n$, but legacy crowd benchmarks are small. We will therefore compare the raw single step, a damped step that stops at the root, and a second iteration. Overshooting is directly detectable here, because the corrected $\hat\psi_1$ must lie in $[0,1]$ and the estimating equation can be evaluated after the step.

The broader lineage motivates the design and gives no evaluation targets here. That lineage includes the closed-form seed plus one Gauss-Newton step in Consistent and Optimal Solution to Camera Motion Estimation, which also admits that efficiency is lost under non-identical noise, and the averaged-seed Newton step in A Distributed One-Step Estimator. The case for the proposal comes from the benchmark-reliance literature. Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection shows that crowd collection leaves systematic artifacts in benchmark labels. Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning shows how far a score can move with the label source: Claude-3-Opus had a 54.4% win rate against top human captions according to an AI judge, but a New Yorker editor preferred its captions 1.6% of the time. Both show that benchmark conclusions depend on how the reference labels were produced. Neither offers an estimator that accounts for this.

Baselines

The naive reference is the scoring rule every leaderboard uses: aggregate votes by majority vote and report the model's agreement with the result. It is the standard to displace. On Bird its label error is 24.07% according to Spectral Methods meet EM, so its accuracy estimates should be visibly biased.

The strongest existing method scores models against Opt-D&S labels, or better against Opt-D&S posterior probabilities, which gives a plug-in expected accuracy. The file reports a label error of 10.09% on Bird for Opt-D&S. We found no reported result on Lacuna for using Opt-D&S or any crowd model to estimate a model's accuracy, so the comparison on our metric will be established here. We will also run full EM on our difficulty-binned model to convergence, restarted from the Opt-D&S seed, and report its plug-in accuracy. This isolates whether the benefit comes from the richer model or from the targeted step.

The key ablation applies our one-step correction under the homogeneous model with no difficulty bins, which isolates the difficulty ingredient. All methods receive identical votes, identical cross-fitting splits, the same seed, and the same number of difficulty bins, chosen once on synthetic data and frozen. Beating the strongest baseline means lower absolute error against gold-computed accuracy on at least four of the five real datasets, and confidence-interval coverage within a few points of 95% where the plug-in undercovers.

Why not just…?

Why not just score models against Opt-D&S posteriors? Under correct homogeneous specification this is already efficient, and on low-heterogeneity datasets we expect it to tie our method. It becomes a weakness only when items differ in difficulty and the posterior is miscalibrated on hard items. The synthetic difficulty sweep described below settles when, if ever, the gap opens. If no gap appears on real data, we will report that as a finding.

Why not just run the difficulty-aware model's EM to convergence and use the plug-in? Per-item difficulty creates too many parameters, and the binned version is non-convex. The Opt-D&S analysis shows that EM from weak starts settles at worse optima. Converged EM from the same seed is included as a baseline. If it matches our estimator's error and coverage at comparable cost, the one-step contribution reduces to cheaper computation, and we will state that.

Why not just have experts label a small random subset and estimate accuracy there? This is the most serious alternative. It is unbiased by construction and needs no worker model. Its variance is roughly $\psi(1-\psi)/k$ for $k$ gold labels, a standard binomial fact. We will measure the break-even $k$ at which the gold-subset estimate matches ours, and also test a combination that uses the gold subset to calibrate the correction. If the break-even is only a few dozen items, the gold subset is the right recommendation and our method is only useful when no expert labels exist.

Proposed Experiments

The five real crowd datasets used by Opt-D&S (Bird, RTE, TREC, Dog, Web) carry expert gold labels, so the true accuracy of any labeler is computable. We use leave-one-worker-out evaluation. Each held-out worker, and each of several off-the-shelf classifiers on the text and image datasets, is treated as a model to be scored. Its accuracy is estimated from the remaining workers' votes by every method above and compared with its gold accuracy.

The metrics are absolute error of the estimated accuracy, coverage and width of 95% intervals, and the fraction of pairwise orderings that each method gets wrong relative to gold, which counts leaderboard flips. To probe the small-sample regime, we subsample items and workers and record how often the single step overshoots, alongside the damped and two-step variants.

A controlled sweep on the synthetic generalized Dawid-Skene setting of Reliable Crowdsourcing under the Generalized Dawid-Skene Model varies the ratio $\lambda/\lambda_{\min}$ from 1 upward. The decisive check is a crossing pattern. At a ratio of 1 our method should tie the posterior plug-in. The plug-in's bias and undercoverage should grow with the ratio, while ours should stay near nominal coverage. If the real datasets also show larger gains where gold-measured difficulty is more heterogeneous, the claim holds.

A stretch test on the cartoon-caption ratings (yguooo/newyorker_caption_ranking) has no gold labels. There we check only internal consistency, namely whether estimates are stable across disjoint rater halves, and we make no accuracy claim.