Best-worst scaling is an annotation method where a judge sees a small set of items, usually four, and picks only the best and the worst. Counting how often each item is picked, across many overlapping sets, turns these choices into a continuous score. The method has become a reliable way to score fuzzy qualities such as sentiment intensity, bias severity, or the factual consistency of a summary. Increasingly, the judge is a large language model rather than a person. The counting formula behind the scores quietly assumes a balanced design, where every item appears equally often alongside comparable company.
Real annotation campaigns break that assumption. API calls fail. Rate limits cut runs short. LLM judges refuse or return unparseable answers on some sets. Budget-saving schemes deliberately spend more comparisons on the close calls at the top of a ranking. We argue that each of these turns a best-worst leaderboard into a measurement whose ordering depends on which sets happened to survive. That is a new and unexamined source of the benchmarking inconsistency documented across machine learning. We propose to measure this effect on existing benchmarks and to replace counting with a choice-model estimator that accounts for missing sets and reports uncertainty. We then test whether adaptive set allocation can recover full-design reliability at a fraction of the cost.
Core Claim
The standard score is
$$S(x)=\frac{\text{best}(x)-\text{worst}(x)}{n(x)},$$
where $\text{best}(x)$ and $\text{worst}(x)$ count how often item $x$ was chosen at each extreme, and $n(x)$ counts how many sets it appeared in.
The expected value of $S(x)$ depends not only on $x$'s true strength but also on the strength of the items it was grouped with. Under a balanced random design those companions average out. When sets are dropped unevenly, they do not average out, and two items of equal quality can receive different scores.
A best-worst choice model removes this dependence. It is the standard multinomial logit from discrete choice, where each item $i$ has a latent strength $\theta_i$. The probability that, within set $T$, item $i$ is picked best and item $j$ worst is
$$P(i,j\mid T)=\frac{\exp(\theta_i-\theta_j)}{\sum_{a\neq b\in T}\exp(\theta_a-\theta_b)}.$$
The model conditions on which items were actually shown together. It therefore stays consistent when sets are lost at random, and also when the design adapts to past answers, because such losses are "ignorable" for a likelihood-based estimator. Counting does not have this property.
The part neither estimator handles is missingness tied to the items themselves. An example is an LLM judge that fails more often on sets containing hallucinated or truncated-source summaries. For that case we add a selection model that uses observable item features, plus a sensitivity analysis for the residual.
Our claim is concrete and falsifiable. Under realistic missingness, counting-based best-worst leaderboards will reorder systems that the full design separates. A missingness-aware choice model with bootstrap intervals will restore the full-design ordering. Adaptive allocation will reach the same reliability with fewer sets.
How This Relates to Prior Work
The reliability case for best-worst scaling motivates this work, and the counting estimator it relies on is the object we test.
Best-Worst Scaling More Reliable than Rating Scales annotated 3,207 sentiment terms. It reported split-half reliability (the correlation between scores computed from two random halves of the annotations) of 0.98 for best-worst scaling versus 0.95 for a 9-point rating scale. It also showed that best-worst scaling reached 0.95 with only about $3N$ annotations. Big BiRD reached 0.937 on bigram relatedness with the same counting procedure.
Both studies used carefully balanced designs, so neither says what happens when sets go missing. The research file itself flags this as an open question.
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries moved best-worst scaling into system-level evaluation. It ranked BART, PEGASUS, ProphetNet and BERTSUM on CNN/DailyMail and XSum, and found that 5-point Likert scores bunched at 4 and 5. It also reports a counterpoint we must face: a 10-point scale reached 94.85% system-level split-half reliability on XSum.
"You are an expert annotator" showed that LLM judges given 4-item sets produce better emotion-intensity annotations than LLM direct ratings. It is the template for the LLM-judge campaigns we study. Its inputs were clean, and it did not examine failed or refused judgments.
Multiagent Evaluation under Incomplete Information supplies two methodological ideas that we adapt from pairwise game matchups to multi-item choice sets:
- an adaptive sampler (ResponseGraphUCB) that concentrates effort on uncertain matchups;
- propagation of sampling uncertainty into leaderboard intervals.
That paper also constrains our design. Its stopping rule fails when two players are exactly tied, which warns us that adaptive allocation must handle near-ties gracefully.
Two papers supply the application regime where missingness is non-random by construction:
- Roux-lette at "Discharge Me!" documents LLM hallucination when hospital records lack notes, labs, and vital signs.
- Can Tool-augmented LLMs be Aware of Incomplete Conditions? builds the ICT benchmark by stripping parameters or swapping tools, and scores only binary abstention.
Neither scales the severity of failures. Neither considers that the judges scoring such outputs may themselves fail selectively.
Research Program
Characterization. We start with best-worst campaigns where the full design is available. For these we can compute reference scores and then delete sets under controlled mechanisms:
- uniform dropout;
- dropout that depends on observable item features, such as length or whether the source input was truncated;
- dropout that depends on the latent score, such as judge refusals concentrated on the worst items;
- adaptive designs that oversample items near the top of the ranking.
For each mechanism we measure:
- how far counting scores drift from the reference;
- how often adjacent systems swap places;
- whether the drift tracks the average strength of each item's companions, as the bias argument predicts.
Estimator. We fit the best-worst logit by maximum likelihood with a weak Gaussian prior on the strengths. For feature-dependent dropout we add inverse-probability weights, estimated from a logistic model of which sets fail. We produce intervals by bootstrapping over sets. For dropout that depends on latent scores we report a bounded sensitivity analysis rather than claiming identification. Here "identification" means recovering the true scores from the observed data alone, which is impossible when the failures depend on those unobserved scores.
Allocation. We adapt the ResponseGraphUCB idea to choice sets. New sets are drawn to include items whose intervals overlap at rank boundaries that matter for the leaderboard. This allocation is valid only under the likelihood estimator, so the program tests the two pieces together and also apart.
Application. The application is a new LLM-judged factual-consistency leaderboard on summaries generated from inputs with controlled truncation. Truncation makes output quality and judge failure correlate, which produces the hard case naturally rather than by simulation.
Baselines
The naive reference point is counting on whatever sets survived. That is what an analyst does today when a campaign comes back incomplete. Its full-design version is the target. On sentiment, full-design counting reached split-half reliability of 0.98 (Best-Worst Scaling More Reliable than Rating Scales). Any method under missingness is judged by how close it gets to the full-design scores and their reliability.
A second, cheap baseline is counting with re-querying. Failed sets are retried until the original budget is spent, with the extra calls charged to the method.
The strongest existing competitor is the best-worst multinomial logit itself, fitted without any missingness model. It is standard in the discrete-choice literature, although none of the papers in the research file report it on these NLP benchmarks. No reported result was found on Lacuna for a logit-fitted best-worst estimator on AIT, the sentiment terms, or the summarization data.
We also include Bradley-Terry fitted to the pairwise preferences implied by each set. This is the natural import from pairwise leaderboards such as ChatEval.
The key ablations isolate the proposal's ingredients. The first is our estimator with the missingness weights removed, which separates the contribution of the selection model from that of the choice model. The second is adaptive allocation paired with counting, which tests the prediction that adaptivity helps only with a likelihood estimator.
For fair comparison, every estimator sees exactly the same surviving sets, with the same LLM judge, prompt, and decoding settings fixed in advance. Each method gets the same small tuning budget, chosen on a held-out development split, for its single regularization strength.
Beating the logit means one of two things. Under feature-dependent dropout, our estimator must achieve significantly lower error against the reference ordering and fewer top-$k$ swaps. Under adaptive designs, it must achieve higher reliability at a matched number of sets. If plain logit already matches our estimator in every setting, the selection model is unnecessary and we will report that.
Why not just…?
Why not just fit a standard choice model and call it solved? This is the most serious threat. Under random or adaptive dropout, the logit alone should be enough, and we expect our experiments to show exactly that. The contribution in those settings is then empirical. We show that the counting default reorders leaderboards, and we show that switching estimators fixes it. The new estimator matters only where missingness depends on item features. The truncated-source summarization campaign tests whether that case occurs often enough in practice to matter. If the plain logit's error there is not reduced by the weights, we drop the selection model.
Why not just re-query failed sets until the design is balanced? Retrying costs money. It also does not help when failures are systematic: a judge that refuses a set with a fabricated clinical claim will tend to refuse it again. The re-query baseline is run with its extra calls counted. We report how many retries it needs and whether a stubborn residual of failures remains concentrated on low-quality items.
Why not just use a 10-point rating scale, where each item is rated independently? This is a real threat. Independent ratings do not couple items through shared sets, so a lost rating removes only one item rather than biasing others. The 94.85% split-half reliability on XSum (Investigating Crowdsourcing Protocols) shows the 10-point scale can be strong. Against it, the summarization study found ceiling compression on CNN/DailyMail for 5-point scales, and the expert-annotator study found LLM direct ratings worse than best-worst sets (you are an expert annotator). We therefore run 10-point LLM ratings under the same missingness mechanisms and an equal judge-call budget. If they match our estimator's reliability, the practical recommendation becomes ratings, not best-worst scaling.
Why not just use pairwise comparisons? Pairwise designs need many more judgments. The sentiment study notes that paired comparisons scale quadratically with the number of items. The expert-annotator study found them less cost-effective than best-worst sets for LLM judges. We include Bradley-Terry at matched budget so this claim is tested rather than assumed.
Proposed Experiments
Experiment 1: controlled missingness on AIT. We run an LLM best-worst campaign on Affect-in-Tweets, which has human gold intensities, at $2N$ and $12N$ sets, following you are an expert annotator. That paper reports regressors trained on LLM best-worst labels scoring 73.5 at $2N$ and 74.8 at $12N$, against 78.3 for human labels. We delete sets under the four mechanisms described above. For each mechanism and estimator we report:
- Spearman correlation to the full-design scores and to human gold;
- split-half reliability;
- the downstream regressor score, using the same RoBERTa setup.
Experiment 2: system leaderboards with truncated sources. We generate summaries on CNN/DailyMail and XSum with BART, PEGASUS, ProphetNet and BERTSUM. Each source is given at four levels of truncation, creating 16 model-by-condition systems. An LLM judge then runs the best-worst factual-consistency protocol of Investigating Crowdsourcing Protocols. Failures here are natural, not simulated. We keep a fully re-queried design, obtained at high cost, as the reference. We also obtain human best-worst judgments on a subset to check that the reference agrees with people.
As a smaller replication in a domain where failures carry high stakes, we apply the same protocol to severity scaling of ICT failures (ddehun/ICT). That benchmark's reported numbers are binary F1: Gemma-3-4B scores 20.90% zero-shot and 68.57% with Structured Verification, against roughly 81% for humans.
Decisive checks. The proposal is supported if all of the following hold:
- counting-score error correlates with average companion strength under uneven dropout;
- counting paired with adaptive allocation is worse than counting with uniform allocation, while the logit paired with adaptive allocation is better;
- on the truncated-source leaderboard, counting produces at least one rank swap among systems that the reference separates with non-overlapping intervals, and our estimator removes it;
- bootstrap intervals cover the reference scores at close to their nominal rate.
The proposal is refuted, and we will say so, if two things are true together. First, plain logit closes the gap in every setting. Second, the equal-budget 10-point ratings match it. In that case the honest conclusion is that best-worst campaigns with missing sets need only a change of estimator, or should not be run at all.
References
- Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation
- Big BiRD: A Large, Fine-Grained, Bigram Relatedness Dataset for Examining Semantic Composition
- Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries
- "You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling
- Multiagent Evaluation under Incomplete Information
- Roux-lette at “Discharge Me!”: Reducing EHR Chart Burden with a Simple, Scalable, Clinician-Driven AI Approach
- Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?
- ChatEval: A Tool for Chatbot Evaluation