How Many Test Items Decide a Data-Curation Choice?

Teams that build training corpora make many small choices, such as which quality filter to apply, which deduplication threshold to use, or which language-identification cutoff to trust. They usually settle each choice with an ablation: train a small model on each data variant, score both on downstream benchmarks, and keep the variant with the higher average accuracy. We propose to audit these decisions with a simple question: what is the smallest share of benchmark items that would have to be removed for the decision to flip? If that share is tiny and falls below the swing caused by changing the training random seed, the ablation did not measure the data. It measured a handful of test items and luck.

The note has two parts. The first is a diagnostic, the flip fraction, measured across a population of real curation decisions. The second is a decision rule that is harder to fool. It compares the two data variants item by item on continuous per-item scores (for example, how much probability the model puts on the correct answer, rather than a 0/1 correct flag). It weights those comparisons against seed-to-seed noise. It is judged by one external test: does the small-scale decision predict the decision you would reach at a larger scale? Nothing in this setup depends on trusting any single metric.

Core Claim

Two findings motivate this work. The first is that aggregate evaluation numbers can be driven by a few extreme cases. Tangled up in BLEU re-analyzes the WMT 2019 machine translation metrics task and shows that one catastrophically bad system inflates the correlation between metric and human scores. In one language pair, removing the outlier systems drops BLEU's correlation from 0.85 to 0.58, and a 1–2 BLEU gain agrees with human judgment only about half the time. Evaluating the Examiner finds that removing 20 of 1,225 document pairs cuts the QuestEval metric's correlation from 0.33 to 0.11. Better than Average shows that ranking systems by instance-level paired wins (a Bradley–Terry model, which turns pairwise win rates into a strength score per system) instead of by mean score changes the top-ranked system in 30–40% of 296 leaderboards.

The second finding is that data curation is validated in exactly this way. FineWeb justifies its individual filters by training 1.7B-parameter models on each variant and comparing aggregate MMLU and ARC accuracy. HPLT validates its cleaning pipeline by training translation models on 18 language pairs.

All three outlier papers study evaluation after the fact: metrics or finished systems on fixed test sets. None of them asks whether the decisions a curation pipeline makes, which later shape trillions of tokens, are stable to a few test items.

Our claim is sharper than "use robust statistics". Suppose the per-item score is binary correctness and only two data variants are compared. Then the mean accuracy difference and the paired win rate always agree in sign. Both depend only on the items where exactly one model is right, so pairing alone cannot change a two-way accuracy decision. A proposal that simply swaps the mean for Bradley–Terry would therefore be empty in the most common setting. Robust aggregation can only matter in three places:

  1. Per-item scores are continuous, such as answer log-probability margins or segment-level chrF.
  2. Several benchmarks with different scales are averaged together.
  3. More than two variants are ranked at once.

Separately, the flip fraction can expose fragility even in the binary case. We therefore make two predictions. First, a meaningful share of published-style curation decisions have flip fractions below the level explained by seed noise. Second, a seed-aware paired rule on continuous margins predicts the larger-scale decision more often than mean accuracy does, at equal compute.

Research Program

The diagnostic is defined as follows. For data variants $A$ and $B$, let $d_i$ be the per-item score difference on evaluation item $i$, where $i = 1, \dots, n$. The mean-based decision is the sign of $\bar d$, the average of the $d_i$. The flip fraction $F$ is the smallest fraction of items whose removal reverses that sign. Because $\bar d$ is a mean, removing the largest same-sign differences first gives the exact minimum, so $F$ costs one sort to compute. We compare $F$ against a seed-noise reference: the share of the aggregate difference that disappears when the same comparison is rerun with different training seeds. A decision is fragile when $F$ is smaller than the fraction of items that seed noise alone moves across the decision boundary.

The same removal can be applied to each candidate decision rule. We also characterize the items in each flip set: whether they are saturated or duplicated in the sense of SMART filtering, which prunes easy, memorized, and near-duplicate benchmark items; whether they come from particular subjects; and, for localized benchmarks, whether they carry translation artifacts.

The decision rule works at the level of individual items. On each item it takes the continuous paired difference, averaged over seeds. It divides that difference by the item's seed-to-seed spread, so items that flip across seeds count less. It then clips each item's contribution at a fixed bound, borrowing the idea behind Clipped Density and Coverage, which clips each sample's contribution and bounds neighborhood sizes so that outliers cannot dominate a generative-model metric. Across several benchmarks, the rule fits a Bradley–Terry model over all variant pairs and all items, instead of averaging per-benchmark accuracies.

This rule also answers a limitation that Don't Sweat the Small Stuff acknowledges. That paper's Pairwise Difference Pearson meta-evaluates metrics by correlating within-segment score differences, but it still inherits Pearson's sensitivity to extreme differences. Clipping and seed-normalizing each item removes that sensitivity. The rule also differs from DUET, which optimizes data mixtures from scalar downstream feedback. We change what the feedback is, not the optimizer, and the rule could be plugged into DUET afterwards.

To judge rules we need a ground truth that is not itself a metric choice. We use scale transfer. We build a population of roughly 30 curation decisions, varying threshold and filter choices of the kind FineWeb ablates: terminal-punctuation ratio, deduplication granularity, quality-classifier cutoff, and language-ID threshold. Each decision is trained at a small scale, around 150M parameters and a few billion tokens, with several seeds. Each is also trained at a larger scale, around 1B parameters and tens of billions of tokens. The larger-scale outcome on a broad held-out suite, averaged over seeds, serves as the reference answer. Where both conventions give the same answer, we take it as settled. Where they disagree, we report those decisions separately rather than scoring them.

The second regime is parallel-data filtering for translation, modeled on HPLT's OpusCleaner pipeline, with small MarianNMT models trained per filter variant. Here per-segment chrF is continuous by construction. Sentence-level Aggregation already shows that averaging segment scores correlates with human MQM ratings far better than the corpus-level ratio (0.806 against 0.392 for chrF on WMT 2023), so we use segment-level aggregation as the starting point, not as a contribution.

Baselines

The naive reference is the current practice: the sign of the difference in mean accuracy, averaged across benchmarks, from a single seed. This is the FineWeb-style rule. The only reported numbers in the file for this rule are outcomes, not decision reliability. FineWeb reports that FineWeb-Edu raises 1.7B-model MMLU accuracy from 33% to 37% and ARC from 46% to 57% over base FineWeb. We found no reported result on Lacuna for how often small-scale curation decisions agree with larger-scale ones, so the naive rule's agreement rate is measured here for the first time. We will also compute flip fractions for the large-margin FineWeb-Edu comparison as a sanity anchor, where we expect robustness.

The strongest existing competitor is a multi-seed mean with a paired bootstrap confidence interval: several seeds per variant, a decision only when the interval excludes zero, and otherwise "no decision". This is the rigorous statistics answer, and it directly targets seed noise.

The strongest robust-aggregation competitor is instance-level Bradley–Terry as implemented in Better than Average, applied to binary correctness across benchmarks. That paper's simulations show it beats both mean and median aggregation when outliers are present.

All rules see the same trained checkpoints, the same evaluation items, and the same total number of training runs. The proposal gains no extra seeds that the bootstrap baseline lacks. Any tuned constants, such as the clipping bound or the Bradley–Terry prior, are fixed on a held-out subset of decisions, and the other rules receive the same tuning budget.

Two ablations isolate the key ingredient. One removes seed normalization and keeps continuous paired clipping. The other replaces continuous margins with binary correctness while keeping seed weighting. Beating the baselines means higher agreement with the larger-scale decision across the decision population, with a paired test over decisions. For the bootstrap baseline specifically, it means higher agreement at matched abstention rate, since abstaining trivially raises precision.

Why not just…?

Why not just take the median instead of the mean? Better than Average gives a constructed case where the system with the higher median loses on 80% of paired items. Its simulations show median aggregation losing rank agreement with true system strength when pairing variance is present, because the median ignores that both systems saw the same items. On binary accuracy the per-item median is also degenerate. We include median of continuous margins as a rule in the comparison anyway.

Why not just run more seeds and average? This is the real threat. If fragility is mostly seed noise, more seeds solve it and our item-level machinery adds little. The matched-compute comparison settles it. We split a fixed budget of training runs between more seeds per variant and the proposed rule with fewer seeds. If the multi-seed mean matches our agreement rate at equal runs, we will report that the problem is seed noise, not item outliers. The flip-set analysis will then still show which items to drop.

Why not just trim outliers by median absolute deviation, or prune the benchmark with SMART first? Trimming fixes the metric's number but, as Tangled up in BLEU shows, can leave competitive comparisons near chance. SMART sets its "easy item" threshold from the accuracies of a model suite and validates with a Pearson correlation of the kind these outlier papers criticize. It is still a cheap, plausible fix, so SMART-pruned benchmarks under the naive rule form an explicit arm. If that arm matches us, the lesson is "clean the benchmark", and we will say so.

Why not just use a paired significance test on accuracy? For two variants on binary accuracy, the paired sign is identical to the mean's sign, as noted above. A test adds confidence, not a different answer. Any gain must come from continuous margins, seed weighting, or multi-benchmark aggregation, and the ablations test each of those separately.

Proposed Experiments

The first decisive check is descriptive. It reports the distribution of flip fractions over the roughly 30 language-model decisions and the HPLT-style translation decisions, set against the seed-noise reference. The hypothesis fails if nearly all decisions have flip fractions well above seed noise.

The second check is the agreement rate between small-scale and large-scale decisions for each rule at matched compute, reported separately for decisions with small and large margins. We expect gains to concentrate in the small-margin decisions, the same regime where Tangled up in BLEU finds metrics near chance among competitive systems.

The third check is a composition stress test. We repeat the language-model decisions on machine-translated Italian MMLU from Teasing LLMs Adapted to Italian and on the Dutch retrieval benchmark BEIR-NL, whose back-translated version costs BM25 1.9 NDCG@10. We then ask whether flip sets are enriched for translation-damaged items, which would explain when localized benchmarks give unreliable curation feedback.

The work succeeds if fragile decisions are common and the seed-aware continuous paired rule improves scale-transfer agreement beyond both the multi-seed bootstrap and the SMART-pruned arms. It is informative either way, because each negative outcome names which simpler fix is enough.