How Much of a Corpus-Filtering Win Is Luck? Bootstrapping the Filter, Not Just the Test Set

Machine translation systems are trained on web-crawled sentence pairs, and much of that data is not actually translation. In one audit, only about 23% of a raw crawl was judged "okay." Researchers therefore run filters, which score each pair and keep the top slice. Filters are compared by training a translation model on each kept slice and reading off a test score such as BLEU. That test score has at least three sources of randomness. The first is which test sentences were used. The second is the random seed of the translation model. The third is usually ignored: the filter is itself a fitted model. Its helper translation models, its pseudo-labels, and its cutoff all depend on a small sample. A different draw of that sample would keep a different set of sentences. Recent work on leaderboard statistics fixes only the first source. We propose to bootstrap the filter's own training inputs as well, and to measure how much this changes which filter is declared best.

The core claim is falsifiable. For low- and mid-resource filtering benchmarks, confidence sets that resample only test sentences will fail to contain the truly best filtering configuration much more often than their nominal 95%. A two-level bootstrap that also resamples the filter's inputs will restore that coverage. The same measurements also yield a cheap diagnostic. How much two bootstrap-resampled filters disagree about which sentences to keep, measured by overlap of the kept sets, should predict how noisy the resulting translation scores are, without retraining any translation model.

What prior work does and leaves open

On the statistics side, The Leaderboard Lottery introduces paired-bootstrap Multiple Comparisons with the Best (MCB). The method resamples test prompts with replacement, scores all models on each resample, applies a Holm correction that controls the chance of any false exclusion, and returns a set of models that contains the true best with at least 95% probability. The paper motivates our work: on GPQA the top-ranked model was the true best 0% of the time, and 0–58% of the time across tasks. It also constrains our design, since it notes that dependent items need cluster resampling (companion version). It treats each model as a fixed object and resamples only the evaluation data. That treatment fits pre-trained language models on a leaderboard. It does not fit a filtering "submission," which is a random function of the filter's small seed data. Clinical work on correcting for overly optimistic scores uses a different loop, re-fitting the whole pipeline on each resample and comparing each refit to the original sample. That loop supplies our second level. Its main drawback is re-fitting cost, which we manage by resampling the cheap filter rather than the expensive translation model.

On the corpus side, the most exposed filter is Dual Conditional Cross-Entropy Filtering. It scores each pair by how well two small translation models, one per direction and both trained on clean data, agree the pair is a translation. It won three of four WMT 2018 subtasks. The research file notes that the paper never quantified how sensitive this score is to the helper models' training sample. OpusFilter trains a logistic-regression filter on pseudo-labels taken from the top and bottom 5% of unsupervised feature scores. It reports a 0.67 BLEU gain on Finnish-English as a point estimate, with no check on how stable those arbitrary percentile boundaries are. Neither shared-task report quantifies any of this. WMT 2018 and WMT 2020 crown winners by BLEU point differences over dozens of submissions. In 2020 the margins were a few BLEU on Pashto and Khmer, where filter seed data is scarcest.

The closest existing idea would apply MCB directly to WMT filtering results. We treat that as our main baseline, not our contribution. What is new here is treating the filter as a fitted, resampleable object and measuring how much this changes both coverage and the identity of the plausible winners.

Research Program

We define a filtering configuration as a triple: a scorer, its seed data, and a keep-size. The scorers are Dual-CE, the OpusFilter classifier, and a fixed pretrained scorer (a multilingual sentence-embedding similarity). The keep-sizes follow the shared-task budgets.

The first level resamples the scorer's inputs with replacement. For Dual-CE, these inputs are the clean sentences used to train the two helper models. For OpusFilter, they are the pairs whose feature distribution sets the percentile pseudo-labels. For the pretrained scorer, only the sample used to calibrate the cutoff is resampled.

Each resample yields a kept subset. Across B resamples this gives every sentence pair a selection probability, which is the fraction of resamples in which it survives. It also gives every configuration a stability score, which is the average overlap (Jaccard) between kept subsets from different resamples.

The second level trains small translation models on a sample of these kept subsets, crossed with random seeds. We then resample FLORES test sentences in document blocks, following the Leaderboard Lottery's warning about dependent items, and compute an MCB confidence set whose bootstrap distribution pools all three sources of variation.

The pretrained scorer gives us a built-in contrast. It has no helper training data, so our account predicts that its rankings will be more stable than Dual-CE's. If Dual-CE turns out to be just as stable, the premise is weakened.

A variance decomposition, a random-effects model over test resample, seed, and filter resample, reports what share of BLEU and chrF variance each source contributes. It will be computed separately for the low-resource WMT 2020 Pashto-English (11.6M words) and Khmer-English (58.3M words) crawls, and for a German-English WMT 2018 10M-word setting.

Baselines

The naive reference point is current shared-task practice: rank configurations by single-run point BLEU and declare the top one the winner. We report how often that winner equals the ground-truth best.

The second baseline adds marginal 95% confidence intervals around each configuration's BLEU. It is included because it is what most filtering papers would report. The Leaderboard Lottery already shows it fails to cover the selection event.

The strongest existing competitor is paired-bootstrap MCB from The Leaderboard Lottery. It is run on exactly the same trained systems and FLORES test documents, with the same document-block resampling and the same number of bootstrap draws, differing only in that it resamples test items alone. Its reported result is at least 95% coverage of the true best on all Open LLM Leaderboard tasks, with sets of 8 to 35 models. No reported result was found on Lacuna for MCB applied to translation or corpus filtering.

For downstream quality anchors, the strongest filtering systems in the file reached 28.6 BLEU on the German-English NMT 10M subtask (WMT 2018). In WMT 2020, the best Pashto system reached 12.9 BLEU against a LASER baseline of 10.3, and the best Khmer system reached 15.5 against 12.9. Our reimplementations must land in a comparable range for the variance findings to be relevant.

Two ablations isolate the key ingredient. The first, "seed-plus-test," retrains translation models across seeds but always on the single original filter output. The second, "filter-only," resamples filter inputs but fixes seeds. Our method beats test-only MCB if MCB's empirical coverage of the ground-truth best falls clearly below 95% while ours stays at or above it. It beats the seed-plus-test ablation if that ablation also under-covers, which would show that filter resampling in particular matters. We also report set size, because a method that achieves coverage only by including every configuration is uninformative, the failure the Leaderboard Lottery reports on Chatbot Arena.

Why not just…?

Why not just run paired-bootstrap MCB on the test set, as the Leaderboard Lottery does? This is the real threat. If filter-induced variance is small relative to test-set variance, MCB will already cover the true best, and our second level adds only cost. The variance decomposition settles the question directly. If filter resampling accounts for a negligible share of variance and MCB coverage stays at 95% or above, we will report that MCB suffices for corpus filtering. That is a useful negative result for shared-task organizers. We expect otherwise in Pashto and Khmer, where Dual-CE helper models are trained on very little clean data.

Why not just retrain the translation model with several random seeds? Seed variation is cheaper to add and is what careful papers already do. However, every seed trains on the same kept subset, so seeds cannot reveal that a slightly different helper sample would have kept different sentences. The seed-plus-test ablation measures exactly how much coverage seeds alone recover.

Why not just bag the scorer, averaging many resampled helper models into one stable filter, and then use ordinary MCB? This may well be the right practical fix, and we will test it as an extra configuration. If a bagged Dual-CE has near-perfect subset overlap and its rankings become stable, then our procedure mainly serves to diagnose which filters need bagging. Bagging still cannot establish whether it was needed or whether it worked without the resampling measurements proposed here. We would reframe the practical recommendation around this outcome rather than argue against it.

Why not just use k-fold cross-validation on the filter's seed data? For Dual-CE, each fold would train helper models on a smaller share of already scarce clean data, confounding instability with a systematic weakening of the helpers. We include 5-fold resampling as a sensitivity check on the stability estimates only.

Decisive Experiments

Ground truth is expensive, so the study runs in the low-resource WMT 2020 setting, where small translation models train quickly. There are roughly eight configurations: three scorers crossed with keep-sizes, plus the bagged Dual-CE. Each gets about twenty filter-resample-by-seed replicates. The "true best" configuration is the one with the highest mean chrF and BLEU over all replicates, scored on the pooled FLORES development and test sets. Coverage is then estimated by repeatedly drawing one "submission" per configuration together with a held-out test subsample, running each procedure, and checking whether its set contains the true best.

Three outcomes would be decisive. The first is coverage and set size for test-only MCB, the two ablations, and the full method. The second is whether Dual-CE shows lower subset overlap and wider BLEU spread than the pretrained scorer, as the account predicts. The third is the rank correlation, across configurations, between subset overlap, which is computable without any translation training, and downstream BLEU variance. A strong correlation would give filtering practitioners a cheap stability check to run before committing compute.

A retrospective re-analysis of the published WMT 2018 and 2020 rankings, using test-only MCB, gives a lower bound on how many announced winners are statistically distinguishable from the runners-up.