Tipping-Point Fold Correlation for Benchmark Claims

When researchers compare models with cross-validation, the scores from different folds are not independent. Each fold's model is trained on mostly the same data as the others, so the models tend to make similar errors. Treating fold scores as independent makes confidence intervals too narrow and produces too many claimed wins. The standard fix is to assume a value for this correlation and inflate the variance accordingly. However, No Unbiased Estimator of the Variance of K-Fold Cross-Validation proved that the correlation cannot be estimated from a single dataset. Every corrected test therefore rests on a guess. This proposal stops pretending to know the correlation. For every claim of the form "A is practically better than B" or "A and B are practically tied", it reports the smallest fold correlation that would overturn that claim. A claim holds only if this tipping point lies above an empirically calibrated upper bound on the correlation for that kind of model and data size. Claims on a whole leaderboard are handled together, so the guarantee covers the full set of claims in a paper and not each pair separately.

The need is concrete. Evaluation practice has moved in two directions at once. One is toward many random splits, to escape the quirks of a single fixed split. The other is toward practical-equivalence margins, a "region of practical equivalence" (ROPE) of differences too small to matter, so that huge test sets stop certifying 0.06-point wins. The first creates more correlated scores. The second makes decisions depend sharply on interval width. Many comparisons are then made at once, so small miscalibration in each interval compounds into many false claims across the family. A single number that states how fragile each claim is to the unknowable correlation would make these three problems visible together, and it can be checked against ground truth.

Core Claim

Let $d_1,\dots,d_K$ be the per-fold score differences between two models, with mean $\bar d$ and sample variance $S^2$. Assume the fold differences share a common correlation $\rho$. Then the variance of $\bar d$ is

$$\operatorname{Var}(\bar d) = S^2\left(\frac{1}{K} + \frac{\rho}{1-\rho}\right),$$

where $K$ is the number of folds. Setting $\rho = n_2/(n_1+n_2)$, with $n_1$ training and $n_2$ test examples, recovers the corrected resampled variance of Inference for the Generalization Error. Fix a practical margin $\epsilon$ and an interval multiplier $t$. A superiority claim holds while $|\bar d| - t\sqrt{\operatorname{Var}(\bar d)} > \epsilon$. The tipping point $\rho^*$ is the value of $\rho$ at which this becomes an equality. It has a closed form:

$$\frac{\rho^*}{1-\rho^*} = \frac{(|\bar d|-\epsilon)^2}{t^2 S^2} - \frac{1}{K}.$$

Equivalence claims get a symmetric tipping point: the correlation at which the interval edge leaves $[-\epsilon, \epsilon]$.

The formula itself is elementary. The new content is in three places. First, an empirical calibration of how large the true effective correlation actually is, measured against ground truth rather than assumed. Second, the question of whether cheap, observable quantities predict that correlation, especially for unstable neural models, where Bengio and Grandvalet warn that the standard correction is too small. Third, a family-level version in which all models are scored on the same folds and the tipping point is taken over the whole set of claims.

Research Program

The first component is ground-truth measurement of the effective correlation. We follow the design Nadeau and Bengio used on Letter: many small, independent datasets of size $n$ are drawn from a large pool, and the full cross-validation comparison is run on each. The spread of $\bar d$ across draws gives the true $\operatorname{Var}(\bar d)$. The average of $S^2$ gives the within-run estimate. Solving the variance equation gives the true effective $\rho$. The pools come from the file's benchmarks: Adult, Covertype and HIGGS from the suite in Multiple-Comparison Testing for Cross-Validation Path Optimization Benchmarks, Letter, 20newsgroups, and OntoNotes sentences for POS tagging. The algorithm pairs range from stable to unstable: regularized logistic regression, Naive Bayes versus linear SVM, gradient-boosted trees, multilayer perceptrons, and a small fine-tuned transformer classifier on 20newsgroups. We vary $n$ and $K$ so that the dependence on $n_2/n_1$ is visible.

The second component is a predictor for an upper bound on $\rho$. The bound uses $n_2/n_1$ and an instability signal that a single run can observe: how often fold-trained models disagree when scored on the same points. This signal requires no extra training runs, which the approach in Uncertainty Matters does require. That paper estimates its correlation parameter from additional half-sized CV runs and a reference model. We fit the bound on some pools and test whether it still covers the true $\rho$ on held-out pools and held-out model families.

The third component is the family-level decision. All $M$ models share the same folds, so their fold scores are modeled jointly: a multivariate model whose fold-level correlation is fixed at a candidate $\rho$. From it we compute the probability that every declared superiority and every declared tie holds simultaneously, and the family tipping point is the smallest $\rho$ at which that probability drops below 0.95. This builds on the joint treatment of comparisons in A Bayesian nonparametric procedure for comparing algorithms, which respects the logical links between pairwise comparisons but takes dataset-level summaries as input and ignores within-dataset fold dependence. It also addresses what Is the Best Better? leaves open: that paper's hierarchical ROPE model over POS taggers treats folds as conditionally independent and summarizes each pair separately. The same computation also sweeps $\epsilon$, giving a two-dimensional robustness map over correlation and margin. We do not settle the subjective choice of $\epsilon$, but readers can see exactly how much a claim depends on it.

Baselines

The naive reference point is a hierarchical Bayesian ROPE analysis with independent folds, run pairwise without any family adjustment. This is the analysis style of Is the Best Better? and A Bayesian Hierarchical Model for Comparing Average F1 Scores. It is run on the same fold scores, with the same $\epsilon$ and the same 95% interval. This is the configuration the proposal predicts will produce too many false practical-superiority claims.

The strongest existing competitor is the Nadeau–Bengio correction applied inside a three-way ROPE decision, with Holm adjustment across pairs. It is strongest because it costs nothing extra and is the correction most papers already accept. On Letter, Inference for the Generalization Error reports that the corrected test "stayed very close to the nominal 10% significance level"; the file gives no exact figure. On the MOOC data, Statistical Approaches to the Model Comparison Task in Learning Analytics reports a corrected 10×10 CV p-value of 0.089 for accuracy and 0.016 for AUC. On the 11-dataset suite, Holm reduced four significant datasets to two. We run this competitor on identical fold scores, with identical $\epsilon$ and interval level. Beating it has a precise meaning: in the pool-subsampling ground truth, the tipping-point procedure must keep the family-wise rate of false practical-superiority claims at or below 5% in settings where the Nadeau–Bengio version exceeds it, and must lose no more than a stated fraction of true detections where the Nadeau–Bengio version is already calibrated.

Two conservative references bound the power side of the comparison. Dietterich's 5×2 CV and the conservative Z procedure are both run with the same total training budget where feasible. We Need to Talk About Reproducibility in NLP Model Comparison reports that its 3×2 blocked CV with mixture estimator raised NER signal-to-noise from 0.510 to 1.421, with reproducibility 0.940, so we include it as a split design with the tipping-point decision placed on top.

The key ablation replaces the calibrated instability-based bound with the fixed $n_2/n_1$ value. This isolates whether the empirical bound, and not the reporting format, is what changes decisions.

Why not just…?

Why not just use the Nadeau–Bengio correction and stop? This is the real threat. If the measured true $\rho$ sits at or below $n_2/(n_1+n_2)$ for every model family we test, then the tipping point becomes a reporting convenience and not a correction, and we will say so. The warning in No Unbiased Estimator of the Variance of K-Fold Cross-Validation that unstable learners break the heuristic gives reason to expect a gap for neural models. The ground-truth experiment is designed to settle this directly, model family by model family.

Why not just use frequentist equivalence tests with Holm? Two one-sided tests can in fact accept practical equivalence, contrary to the common claim that only Bayesian ROPE can. The Bayesian machinery is not what matters here. The tipping-point computation applies to intervals of either kind, and we run both. The problem both share is that their interval width depends on the assumed $\rho$. The 0.5-point floor in Multiple-Comparison Testing for Cross-Validation Path Optimization Benchmarks has the same dependence.

Why not just run 5×2 CV, or many independent splits? 5×2 CV controls error but loses power and trains on half the data, as both Nadeau–Bengio and Gardner–Brooks report. Repeated random splits of the kind used in Pitfalls of Graph Neural Network Evaluation and We Need to Talk about Standard Splits draw from the same finite pool. They therefore reintroduce the overlap correlation, and they are too expensive for large models. Our power comparison under a fixed budget quantifies this trade-off.

Proposed Experiments

The decisive experiment is the pool-subsampling study, with at least 500 independent draws per setting. For each method it reports the true effective $\rho$ by model family and $n_2/n_1$, the family-wise false-claim rate, and the detection rate. The result that would sink the proposal is a Nadeau–Bengio value that is already calibrated for neural models. The result that would confirm it is a true $\rho$ above the heuristic for unstable learners, together with an instability-based bound that covers it on held-out pools.

The second experiment re-analyzes published claims using fold scores regenerated under the same protocols. The targets are the Adult and Covertype failures and the 0.06-point HIGGS difference from the optimization suite; the Stanford-versus-LAPOS and Flair-versus-NLP4J comparisons, for which Szymański and Gorman report an 80% chance that Flair beats NLP4J; and the MLP-versus-tree comparison on the MOOC data. For each we report its tipping point next to the calibrated bound.

The third experiment checks the family-level procedure on the 15-model MMLU setting of Quantifying Ranking Uncertainty in LLM Benchmarks. There the resampling unit is subjects rather than training folds. This tests whether the same family tipping point, computed over correlated subject slices, changes the rank intervals of three to five positions that the paper reports.