How Much Did Reusing the Test Set Inflate This Result? A Disclosure Rule for Small Private Benchmarks

Many results in medicine, education and enterprise software are reported on datasets nobody else can see, and these datasets are often tiny. The pneumoconiosis detector in Deep Ensemble Learning for the Automatic Detection of Pneumoconiosis in Coal Worker's Chest X-ray Radiography reports 91.50% accuracy on 153 private radiographs, reached after three rounds of ensembling schemes over five architectures. A reviewer cannot rerun the experiment. The question that matters is one the reviewer also cannot answer: how much of that number comes from the authors trying many things against the same few examples and keeping what worked? Theory for adaptive data analysis, meaning analysis in which each new question depends on earlier answers from the same data, gives a pessimistic answer. With $n$ examples, efficient methods can only answer about $n^2$ adaptive queries safely. Empirical work on Kaggle gives a reassuring one: public and private leaderboard scores track each other almost perfectly. Neither answer applies to a 153-example private cohort.

We propose to measure and model the gap between these two views, and to turn the result into a disclosure that private-data authors can publish without releasing data. The central idea is that the inflation caused by test-set reuse depends mostly on an effective number of distinct evaluations. That number is computable from how correlated the evaluated models' errors are. It is usually far smaller than the raw submission count, and it enters only logarithmically unless the analyst actively feeds holdout noise back into model design. If this holds, a data holder can release two numbers, $n$ and the effective evaluation count, and readers can discount a private claim by a calibrated amount.

Core Claim

Let $\hat{a}$ be the reported accuracy of the model an analyst selected on a reused holdout of size $n$. Let $\sigma_n=\sqrt{p(1-p)/n}$ be the sampling noise of an accuracy near $p$. Let $k_{\text{eff}}$ be the effective number of independent evaluations, computed as the participation ratio of the eigenvalues of the correlation matrix of per-example losses across all evaluated models. We hypothesize that for realistic development loops the expected inflation obeys a selection law:

$$\mathbb{E}[\hat{a}-a] \approx \sigma_n\sqrt{2\ln k_{\text{eff}}},$$

where $a$ is the true accuracy. This form is the expected maximum of $k_{\text{eff}}$ roughly independent Gaussian noise terms. It sits far below the worst-case adaptive scaling of order $\sqrt{k/n}$ in raw count $k$ that follows from Interactive Fingerprinting Codes and the Hardness of Preventing False Discovery. It also departs from that scaling only when a measurable feedback signal is present: later models that correlate with the holdout-specific residuals of earlier ones.

The claim is falsifiable in both directions. If inflation in real loops exceeds the selection law, practical adaptivity is closer to the fingerprinting adversary than A Meta-Analysis of Overfitting in Machine Learning suggests. If it matches the law, the law explains that paper's observations: near-perfect $y=x$ public/private agreement in 33 of 34 accuracy competitions, an effect size below 1% among the top 10% of teams, and shake-ups confined to test sets under roughly 100–200 examples. At $n=10{,}000$, $\sigma_n$ is under half a point, so even large $k_{\text{eff}}$ yields sub-1% inflation. At $n=153$, the same formula yields several points.

Relation to Prior Work

Two lines of work motivate this proposal. The hardness results show that the worst case is real and cannot be avoided: Preventing False Discovery in Interactive Data Analysis is Hard, Interactive Fingerprinting Codes and the Hardness of Preventing False Discovery, Fingerprinting Codes Meet Geometry and Adaptive Data Analysis in a Balanced Adversarial Model. Roelofs et al. show that the worst case rarely appears in practice, but they offer no mechanism for why. We do not contest either result. We aim to find the observable quantity that tells a given workflow which regime it is in.

The method comes from Generalization in the Face of Adaptivity: A Bayesian Perspective. Its Covariance Lemma writes adaptive overfitting as the covariance between a query and the information leaked about the sample. Our feedback term is an empirical estimate of that covariance, and our $k_{\text{eff}}$ is its non-adaptive, correlated-multiplicity part. That paper proves variance-dependent bounds for worst-case analysts. We instead estimate the realized quantity for real analysts.

The closest competitors are the protective mechanisms. Generalization in Adaptive Data Analysis and Holdout Reuse introduced Thresholdout, which answers from training data unless the holdout disagrees and then releases a noised holdout score. It was tested only on synthetic Freedman's-paradox noise. The Ladder releases only a running best score, with error growing in $\log k$. Both are prospective: they must mediate every query before it happens. Neither can say anything about a result already published from a private cohort. Our contribution is retrospective and needs no change to how analysts work, only logging of per-example losses by the data holder. Our small-cohort experiments also test these mechanisms at $n\approx150$, which no paper in our file has done. Adaptive Data Analysis for Growing Data constrains the design: it shows that naive splitting wastes samples linearly in $k$, which rules out "one fresh split per experiment" at clinical sizes.

Research Program

The first component is the estimator. The data holder computes a loss vector over the holdout for each evaluated model, which is private-compatible because it never leaves the holder. From these vectors it computes $k_{\text{eff}}$ and a feedback statistic: the partial correlation between each new model's holdout losses and the holdout residuals the analyst was shown for previous models, controlling for the same correlation on a small shadow split the analyst never saw. The released disclosure is $n$, $k_{\text{eff}}$, the feedback statistic, and the implied discount. These are aggregates of evaluation noise, not of patient data.

The second component is validation of the law in controlled development loops. Here ground truth is available because we hold a large reserve set that the loop never touches.

The third component is observational validation on MetaKaggle. Only per-submission public and private scores exist there, so we can only approximate $k_{\text{eff}}$, using the correlation of public-minus-private residuals across a team's submissions. We treat this as a coarse check rather than the main evidence.

Baselines

The do-nothing reference is the reused holdout score reported as is, the practice behind the 91.50% figure. It sets the upper end of the inflation we must explain.

The raw-count selection correction plugs raw $k$ into the same formula. This is what a careful reviewer applying a max-of-$k$ or Bonferroni-style correction would do. It uses identical inputs except for the correlation structure, so it serves as the ablation isolating $k_{\text{eff}}$. We beat it if our calibration error on inflation is lower across holdout sizes without systematic under-coverage.

The worst-case adaptive bound of order $\sqrt{k/n}$ from the fingerprinting results is the principled conservative competitor. We beat it by being much tighter while keeping coverage in non-adversarial loops. We expect to lose to it on the adversarial boosting loop, and we should.

The strongest existing methods, Thresholdout and the parameter-free Ladder, are run prospectively on the same loops, with the same models, the same holdout, and the same compute. Their noise thresholds are tuned on a separate calibration loop with a matched budget. For Thresholdout, the best reported figure is on Freedman's paradox: standard holdout reported over 60% accuracy on pure noise, while Thresholdout held about 50% Generalization in Adaptive Data Analysis and Holdout Reuse. For the Ladder, the only reported result is that it produced a leaderboard "nearly identical" to Kaggle's on 1,785 Photo Quality submissions, with no numeric error given. We found no reported result on Lacuna for either mechanism at $n$ below 1,000. The comparison measures two things. For mechanisms, it is the true accuracy of the model the analyst ends up selecting. For our estimator, it is the error of the discounted score. Our method does not need to select better models. It needs a discounted estimate as close to truth as the mechanisms' reported estimate, without having mediated the loop.

The feedback-term ablation drops the feedback statistic and keeps only $k_{\text{eff}}$. It shows whether adaptivity beyond correlated selection matters in realistic loops.

Why not just…?

Why not just keep a sequestered, touch-once test set? For a data holder planning ahead, this is the simplest correct answer and a real threat to the prospective use case. It does not address results already published from reused holdouts. At $n=153$, sequestering even 50 examples leaves a test set whose own noise ($\sigma_n$ near 4 points at $p\approx0.9$) swamps the differences being claimed. The splitting analysis in Adaptive Data Analysis for Growing Data shows that repeated fresh splits exhaust data linearly in $k$. We frame the estimator as complementary to sequestration: it tells authors how much reuse the remaining data can absorb.

Why not just apply a multiple-comparisons correction with the number of models tried? This is our raw-count baseline. Ensembles and hyperparameter sweeps produce heavily correlated models, so raw $k$ over-corrects. And if analysts do feed back holdout residuals, a count-based correction assumes non-adaptivity and can under-correct. The experiment settles this by reporting coverage and tightness for both at each $n$.

Why not just use cross-validation, as the pneumoconiosis study did with leave-one-out? Cross-validation estimates one model's error. It does not account for selecting among many models by their cross-validated scores, and The Limits of Post-Selection Generalization shows that such post-selection guarantees do not compose. Our loops include a cross-validation-driven selection condition to measure its inflation directly.

Why not just conclude, with Roelofs et al., that reuse is harmless? Their own data locate the harm in small test sets, and our law predicts exactly where harmlessness ends. If controlled loops at $n\approx150$ show inflation under one point, the skeptic is right, and we will report that as a negative result.

Proposed Experiments

The controlled loops use CheXpert CheXpert, a chest X-ray benchmark with 224,316 images. We draw feedback holdouts of 150, 500, 2,000 and 10,000 studies from patient-disjoint pools and keep a large disjoint reserve as ground truth. We add a matched tabular setting from MIMIC-III MIMIC-III, a de-identified intensive-care database, predicting in-hospital mortality. Within each setting we run four development loops that see holdout feedback: a hyperparameter sweep, architecture and ensembling search mirroring the pneumoconiosis study, an automated agent that edits models after seeing per-class holdout scores, and the adversarial boosting attack from The Ladder as a positive control. The metrics are inflation (holdout score minus reserve score of the selected model), calibration error and coverage of each estimate, and, for mechanisms, reserve accuracy of the selected model.

We will judge the proposal decisive on three checks. First, in the three realistic loops, the selection law with $k_{\text{eff}}$ should predict inflation within its own sampling error at all four sizes. Inflation at $n=150$ should be several times larger than at $n=10{,}000$, scaling as $\sigma_n$. Second, on the boosting control, the selection law should fail and the feedback statistic should flag the loop, so the disclosure honestly moves to the worst-case regime. Third, on MetaKaggle, the approximate $k_{\text{eff}}$ law should reproduce the pattern in A Meta-Analysis of Overfitting in Machine Learning: small shifts for large test sets and the observed shake-ups for sets under a few hundred examples. We found no reported number on Lacuna for predicting per-competition gap magnitude, so this check has no prior baseline to beat, only raw-count and worst-case predictions to compare against.

If the first check fails with inflation above the law, practical adaptivity is stronger than the Kaggle evidence suggests, and the right recommendation for small private cohorts becomes mandatory mediation by Thresholdout or the Ladder. The small-$n$ mechanism results would then be the main contribution.