Leave-One-Out Validation Without Label-Prior Drift for Small-Sample Disease Staging

When a clinical cohort holds only a few dozen patients, researchers often validate models with leave-one-out cross-validation: train on every patient but one, predict that patient, and repeat for each patient in turn. There is a known flaw. Removing a patient also removes their label from the training set, so every training fold slightly under-represents the held-out patient's own class. Models that lean on class frequencies then push each prediction away from the correct answer. On pure noise this yields scores well below chance. This proposal argues that the flaw becomes much worse when the task is fine-grained staging, such as five thyroid risk tiers or three Gleason patterns, instead of a binary detection. A rare intermediate grade with three patients loses a third of its training examples every time one of them is held out.

We propose a fix that discards no data. In each fold, the remaining members of the held-out patient's class are given slightly larger training weights, so the weighted class proportions match the full cohort exactly. The correction extends to any number of classes and still works for a class with two patients. We pair it with permutation significance testing, in which the whole pipeline is rerun on shuffled labels, including feature selection and tuning. The permutation test answers whether any signal exists. The reweighting makes the reported number mean what readers think it means.

Core Claim

The claim has two parts. The first is that distributional bias in leave-one-out validation scales with the inverse size of the held-out class. Refining the label space therefore amplifies the bias. Evaluations of granular staging in small cohorts are distorted more than binary ones, and most distorted on the rare intermediate grades that give staging its clinical value.

The mechanism is simple. Consider a model whose decision depends on the log of the training prior for class $c$. Holding out one of the $n_c$ members of that class, in a cohort of size $N$, lowers this log-prior by about $1/n_c - 1/N$. That penalty is negligible for a large class and severe for a class of three.

The second part is that a per-fold reweighting removes this prior shift without discarding samples. If the held-out patient belongs to class $c$, each remaining class-$c$ training sample receives weight

$$w_c = \frac{n_c}{n_c - 1},$$

and all other samples keep weight 1. The weighted total is then exactly $N$, and each class's weighted share equals its full-cohort share $n_k/N$ in every fold. Any pipeline stage that accepts sample weights sees the same label prior in every fold. That includes the classifier itself, weighted feature filters and inner tuning loops.

The method builds on Distributional bias compromises leave-one-out cross-validation. That paper diagnosed the negative correlation between training label mean and held-out label. It showed median auROC near 0.25 on random data, and it fixed binary tasks with rebalanced leave-one-out (RLOOCV), which drops one opposite-class training sample per fold. Our work differs in two ways: we treat the multi-class and ordinal case, and we reweight instead of deleting. Deletion undercuts the reason small studies use leave-one-out in the first place. That reason is the near-zero pessimistic bias of training on $N-1$ samples, documented in A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. With five classes, keeping every fold's proportions fixed by deletion would require removing several patients per fold.

A second, sharper point frames the whole study. A full-pipeline permutation test run on leave-one-out scores remains valid even under distributional bias. The bias is present in every shuffled replicate, so the observed score is compared against a null that shares it. Distributional bias therefore does not create false positives in a correctly permuted analysis. What it damages is the reported performance estimate, the ranking of candidate models, and possibly the test's power. Existing work has not separated these harms, and doing so tells practitioners which fix they actually need.

Research Program

The work has three strands.

The first is analytical and simulation-based. We derive the per-fold prior shift for $K$ ordinal classes. We then simulate null and weak-signal data with class counts drawn to mimic clinical staging, including rare middle grades. On these simulations we measure how far each validation scheme's score departs from chance under the null, and from the true out-of-sample score under signal. The models are those small clinical studies actually use: regularized logistic and ordinal regression, decision trees and random forests, nearest-neighbour classifiers, and pipelines with recursive feature elimination, which is the setup of Machine learning to predict hemorrhage and thrombosis during extracorporeal membrane oxygenation.

We expect reweighting to help most where the class prior enters model fitting, as in logistic intercepts, leaf class frequencies and weighted losses. We expect it to help least for unweighted nearest-neighbour voting, and we will report that boundary rather than hide it.

The second strand is the permutation layer. For each validation scheme we build a Monte Carlo permutation null by rerunning the entire pipeline on shuffled labels. Rerunning selection and tuning inside every shuffle matters: The choice of null distributions for detecting gene-gene interactions in genome-wide association studies showed that permuting after a search step turned $p = 4.226 \times 10^{-7}$ into $p = 0.148$ once the search was included.

Where samples are nested, shuffles are blocked at the level of the unit. Examples are the paired transverse and longitudinal views per nodule, or several videos per newborn. Multivariate classification of neuroimaging data with nested subclasses: Biased accuracy and implications for hypothesis testing showed that unblocked shuffling of nested data yields false positives, with null means as high as 78.5%.

We use Monte Carlo rather than the exact convolution method of Exact Paired-Permutation Testing for Structured Test Statistics. That method needs a statistic that adds up over samples under fixed predictions. Our statistic requires retraining the model for every shuffle, so it does not apply.

The third strand is empirical validation on real staging data, set up so that the true performance is known. We take a large labelled cohort, repeatedly draw small development subsets of 30, 50 and 100 patients, and run each validation scheme on the subset. The remaining patients serve as a large external-style test set that gives the subset model's actual performance. This lets us measure bias and variance of each estimator directly, not merely assert them. It also lets us check the error bars that Cross-validation failure: Small sample sizes lead to large error bars reported for binary tasks: ±15% at $N=30$ and ±10% at $N=100$. We test whether those error bars widen further for five-class staging.

Baselines

The naive reference is plain leave-one-out with all $N-1$ training samples and no correction. For ranking metrics, out-of-fold predictions are pooled across folds. This is what small clinical studies typically report. It is the scheme whose pooled AUC A comparison of AUC estimators in small-sample studies found to fall to about 0.3 on no-signal data at $N=30$.

The strongest existing method is RLOOCV from Distributional bias compromises leave-one-out cross-validation, released as the RebalancedCV package. On binary tasks it is the only published correction aimed at this exact bias. Its reported result is that across 49 UCI classification datasets, median auROC rose from 0.74 under LOOCV to 0.81 under RLOOCV. RLOOCV is defined only for binary labels and scalar regression, so we test its natural multi-class extension. In each fold we drop one training sample from each class other than the held-out one, chosen at random with fixed seeds, which holds proportions near the cohort's. We also run the original binary version on the binary UCI subset to confirm we reproduce the 0.74 to 0.81 shift before extending it.

The third baseline is a leave-one-per-class-out scheme. It generalizes the leave-pair-out validation that A comparison of AUC estimators in small-sample studies showed to be nearly unbiased for binary AUC. Each fold holds out one patient from every stage, and scores are averaged over sampled tuples.

The fourth baseline is stratified five-fold cross-validation, the setting used by CMCNet. On the full STN set, CMCNet reported 75.17% balanced accuracy and 91.78% weighted F1 for five-class TI-RADS (CMCNet: Aligning Ultrasound Image Embeddings with Textual TI-RADS Representations for Fine-Grained Thyroid Classification). That figure comes from 600 nodules and an end-to-end trained model, so it serves only as a ceiling for context, not as a target for our small subsets.

Two ablations isolate the key ingredient. The first applies the weights to the final classifier only, not to feature selection and tuning, which tests whether propagating the correction through the pipeline matters. The second is post-hoc prior correction: train unweighted, then rescale each fold's predicted class probabilities by the ratio of full-cohort to fold class proportions.

All schemes share the same subsets, feature extractors, model families, inner tuning grids and random seeds. Where the number of fitted models differs across schemes, we match compute by fixing the number of sampled folds.

Reweighting beats the baselines if three conditions hold across the small-sample regimes. First, its estimate of balanced accuracy and quadratic weighted kappa has lower mean-squared error against the held-out truth than both naive leave-one-out and multi-class RLOOCV. Second, its null-data score sits at chance. Third, its model rankings agree better with the held-out rankings. The gains should be largest for the rarest grades.

Why not just…?

Why not just run a permutation test and ignore the bias? For deciding whether any signal exists, this is essentially right, and we say so: a full-pipeline permutation test is valid under distributional bias. It still fails as a reporting tool. A leave-one-out balanced accuracy that is significant against its permutation null can be far below the model's real accuracy. Readers take that number as the estimate, and model selection uses it. Our subsampling design measures this gap directly. We also compare power at matched significance levels with and without reweighting. If power does not change, we will report that permutation alone suffices for detection.

Why not just use stratified five-fold cross-validation? With a grade of three patients, five folds cannot be stratified evenly. Distributional bias compromises leave-one-out cross-validation notes that imperfect stratification leaves residual prior shift. Training on 80% of 30 to 50 patients also adds the pessimistic bias that A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection attributes to smaller training folds. This is a real competitor at $N=100$, where folds are less starved, and the held-out comparison will show where the crossover lies.

Why not just drop samples, as RLOOCV does, or hold out one patient per stage? Both keep proportions fixed by shrinking the training set. With five stages, leave-one-per-class-out trains on $N-5$ patients. At $N=30$ that is a 17% loss, concentrated in the rare grades that matter most. The comparison will show whether the data loss costs accuracy against the held-out truth or only adds variance. If leave-one-per-class-out matches reweighting, it is the simpler recommendation for ranking metrics, and we would say so.

Why not just correct the predicted probabilities afterwards? This is the most serious threat. For a well-calibrated logistic model, rescaling probabilities by the prior ratio is nearly equivalent to reweighting. It cannot, however, reach bias that enters through fold-level feature selection, tree construction or inner tuning, since those stages already acted on the shifted prior. The ablation settles it. If post-hoc correction matches full reweighting on the elimination and tree pipelines as well as on logistic models, we will recommend the post-hoc version as the simpler method.

Proposed Experiments

The first check is null calibration. On shuffled labels at $N \in \{30, 50, 100\}$ with realistic five-class and three-class count profiles, uncorrected leave-one-out should give quadratic weighted kappa below zero and balanced accuracy below $1/K$, where $K$ is the number of classes. The deviation should grow as the smallest class shrinks, following the $1/n_c$ prediction. Reweighting should recentre both metrics at chance. Every scheme's full-pipeline permutation test should hold the nominal 5% false-positive rate, with blocking where samples are nested.

The second check is estimation accuracy on real staging data. The main testbed is the STN thyroid set of 600 nodules with TR1–TR5 labels, which is public through the CMCNet authors. Features come from frozen image embeddings of the kind CMCNet used, so that hundreds of leave-one-out fits are affordable. We draw 200 small subsets per size and score each against the nodules not in that subset. The Gleason pattern 3/4/5 tissue microarray data from Pathologist-like explainable AI for interpretable Gleason grading in prostate cancer serves as a second three-class cohort, using majority-vote labels. We also test whether the pathologists' soft labels change the size of the bias.

The third check is breadth and replication. We rerun on the 49 binary UCI classification datasets, where we should recover the published RLOOCV gain and test whether reweighting matches it without discarding samples. We then run the multi-class datasets in the same collection.

The decisive outcome concerns the rare grades. On five-class STN subsets at $N=30$, reweighting must cut the error of per-grade recall estimates for the rare intermediate grades relative to multi-class RLOOCV. It must also be matched by post-hoc correction only on logistic pipelines. If instead post-hoc correction or leave-one-per-class-out does as well everywhere, the contribution reduces to the scaling result and a recommendation for the simpler fix, which is still a useful answer for small clinical staging studies.