Fold-Invariant Class Priors for Leave-One-Subject-Out Evaluation of Motion-Artifact Models

When a medical imaging model is trained on fewer than a hundred patients, the standard honest way to test it is leave-one-subject-out cross-validation (LOSO). You train on every patient but one, test on the patient left out, and repeat until every patient has been the test case. LOSO stops the model from scoring well just by recognizing a patient it has already seen. It has a quieter flaw, though. Removing one patient's scans shifts the label balance of the training set against that patient's labels. A child who moves a lot contributes several badly motion-corrupted scans. Holding that child out leaves a training set that looks cleaner than usual, and the model then leans toward predicting "clean" for exactly the child whose scans are not. Pooled across folds, this produces a systematic downward bias. Distributional bias compromises leave-one-out cross-validation showed this bias can push the area under the ROC curve (auROC) on pure noise to 0.25 instead of 0.5, and that it steers model selection toward under-regularized models.

We propose to measure this bias where it should be largest but has never been checked: grading motion artifacts in small, imbalanced pediatric MRI cohorts. Here severe scans are rare and clustered within a few restless subjects. We also propose to remove the bias without throwing away data. The idea is to train every fold toward the same fixed class prior, meaning the class proportions the model is pushed to assume, rather than the prior the fold happens to contain. We do this with fixed-ratio batch sampling plus a logit offset. The core claim is that this makes LOSO estimates unbiased on null data and closer to true held-out performance on real data. It achieves this without the subject discarding that the existing fix, Rebalanced LOOCV (RLOOCV), requires, and which small imaging cohorts cannot afford.

Why this gap is real

RLOOCV was developed and tested on tabular and microbiome data, with one sample per held-out unit. On those benchmarks it raised median auROC across 49 UC Irvine classification datasets from 0.74 to 0.81. Its fix is to delete majority samples from each training fold until every fold has the same label distribution. Imaging LOSO differs in two ways that motivate this work.

First, a subject contributes many scans whose labels are correlated. The LISA 2025 data used by BRIQA has 532 scans from 244 children. The fold-to-fold shift in the training prior therefore scales with how many scans the held-out subject has and how unusual they are, not with $1/N$. Concretely, if the cohort has $M$ scans of which $S$ are severe, and held-out subject $i$ has $m_i$ scans of which $s_i$ are severe, the training fold's severe rate is

$$p_{-i} = \frac{S - s_i}{M - m_i},$$

which falls exactly when subject $i$ is motion-prone ($s_i/m_i$ high). The effect grows as the cohort shrinks.

Second, deleting subjects to equalize folds is costly when the cohort is already tiny. An Open-Access Deep Learning Tool for Early Detection of Chiari Type I Malformation via MRI Classification trained on just 60 images and found that model variance, not bias, dominated failure.

Separately, BRIQA's rotating batch buffer, which holds a fixed 2:1:1 ratio of clean, moderate and severe scans in every batch, already fixes the class mix the network sees, whatever the fold. BRIQA was evaluated on challenge splits, never under LOSO. Nobody has asked whether this kind of sampling already neutralizes the distributional bias, or what it leaves behind. That is the opening.

Core Claim

Holding the effective class prior constant across folds removes the first-order LOSO bias. This requires two things: fixed-ratio sampling, and a logit offset of $\log \pi_{\text{ref}} - \log p_{-i}$, where $\pi_{\text{ref}}$ is the full-cohort prior. Any decision threshold must also be set against $\pi_{\text{ref}}$, never against the fold.

Sampling alone is not enough. It leaves a second-order bias, which we will measure directly. When the held-out subject is a minority-class subject, the training fold has fewer distinct minority examples, so oversampling repeats them more often. The model then memorizes them more, a failure already noted for oversampling in Mitigating Class Imbalance in Clinical Machine Learning. We expect this "diversity bias" to be the residual, and we expect equal per-subject weighting to reduce it.

Research Program

The work has three parts.

The first is a characterization. We will take LISA 2025 and subsample cohorts of 20, 40, 80 and 160 subjects. Under subject-level label permutation, we will measure how far the pooled LOSO auROC and macro F1 fall below chance. Labels are shuffled across subjects, so each subject's internal label structure is preserved but carries no information. We will do this for three training recipes: plain cross-entropy, BRIQA's sampling plus gradient-norm reweighting, and the proposed fold-invariant prior. This tells us how big the problem is as a function of cohort size and of how unevenly severe scans are spread across subjects.

The second part is an estimator-accuracy study, which is the decisive one. We permanently reserve about 100 LISA subjects as an oracle test set. For each small cohort size, we train on the remaining subjects under each protocol. We then compare the LOSO estimate with the actual performance, on the oracle set, of the same recipe trained on all $N$ subjects. The quantity of interest is the estimate minus the truth, along with the rank agreement between estimated and true model orderings.

The third part is model selection. Austin et al. predict that biased LOSO favors under-regularized models. Our small-sample sources point the same way with architecture: the Chiari study found high variance in a ResNet50V2 and better results from a compact DenseNet121, and BRIQA found that ResNet18 and DenseNet169 each won on different artifact types. We will choose among those backbones and among weight-decay settings by each protocol's LOSO estimate. We then report the regret on the oracle set: how much worse the chosen configuration does than the best one.

To show the result is not specific to quality grading, we repeat the second part on a downstream task where motion and a rare label interact. We simulate rigid motion in $k$-space (the raw frequency-domain data the scanner records) on fastMRI+ brain studies and detect rare annotated pathologies. Studies serve as subjects. As a real-motion stress test, we also run severity regression on PMoC3D, which has 8 subjects with four scans each and expert severity ratings. That is extreme small-$N$, and we report it only as a null-calibration check, since 8 subjects cannot support an oracle split.

Baselines

The naive reference point is LOSO with plain cross-entropy, no rebalancing, and pooled predictions. This is what most small-cohort imaging papers implicitly do. The proposal beats it if, on subject-permuted labels, its pooled auROC has a 95% interval across 200 permutations that covers 0.5 while the naive interval sits below.

The strongest existing training recipe is BRIQA's rotating 2:1:1 buffer with $L_2$ gradient-norm loss reweighting. Its paper reports an average macro F1 of 0.706 on LISA 2025, up from 0.659. That number comes from the challenge split, not LOSO, so we re-run BRIQA under LOSO ourselves. We use the same backbones, the same augmentation, and the same per-fold hyperparameter budget as every other arm. This baseline is the real competitor because it already fixes the batch prior. If it matches the proposal on both null calibration and oracle error, the logit offset and threshold anchoring add nothing.

The strongest existing evaluation fix is RLOOCV adapted to subjects. In each fold we drop whole majority-class subjects, chosen at random with a fixed seed, until the fold's prior equals $\pi_{\text{ref}}$. The proposal beats it if it matches RLOOCV's null calibration but shows lower oracle estimation error and lower selection regret at $N \le 40$, where discarding subjects should hurt most. Austin et al. report the 0.74 to 0.81 median auROC gain for RLOOCV on tabular data. No reported result for RLOOCV on imaging or subject-level LOSO was found on Lacuna.

Stratified grouped five-fold cross-validation, grouped by subject and stratified by subject-level severity, is run with the same budget. The ablations isolate the key ingredient. We run sampling only, logit offset only, both together, and both with equal per-subject weighting. The last arm tests the diversity-bias hypothesis specifically.

Why not just…?

Why not just use stratified grouped k-fold instead of LOSO? This is the most serious threat. Larger test folds that are stratified by severity keep the training prior nearly constant, which could make the whole problem disappear. In very small cohorts with only a handful of motion-prone subjects, exact stratification is impossible and five folds leave noisy estimates. Whether this matters is an empirical question, and the second part answers it directly by comparing estimation error. If grouped five-fold matches the proposal at every $N$, we will say so and recommend it. The surviving contribution would then be the bias characterization, together with evidence that LOSO, which remains common in small clinical studies, needs the correction.

Why not just apply RLOOCV as published? It works by discarding data. The small-sample evidence in our sources, including the Chiari study's variance-dominated failures and the COVID chest X-ray study Unveiling COVID-19 from Chest X-ray with deep learning on small-data overfitting, suggests that dropping subjects from 30-subject cohorts will widen the gap between the estimate and the truth even as it fixes the mean. The RLOOCV baseline arm measures this trade-off directly.

Why not just ignore it, since deep networks with rebalanced batches may not track the fold prior at all? That is possible, and it is exactly what the BRIQA-under-LOSO arm tests. If the null curves for BRIQA sit at 0.5 for every $N$, then fixed-ratio sampling is already sufficient. In that case the paper's message becomes "use balanced sampling under LOSO, and here is the proof." We consider the residual diversity bias likely enough to test, but we do not presume it.

Decisive Checks

Two outcomes would settle the proposal. The first is null calibration: across 200 subject-level permutations at $N = 20$ and $N = 40$ LISA subjects, the proposed protocol's pooled auROC centers on 0.5, while the naive protocol's falls clearly below it and BRIQA-under-LOSO's either matches the proposal or does not. The second is oracle agreement: at those same sizes, the proposal's mean absolute gap between the LOSO estimate and the oracle macro F1 is smaller than RLOOCV's and no worse than grouped five-fold, and it picks configurations with lower oracle regret.

If the proposal fails both checks against grouped five-fold, the correction is not worth adopting. If it passes them only below about 40 subjects, that size becomes the practical boundary we report.