Counting What a Few Scans Actually Tell You: Spatially Honest Model Selection at a New Hospital

A hospital that receives a segmentation model trained elsewhere usually has only five to ten of its own annotated scans for choosing among candidate models or checkpoints. Each scan contains millions of labeled voxels, so it is tempting to treat those voxels as millions of independent test cases. They are not. Neighboring voxels and adjacent slices share anatomy, noise, and scanner artifacts. Statistical tests that pool voxels therefore report false confidence and switch to worse models. The opposite choice, one number per patient, is honest but has almost no statistical power with five patients. This proposal sits between those two extremes. We estimate how far apart two regions of a scan must be before the difference in error between two candidate models stops being correlated. We then use that distance to cut each scan into roughly independent blocks, score the models on a continuous per-voxel signal within each block, and run a paired comparison whose variance accounts for both between-patient and within-patient spatial dependence.

The claim is specific and testable. On real cross-scanner shifts, the correction should keep the rate of false model switches at its nominal level, which voxel-pooled tests fail to do. It should also select the model that is truly best at the target site more often than per-patient tests, using the same handful of scans. If between-patient variability in the model difference dominates, the gain disappears. Measuring that variance split is itself a result the field lacks, because multi-center benchmarks such as AMOS and FeTA 2024 report shift magnitudes from hundreds of scans and never ask what a practitioner with five scans can conclude.

Core Claim

Three bodies of work each solve part of the problem and each break at this intersection.

The small-sample literature supplies the statistical tools, but all of them assume independent evaluation units. Pointwise sampling uncertainties on the Precision-Recall curve gives fast likelihood-ratio intervals that stay valid down to $N=10$, but it models the confusion matrix as independent multinomial trials. Making Learners (More) Monotone accepts a model update only if a paired test such as McNemar's finds it significantly better. Its guarantee of at least $(1-\alpha)^n$ probability of never degrading assumes independent validation points. DataDecide shows that a continuous signal, the probability assigned to the correct label, ranks candidates far better than discrete accuracy at small scale. It does so for i.i.d. text. We reuse the continuous-signal idea from DataDecide and the test-gated selection idea from the monotone-learner work. The independence assumption behind both is the target we fix.

The spatial-statistics literature supplies the diagnosis. Cross-validation for Geospatial Data and Model Evaluation for Geospatial Problems show that random cross-validation is optimistic under spatial autocorrelation: it was 43% optimistic on California Housing. Their buffered, importance-weighted fix (IBCV) breaks on small datasets because buffers discard training data. Localization is All You Evaluate shows the same leakage at scale: MapTRv2 fell from 72.7 to 26.2 mAP under geographically disjoint splits.

Our setting avoids the starvation problem Wang et al. identify. The candidate models are trained on source hospitals, and target scans are used only to evaluate them. A spatial buffer therefore removes no training data. It only determines how many roughly independent blocks each target scan contributes. That shift, from buffering for training to blocking for evaluation, is what makes spatial correction affordable in a small sample.

The medical domain-shift work motivates the problem and sets the testbeds. AMOS reports drops exceeding 20% mDice on unseen MRI vendors. FeTA 2024 attributes about 0.075 Dice to acquisition site. Improved post-hoc probability calibration for out-of-domain MRI segmentation shows that raw probabilities become overconfident under shift. This last finding constrains the design. Our continuous signal compares two models on the same labeled voxels, so it measures how much probability each assigns to the true class. It does not rely on either model's confidence being calibrated, which is the failure that sinks unlabeled proxies.

Research Program

We build the estimator in three steps.

Step 1: a paired score map. For each target scan $i$ and candidate models A and B, we compute a per-voxel paired score: the difference in log probability each model assigns to the ground-truth label. We also compute the difference in a soft-Dice contribution as a secondary signal.

Step 2: a correlation range. We fit an empirical variogram to these difference maps, pooled across the few target scans and across the candidate pairs being compared. A variogram is a curve showing how the similarity of two values decays with the distance between them. From it we read off a correlation range $r$: the distance at which paired differences become roughly uncorrelated. Each scan is then tiled into $m_i$ blocks of side $r$, and we compute the mean paired difference $d_{ib}$ for each block $b$ of scan $i$.

Step 3: an honest variance. The mean difference $\bar d$ over $n$ scans and blocks has variance under a two-level model:

$$\mathrm{Var}(\bar d) \approx \frac{\sigma_p^2}{n} + \frac{\sigma_w^2}{n\, \bar m},$$

The symbols are as follows. $\sigma_p^2$ is the between-patient variance of each scan's mean paired difference. $\sigma_w^2$ is the between-block variance within a scan after blocking. $\bar m$ is the average number of blocks per scan.

We estimate both variance components by restricted maximum likelihood. A t-reference with degrees of freedom set by a Satterthwaite approximation drives the accept/reject gate. This drops directly into the monotone-learner wrapper in place of McNemar's test.

The formula makes the scientific question concrete. Blocking helps only to the extent that $\sigma_w^2/\bar m$ is not negligible next to $\sigma_p^2$. We expect paired differences between similar models to vary less across patients than raw Dice does, which would give blocks real value. This is a hypothesis, and we will report the estimated ratio on every benchmark.

Topology gets a guardrail rather than the headline. FeTA 2024 shows that the winning team halved the Euler Characteristic Difference, a count of spurious holes and disconnected pieces, relative to the runners-up at similar Dice. We therefore compute per-block topological error as a second paired score and examine whether selecting on the continuous signal alone ever picks a topologically worse model.

Baselines

Source-validation selection (the naive reference). The hospital picks whichever candidate scored best on the source institution's validation set and uses none of its own labels. This fixes the floor. AMOS's more than 20% mDice drop on unseen MRI vendors suggests source rankings may not transfer, but whether rankings flip is exactly what we measure.

Best mean Dice on the k target scans. The practitioner simply takes the candidate with the highest mean Dice over the available target scans, with no test. This is what practitioners do in practice. Beating it means lower regret, defined as the gap between the full-site Dice of the chosen model and that of the truly best model, at equal k.

Voxel-pooled testing. We apply the monotone-learner test gate, using McNemar's test and the Wilks intervals from the MMU library of Urlus et al., to all voxels pooled as independent. This is the "honest-looking" competitor that the spatial correction must beat on false-switch rate. We expect it to have high power and badly inflated Type I error. Beating it means holding the false-switch rate at the nominal α while giving up little power.

Per-patient paired test (the strongest competitor). We run a paired t-test or Wilcoxon test on per-scan mean differences in the same continuous signal, and separately on per-scan Dice. It is statistically valid and dead simple, so any gain from our method must come from within-scan information.

Spatial block bootstrap with the same blocks. This is the resampling alternative. It tests whether the analytic two-level variance adds anything beyond the blocking itself.

BUS-stop-style unlabeled selection. Early Stopping Based on Unlabeled Samples reached 0.872 average accuracy versus 0.826 for split-validation stopping on text. We adapt its signals, conf-sim (similarity of confidence distributions) and class-sim (similarity of predicted class proportions), to predicted organ volumes on unlabeled target scans.

Fairness conditions. Every method sees the same k target scans, drawn from the same 1,000 random subsamples, and the same candidate pool. No method tunes its parameters on the held-out full-site labels. Our correlation range $r$ is estimated from the k scans alone.

Ablations. Two ablations isolate the key ingredients. Replacing the continuous signal with block Dice isolates the DataDecide ingredient. Fixing $r$ at one voxel (no blocking) or at one scan (the per-patient test) isolates the spatial ingredient.

Reported reference numbers. For the strongest reported number, the file offers AMOS's 3D-UNet at 89.04% mDice on CT, against 86.32% for Swin-UNetr. This supplies a real candidate gap that should be detectable. Beyond that, no reported result found on Lacuna for small-sample model-selection accuracy or false-switch rates on any medical target site. Every selection-level baseline number will therefore be produced here.

Why not just…?

Why not just test per patient, the standard grouped analysis? This is the real threat, and we treat it as the main comparison rather than a straw man. It is valid by construction. If $\sigma_p^2$ dominates, it will match our method, and we will report that outcome as a negative result with a practical message: "with five scans, label a sixth instead of modeling voxels." What it cannot do is use the thousands of partly independent regions inside each scan. The variance decomposition from the Research Program tells us in advance on which benchmarks our method should win. Weiss and Indurkhya's result, that cross-validation variance explodes at tiny sample sizes, is the reason a one-number-per-patient test at $n=5$ is likely to be underpowered.

Why not just block-bootstrap the voxels? A spatial block bootstrap with a well-chosen block size is a legitimate alternative, and it is in the baseline set. Two concerns remain. Bootstrapping five patients badly understates between-patient variance, and Urlus et al. note that resampling behaves erratically in data-scarce tails. Their Monte Carlo reference also took hours per curve, which is prohibitive inside a repeated selection loop. If the block bootstrap achieves equal coverage at equal cost on our subsamples, we will recommend it, since blocking by the estimated range $r$ is still our contribution in that case.

Why not just use IBCV from geostatistics? IBCV buffers training points and reweights validation losses by density ratios. In our setting the models are frozen and trained on source data, so there is nothing to buffer, and density-ratio estimation on five high-dimensional scans is exactly where Wang et al. report instability. We will still run importance-weighted per-scan scores as a check that covariate shift within the target cohort is not biasing the selection.

Why not just select using unlabeled target scans? Unlabeled selection needs no annotation, which is attractive. Ouyang et al., however, show ECE of 16.78% under the bSSFP-to-LGE sequence shift. Confidence-based signals such as BUS-stop's conf-sim therefore reward exactly the overconfident models we want to reject. We include the adapted BUS-stop baseline, and we expect it to underperform even per-patient labeled tests.

Proposed Experiments

Testbeds. We use three shifts that come with enough labeled target scans to define ground truth.

The first is AMOS MRI. Candidates are trained on CT plus MRI from seen vendors, and the target is scans from the unseen vendor. Candidates come from the deform-uxnet and U-Shaped-Connection code: 3D-UNet, Swin-UNetr, and augmentation and checkpoint variants.

The second is FeTA 2024, with training at KISPI and Vienna and the target being the 0.55T low-field scans from KCL. There the reported Dice is 0.86.

The third is ACDC to MS-CMRSeg LGE, the cross-sequence cardiac shift studied by Ouyang et al.

Protocol. On each target set, the "true" ranking of candidates comes from all available labeled target scans. We then draw 1,000 random subsets of k ∈ {3, 5, 10} scans and run every selection method on each subset.

Metrics. The primary metrics are selection regret, the probability of picking the true best model, and the false-switch rate. We measure false switches in a null construction: pairs of checkpoints whose full-site performance differs by less than 0.2 Dice points, where any "significant" switch is an error. We also report the coverage of 95% intervals on the model difference, the estimated ratio $\sigma_w^2/(\bar m\,\sigma_p^2)$, and the estimated correlation range $r$ per organ and modality.

Decisive checks. Three outcomes decide whether the idea holds. First, voxel-pooled tests should show false-switch rates well above α while our method stays near α. If both stay near α, spatial dependence is irrelevant and the proposal fails. Second, at k=5 our method must lower regret relative to the per-patient test on at least the benchmarks where the variance ratio is large. The ratio should predict where it wins; a gain where the ratio is small would indicate a bug rather than a win. Third, the continuous-signal version must beat the block-Dice ablation, which replicates DataDecide's finding in a spatial, shifted setting. Because FeTA 2024 found similar Dice alongside very different topology, we also report how often the selected model has worse topological error than the runner-up. That tells us whether topology needs to enter the gate itself.