Choosing the Active Learning Strategy at a New Hospital Before Labels Exist

Active learning means letting a model choose which images, or which pieces of images, a human should label next, so that a good model needs fewer labels. For 3D medical segmentation the gains are real but fragile. Which selection rule works best changes from dataset to dataset, and the published rankings of these rules all come from offline simulations on a single data source. Deployment does not look like that. A hospital that wants to adapt a segmentation model to its own scanners has to commit to one selection rule before it has any local labels, and models are known to become confidently wrong on unfamiliar scanners. Selection rules that hunt for uncertain regions may then look in the wrong places.

This note proposes to measure that problem directly and to fix it cheaply. We take the standard self-configuring segmentation pipeline (nnU-Net, which sets its own preprocessing and architecture from statistics of the dataset). We adapt it to held-out scanners and vendors under a protocol that copies deployment: target data arrive in acquisition order, and the selection rule must be fixed in advance. We ask two questions. Does the ranking of selection rules measured on the source site predict the ranking at the new site? And can a small "probe" round, whose labels are needed anyway, pick a better rule than either transferring the source ranking or defaulting to foreground-aware random sampling?

Core Claim

Our hypothesis is that active learning strategy rankings do not transfer across scanner shift in 3D segmentation, and that entropy-based rules are the most likely to fail there. Entropy-based rules pick regions where the model's predicted class probabilities are most spread out. Under shift, the model's errors become low-entropy. In-domain, entropy already wastes labels on ambiguous boundaries; this is the "Predictive Entropy Paradox" of nnActive. Out of domain, the errors themselves look confident (Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift; Efficient Bayesian Uncertainty Estimation for nnU-Net). Both effects point the same way: away from where the errors actually are.

The method follows from this. The first labeled batch at the new site is split evenly across the candidate rules, with one share drawn by foreground-aware random sampling. The random share serves as an unbiased local validation set. Each rule's share is then scored by how much a short fine-tune on it improves performance on that random share. The pipeline then commits its remaining budget to the winning rule. This is simpler than a full adaptive search. Its claim to novelty is that it chooses the rule using target-site evidence at essentially zero extra labeling cost. We also take the "prospective" framing seriously: the pre-commitment constraint and the order of arrival are part of the evaluation, not something added afterward.

Relation to Prior Work

Two works supply the setting. nnActive built a rigorous 3D active learning benchmark on nnU-Net. Its contributions include patch-level partial annotation, the foreground-aware random (FAR) baseline, and a foreground efficiency metric, evaluated over 7,500 runs on AMOS2022, KiTS2021, MSD Hippocampus and ACDC. It evaluates every rule within a single domain and uses one fixed rule for all rounds. We reuse its protocol and change both of those choices.

Active Learning with the nnUNet and Sample Selection with Uncertainty-Aware Submodular Mutual Information Measure combines Monte Carlo dropout uncertainty with submodular diversity in 2D nnU-Net. It reaches near-optimal Dice with under 8% of Spleen labels. It is also offline, single-domain, and 2D, and nnActive identifies training 2D models on 3D data as a pitfall.

Two works supply the method idea. AutoAL searches over mixtures of selection rules by splitting labeled data into halves, which lets it estimate validation loss without labels for the unlabeled pool. It does this only for 2D classification, and its runtime scales with the slowest candidate rule. Our probe round borrows the split-the-labels idea but makes a single committed choice, which keeps the cost of 3D training bounded. Multi-armed bandit formulations of strategy choice, discussed in the strategy-sensitivity literature (Multi-Arm Active Transfer Learning for Telugu Sentiment Analysis), are the natural online alternative. We test them as a competitor rather than assume them.

Several works motivate the problem by showing that rules are sensitive to the data, sometimes to the point of losing to random sampling: Reducing Annotation Effort for Quality Estimation via Active Learning, Activist, and A study of deep active learning methods to reduce labelling efforts in biomedical relation extraction. Uncertainty can also be trapped by out-of-distribution inputs (Bidirectional Uncertainty-Based Active Learning for Open-Set Annotation).

Shift-aware selection exists, but only outside 3D radiology. HALO multiplies prediction entropy by a hyperbolic "scarcity" radius for 2D driving scenes. Cluster Entropy selects a single pathology slide and shows that unsupervised alignment (DANN) falls below no adaptation. Neither addresses 3D volumes, self-configuring pipelines, or which rule to trust at a new site.

Finally, the prospective studies constrain how we evaluate. Prospective implementation of AI-assisted screen reading and AI support for accurate and fast radiological diagnosis of COVID-19 report workload and reading time, not only accuracy. For that reason we measure annotator time per queried patch.

Research Program

The first component is a deployment-ordered protocol for active domain adaptation in 3D nnU-Net. On AMOS, which spans two centers and eight scanners, we hold out scanners: train on the CT scans plus the MRI scanners seen during training, then adapt to each held-out MRI scanner. AMOS reports a drop of more than 20% mean Dice on unseen MRI scanners (AMOS), so there is room to recover. On cardiac MRI we train on ACDC and adapt to M&M Vendor A and Vendor B separately.

Target scans are ordered by acquisition metadata where it is available, and by fixed random order otherwise. Labels are revealed only for queried 3D patches, following nnActive's patch-annotation scheme. Budgets are counted in annotated voxels. Every method must declare its rule, or its rule-choice procedure, before seeing any target labels.

The candidate pool of selection rules contains:

  • predictive entropy;
  • mutual-information disagreement (BALD) from Monte Carlo dropout;
  • disagreement within a checkpoint ensemble from a cyclical learning-rate schedule, a shift-robust signal taken from Efficient Bayesian Uncertainty Estimation for nnU-Net;
  • a k-means diversity rule on bottleneck features;
  • FAR.

The second component is the probe-and-commit selector. Round one spends budget $b_1$, divided equally among the $K$ rules. Each rule's patches are added to the source model in a short fine-tune of a fixed number of epochs. The result is evaluated by Dice on the FAR share, and the remaining rounds use the winning rule. All probe labels, from every share, go into the final training set, so the only overhead is $K$ short fine-tunes.

A variant that keeps a mixture of rules, weighted by probe utility, isolates whether a hard commitment matters. We also test a label-free step: recompute nnU-Net's intensity normalization and spacing from the target scanner's own images before adaptation. This addresses the fact that nnU-Net computes its dataset "fingerprint" once, from the source data, and never updates it.

The third component is a small annotation-time study. Two to three annotators label the patches selected by each rule on one AMOS MRI scanner, and we record seconds per patch. This tests whether rules that favor boundaries also cost more human time. That cost appears in a clinical workflow but is invisible in simulation.

Baselines

The do-nothing reference is the source nnU-Net applied to the target scanner without adaptation, run once more with target re-fingerprinting. No adaptation result specific to these scanner-held-out AMOS splits is reported on Lacuna. The in-domain CT figure of 89.04% mean Dice for 3D-UNet, together with the reported drop of more than 20% on unseen MRI scanners (AMOS), gives the expected range.

The strongest naive competitor is FAR sampling at matched voxel budgets. nnActive shows that the advantage of active learning over FAR narrows substantially, although no numeric gap is reported on Lacuna. Beating FAR means a higher area under the Dice-versus-voxels curve across seeds, with paired confidence intervals that exclude zero, on at least most target scanners.

The strongest existing practice is to transfer the source ranking: run the nnActive comparison in-domain and deploy whichever rule wins there. We also include USIM-style uncertainty combined with submodular diversity, using its 2D numbers only as context. For this baseline, Lacuna reports USIMF reaching near-optimal performance with under 8% of Spleen and under 5% of Liver data (Active Learning with the nnUNet), and nothing under cross-site shift.

The online competitor is a UCB bandit (upper confidence bound) over the same pool. It is rewarded by Dice gain on the FAR share in each round.

The upper reference is an oracle: the best single rule in hindsight. We report "strategy regret," the Dice gap to this oracle at a fixed budget. All methods share the nnU-Net configuration, seeds, epochs per round, and labeled-voxel budget. The bandit and the probe use identical FAR validation shares.

Two ablations isolate the key ingredient. The first replaces the probe's target-site evidence with the same procedure run on source-domain data. The second removes the checkpoint-ensemble rule from the pool.

Why not just…?

Why not just use foreground-aware random sampling and skip strategy choice entirely? This is the most serious threat. nnActive found that FAR narrows active learning's advantage even in-domain, and under shift every uncertainty signal gets worse. Our probe includes FAR as an arm, so if FAR is truly best, the selector should pick it and lose only the cost of the probe. If FAR matches the full method within seed noise on most scanners, we will report it as the recommended practice. That would be a useful negative result, not a failure.

Why not just carry over the strategy that won in the source-domain benchmark? This is free, and it is what readers of nnActive would do. It fails only if rankings reorder under shift. We measure the reordering directly with Kendall rank correlation between the source and target rankings. If the correlation is high and source transfer has near-zero regret, the probe is unnecessary.

Why not just run a bandit across rounds instead of committing early? Bandits adapt without committing. However, 3D active learning runs have only a handful of rounds, so each arm is tried a few times at most, and early mistakes spend expensive voxels. The comparison under a matched budget settles this. If the bandit's regret is equal or lower, we adopt it.

Why not just rely on nnU-Net augmentation, unsupervised alignment, or post-hoc calibration? AMOS reports a drop of more than 20% on unseen scanners despite nnU-Net-style augmentation. DANN fell below the unadapted baseline (0.366 versus 0.406 mIoU) in Cluster Entropy. Calibration reduces expected calibration error, for example from 16.78% to 15.37% on MS-CMRSeg (Improved post-hoc probability calibration), but it does not improve segmentation. The re-fingerprinting baseline tests the cheapest label-free fix directly.

Proposed Experiments

The first decisive check is rank transfer. Across AMOS held-out MRI scanners and the ACDC to M&M vendor splits, we compute the correlation between source and target strategy rankings and the regret of source-ranking transfer. We predict low correlation, with entropy-based rules falling furthest. We will also check whether target patches with confident errors, where the model is wrong with low entropy, are under-selected by entropy relative to checkpoint-ensemble disagreement.

The second is probe-and-commit against FAR, source transfer, and the bandit. We compare area under the Dice-versus-voxels curve and its speed-up factor (The Speed-up Factor), regret against the oracle, and foreground efficiency, using five seeds per split.

The third is the annotation-time study. We report seconds per patch and Dice gained per annotator-minute for each rule. This tests whether a ranking based on voxels survives a ranking based on human time.

The proposal is falsified if source rankings transfer and FAR is within noise of every adaptive method. In that case the practical guidance becomes "use FAR plus re-fingerprinting," and we will report exactly that.