When a segmentation model trained at one hospital is run on scans from another, its predictions degrade. A common fix is self-training: the model labels the new hospital's unlabeled scans and then trains on its own most trustworthy guesses, called pseudo-labels. Under scanner shift, deciding which guesses to trust is the hard part. Softmax confidence is miscalibrated. The usual alternative is to check each voxel against its neighbors, meaning nearby pixels in the same image or the average feature of its predicted class. That check has a blind spot. The errors a new scanner introduces, such as a bias field, an unfamiliar contrast, or an artifact, are shared across a whole scan. So a voxel's spatial neighbors tend to be wrong in the same way it is, and voting among them confirms the error.
This note proposes that a pseudo-label for a voxel should be accepted only if voxels from other patients' scans agree with it. These are voxels whose learned features are close to it, while scan-level errors are roughly independent of its own. The refinement is a nearest-neighbor vote, but the neighborhood is defined across the target cohort and excludes the voxel's own volume. The claim is specific and can be falsified. Consensus across patients should detect scan-wide systematic errors that same-scan consensus and global class prototypes miss. The gain should be largest on 3D, multi-structure targets whose classes look different in different places.
Core Claim
Neighbor-based label checking only works when the neighbors' errors are not strongly correlated with the sample's own error. Methods for learning from noisy labels on natural images, such as SimiFeat (training-free nearest-neighbor voting to flag corrupted labels) and Jo-SNC (neighbor-prediction divergence to sort clean from noisy samples), implicitly have such neighbors, because each image is an independent sample. Dense medical segmentation breaks this assumption. The two medical methods closest to this proposal each pick a neighborhood that inherits the correlation:
- Local-global pseudo-label correction (LGDA) refines fundus pseudo-labels by comparing each pixel with its spatial neighbors in the same image, then with global foreground and background prototypes.
- UP2D filters pixels by their distance to batch-level class prototypes plus Monte Carlo dropout uncertainty.
In LGDA the spatial neighbors share the scan's errors. In both methods the prototypes are a single averaged point per class, which blurs over the fact that, for example, the right ventricle looks different at basal and apical slices.
The proposal defines a voxel's neighborhood as its $k$ nearest feature-space neighbors drawn only from other target volumes. The pseudo-label $\hat y(v)$ of voxel $v$ in volume $i$ receives an agreement score:
$$ s(v) = \frac{1}{k}\sum_{u \in \mathcal{N}_k^{-i}(v)} w(u)\, \mathbf{1}[\hat y(u) = \hat y(v)]. $$
Here $\mathcal{N}_k^{-i}(v)$ is the set of $k$ nearest voxels to $v$ in feature space after excluding volume $i$. The weight $w(u)$ is the cosine similarity between the two voxels' features, and $\mathbf{1}[\cdot]$ is 1 when the labels match. Voxels with high $s(v)$ supervise the student network. Voxels with low $s(v)$ are relabeled with the weighted cross-patient majority or masked out, depending on how strong that majority is. The single ingredient being tested is the exclusion of the voxel's own volume.
Research Program
The base system is a standard source-free student-teacher loop on an nnU-Net-style 3D backbone. "Source-free" means the source data is not available during adaptation, only the trained source model. The proposal changes only how pseudo-labels are filtered.
Memory bank. Each epoch, the teacher's decoder features are sampled from every target volume. Sampling is balanced per class and per volume, and is biased toward predicted boundaries, where errors concentrate. The samples are stored in a memory bank with their volume IDs and an approximate nearest-neighbor index. Exact all-pairs search costs $O(N^2)$ over voxels, which the research file notes is the main cost problem for graph methods, so the bank's per-volume budget is a controlled variable. The bank is rebuilt every epoch, so the neighborhoods change as the representation adapts. LGDA, by contrast, denoises once, offline.
Tuning without target labels. UPL-SFDA admits to choosing thresholds with labeled target data. Here, $k$ and the acceptance threshold are fixed by a rule set on source data only: a held-out source split with simulated intensity shifts. The same rule-based protocol is applied to every compared method. Sensitivity curves over $k$ and the threshold are reported after the fact, but they are not used for selection.
Diagnostic study. Before any method comparison, the motivating premise is measured directly. On target sets that have ground truth, used for evaluation only, the work computes how strongly a voxel's pseudo-label error correlates with errors among (a) its spatial neighbors, (b) its feature-space neighbors within the same volume, and (c) its feature-space neighbors from other volumes. The proposal predicts that (a) and (b) are much more correlated than (c), and that the gap grows with shift severity. If the gap does not appear, the idea is wrong regardless of downstream Dice.
Pathology analysis. Neighbor filtering can discard rare but real anatomy as if it were noise. Results are therefore stratified by diagnosis on M&Ms, which includes several cardiac diseases, so it is visible whether atypical hearts lose supervision disproportionately.
Baselines
Source model with heavy augmentation, no adaptation. This is the source model trained with heavy intensity augmentation and never adapted, following the recipe the M&Ms challenge found most effective: histogram, gamma, and noise augmentation, which cut the unseen-vendor drop from about 6% to 1.5%. The winning team reached roughly 0.88–0.92 left-ventricle Dice across vendors. Every adaptation method starts from this model, so any adaptation gain is measured over a strong starting point rather than a weak, unaugmented one. Beating it means a statistically reliable Dice gain on the held-out vendor and on the cross-sequence fetal task.
Plain confidence-threshold self-training. This applies a fixed softmax threshold, FixMatch-style, and is the naive self-training reference.
UPL-SFDA. UPL-SFDA builds pseudo-labels from agreement among several perturbed decoder heads. It reports +5.54 Dice points over the source model on M&Ms and +5.01 points on fetal brain HASTE→TrueFISP. It is the strongest published method on the 3D benchmarks used here. It will be rerun from the same augmented source model under the same rule-based tuning, which may lower its numbers relative to the paper; both the paper's tuning and the rule-based tuning will be reported.
UP2D. UP2D represents the prototype approach. On fundus it reports 95.17% optic-disc Dice on RIM-ONE-r3, and 96.61% disc and 86.61% cup Dice on Drishti-GS. It is the number to match on 2D fundus data.
LGDA. LGDA represents the spatial-neighbor approach. It reports 86.89% average Dice on RIM-ONE-r3 and 91.48% on Drishti-GS.
Key ablation. The proposal's code is run three times with only the neighbor pool changed:
- the voxel's own volume only;
- all volumes, including its own;
- other volumes only.
Compute and bank size are matched across the three. The proposal succeeds only if the other-volumes-only variant clearly beats the own-volume variant. A fourth variant replaces the neighbor vote with distance to a single class prototype, isolating local neighborhoods from global centroids.
Why not just…?
Why not just augment harder and skip adaptation? On moderate vendor shift this is a real threat. The M&Ms results suggest that augmentation alone may leave little room for improvement, and on M&Ms the proposal could tie with it. The research file's evidence against augmentation is for larger shifts: AMOS reports a drop of more than 20% mean Dice on unseen MRI scanners, and UPL-SFDA recovers 5–7 points on cross-sequence fetal MRI. The experiments report gains as a function of shift severity, so the region where adaptation helps is shown rather than assumed.
Why not just use class prototypes, as UP2D does? If every class forms one compact cluster in feature space, a prototype amounts to a cross-patient average, and it would match the proposal at lower cost. The proposal's advantage should appear only for classes that look different in different places, such as basal versus apical ventricle or multi-organ CT/MRI. The prototype ablation tests this. If prototypes tie on AMOS and M&Ms, the right conclusion is that the simpler prototype method is sufficient.
Why not just use ensemble agreement, as UPL-SFDA does? Perturbed decoder heads share an encoder and see the same scan, so their agreement inherits that scan's systematic error, which is the same concern raised about spatial neighbors. The diagnostic study measures error correlation for ensemble agreement as well. If ensemble errors turn out to be as independent as cross-patient errors, the proposal's rationale weakens.
Why not just calibrate the frozen model? Post-hoc calibration leaves calibration error at 15.37% under the bSSFP→LGE sequence change, even with shape priors. Calibration also only marks which predictions are doubtful; it does not change them. Calibrated confidence is nonetheless included as a filtering rule within the self-training loop, so it is tested on equal footing.
Proposed Experiments
Datasets. Four adaptation settings are used:
- M&Ms cardiac MRI: train on Siemens, adapt to Canon as the unseen vendor and to Philips and GE.
- Fetal brain MRI, HASTE→TrueFISP, and FeTA across reconstruction methods.
- AMOS abdominal multi-organ, CT→MRI, as the hardest shift.
- REFUGE→RIM-ONE-r3 and REFUGE→Drishti-GS fundus, so numbers are directly comparable with UP2D and LGDA.
Metrics. Per-structure Dice and average symmetric surface distance on target test sets, averaged over three seeds. Pseudo-label precision and recall per epoch. The error-correlation diagnostic. Dice stratified by pathology.
Decisive checks. There are three:
- The diagnostic must show lower error correlation across volumes than within a volume.
- The other-volumes-only ablation must beat the own-volume ablation at matched compute.
- On AMOS and FeTA, the method must beat both UPL-SFDA and the prototype variant under identical tuning without target labels.
Expected failure. On AMOS CT→MRI, the features may be so mixed that cross-patient neighbors are wrong too, which is the representation dependence SimiFeat already warns about. In that case, the fraction of pseudo-labels the vote accepts and their measured accuracy will show the failure. That result would be reported as the method's limit, not hidden.
References
- Detecting Corrupted Labels Without Training a Model to Predict
- Combating Noisy Labels through Fostering Self- and Neighbor-Consistency
- Local-global pseudo-label correction for source-free domain adaptive medical image segmentation
- UP2D: Uncertainty-aware Progressive Pseudo-label Denoising for Source-Free Domain Adaptive Medical Image Segmentation
- Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Segmentation: The M&Ms Challenge
- UPL-SFDA: Uncertainty-Aware Pseudo Label Guided Source-Free Domain Adaptation for Medical Image Segmentation
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation