When a segmentation model trained at one hospital loses accuracy at another, the field blames the scanner. The model's Dice score, an overlap measure between predicted and reference masks, falls, and the drop is reported as "domain shift." But the reference masks at the new hospital were drawn by different people trained in a different clinical school, who may outline the same prostate a millimetre more generously or include the seminal-vesicle junction where the source site did not. Part of the measured drop is therefore not the model failing on new images. It is the model correctly reproducing one site's drawing convention and being scored against another's. This proposal does two things. It measures how much of the cross-site gap is annotation convention rather than image appearance. It then builds a segmenter whose drawing convention can be re-fitted from a handful of target-site cases without touching the parts of the model that handle image appearance.
The claim is concrete. Attach a small, low-dimensional "convention code" to a medical foundation model, so that it enters only the mask decoder while the image pathway is adapted separately. At a new site, a large share of the cross-site gap on MRI can then be recovered by fitting only that code from one to five labelled volumes. The fraction recovered this way gives a direct, interpretable estimate of how much of the "domain shift" was never about the scanner. If the estimate is near zero, that is also worth knowing. Domain generalization papers would then be justified in treating target labels as clean, which none of them currently checks.
Why this gap is real
The file shows two literatures that do not meet. Domain generalization work treats target annotations as ground truth and attributes all degradation to acquisition physics. The scanner-shift study of Szumel et al. partitions public data by manufacturer and finds MRI far more fragile than CT, with an average AUC drop of 0.097 and a prostate collapse from 0.791 to 0.505. Leveraging SAM for Single-Source Domain Generalization auto-prompts a SAM decoder from a coarse ResNet mask and reports 79.54% average Dice over five unseen prostate MRI sites. Do Vision Foundation Models Enhance Domain Generalization shows that parameter-efficient fine-tuning, meaning training small adapter modules while freezing the large pretrained network, beats UNets under shift. These papers motivate the work, and all three score against each site's own masks. The second literature models rater disagreement, but always within a single site. Pionono represents each annotator as an 8-dimensional Gaussian whose mean is systematic bias and whose variance is inconsistency. The Latent Doctor Model does the analogous thing with a conditional variational autoencoder. Pionono supplies our core mechanism. We move its rater code from "which pathologist on this team" to "which site's convention," and we test whether the code stays separate from scanner appearance, which Pionono never examines.
CD-Net is the closest work that touches label noise and domain shift together. It acts as a constraint on our design rather than a template. CD-Net downweights boundary pixels as corrupted, uses synthetic dilation and erosion noise, and adapts to an unlabelled target. Its approach throws away exactly the systematic boundary offset we want to estimate. It also cannot distinguish a convention from noise, because a convention is consistent, and that consistency is what makes it recoverable from few cases.
Research Program
The model starts from the MedSAM encoder (Segment anything in medical images). This choice addresses the natural-to-medical feature mismatch, so that boundary sensitivity is not the bottleneck being studied. The encoder is frozen except for Ladder-style adapters, which handle image appearance. Each training site $s$ gets a convention code $z_s \in \mathbb{R}^8$, drawn from a learned Gaussian as in Pionono. The code modulates only the mask decoder, through feature-wise scale and shift. We restrict it to the decoder on purpose. Image appearance should flow through the encoder pathway, while "how tightly do you draw" is a property of the output contour.
Training is leave-one-site-out across the six-site prostate T2 MRI benchmark used by the SAM domain-generalization paper. Each held-out site serves in turn as the unseen target. At test time there are two regimes. In the zero-shot regime, the model uses the population-mean code. In the few-shot regime, $k \in \{1, 3, 5\}$ labelled target volumes are used to fit only $z_{\text{target}}$, with every other weight frozen. That means eight numbers estimated from data, which is why we expect it to work from one or two cases.
The decomposition needs a reference that is free of convention. For this, we re-annotate ten volumes per site under a single written protocol by one annotator. To keep the cost low, we use the sparse-prompting workflow of Kordt et al.. In that workflow, boxes are drawn on one slice in three and interpolated, and it reported 0.831 Dice against 0.848 for fully prompted slices on the Medical Segmentation Decathlon. Every slice is then corrected by the annotator, so the protocol labels are not model outputs. For a model trained on sources and tested on target $t$, define $D_t^{\text{site}}$ as Dice against the site's own masks and $D_t^{\text{prot}}$ as Dice against protocol masks on the same volumes. Define $D^{\text{in}}$ as in-domain Dice. The share of the gap attributable to convention is then
$$ \rho_t = \frac{D_t^{\text{prot}} - D_t^{\text{site}}}{D^{\text{in}} - D_t^{\text{site}}}, $$
where both $D^{\text{in}}$ and $D_t^{\text{prot}}$ are evaluated against protocol-consistent references. Before computing $\rho_t$, the protocol masks are compared against each source site's original masks on its own data. This step confirms that the protocol is a neutral reference and not just a copy of one site's style.
We supplement Dice with mean signed surface distance, which is positive when a contour is more generous than the reference. A convention shift shows up as a consistent signed offset with intact topology. Scanner failure shows up as missed regions, fragments, and large unsigned errors. Fitting $z$ should move the first quantity and leave the second unchanged.
As a semi-synthetic control with known ground truth, we take one source site and impose a fixed contour convention on its labels: uniform 1 to 2 mm dilation, or exclusion of the apex slices. We then check that the fitted code recovers the imposed offset and that the image adapters do not absorb it. As a secondary setting with real multi-rater labels, we use the QUBIQ prostate task (QUBIQ), scored with its threshold-staged Q-Dice. There we test whether site-level codes and rater-level codes can coexist in a hierarchy.
Baselines
The naive reference point is a MedSAM-adapter model trained on pooled source-site masks with no convention code. It is evaluated zero-shot on the held-out site. Every other number is read relative to it. Majority-vote or STAPLE training is the natural variant where several raters exist, as in QUBIQ.
The strongest existing method on this benchmark is the auto-prompted SAM of Leveraging SAM for Single-Source Domain Generalization. It reports 79.54% average Dice over unseen prostate sites, 8.12 points above the prior best, ACSDG. We rerun its released recipe on our splits. Because it is single-source, we report it both under its own protocol and with pooled sources. On the broader multi-center benchmark, the HQHSAM decoder of Cekmeceli et al. reports 52.63 average domain-generalization Dice. We use its PEFT recipe as our image-pathway baseline. For few-shot adaptation by convention code, we found no reported result on Lacuna. Beating the strongest method in the zero-shot regime is not our claim. Our claim is that, in the few-shot regime, fitting the code closes more of the gap per labelled volume than any alternative, and that it barely moves the evaluation against protocol masks. The second condition shows it corrected the convention and did not overfit the site.
The most important competitor is few-shot fine-tuning of the same model with the same $k$ target volumes. We test three versions: LoRA on the encoder adapters, decoder-only fine-tuning, and full adapter fine-tuning. Each gets the same optimizer budget and the same validation-based early stopping on a held-out target volume. All methods share the backbone, the sources, the target cases, and a hyperparameter search of twenty trials.
Two ablations isolate the key ingredient. The first injects the code into the encoder as well, which should let it absorb appearance and degrade the decomposition. The second fits a randomly initialized code with no learned prior, which shows how much the Gaussian structure of Pionono matters at $k=1$.
Why not just…?
Why not just fine-tune the whole adapter on a few target cases? This is the real threat, and the experiments are designed around it. If LoRA fine-tuning on five volumes matches code fitting on site-mask Dice, the label-efficiency claim fails. Two differences remain testable even then. First, at $k=1$, an eight-number fit should be far less prone to overfitting than thousands of adapter weights. Second, fine-tuning has no way to tell whether it fixed scanner appearance or learned the site's drawing habit, whereas code fitting answers that by construction. If fine-tuning wins at every $k$, we will report the decomposition $\rho_t$ as the contribution and drop the adaptation claim.
Why not just harmonize intensities or augment harder? Szumel et al. already found that Gaussian-noise augmentation does not close cross-scanner MRI gaps and often hurts in-domain accuracy. More to the point, no image-space transform can change where a target-site radiologist chooses to draw the apex boundary. We include a strong intensity-augmentation baseline anyway. We predict it reduces unsigned and topological errors while leaving the signed surface offset unchanged, and the signed-distance analysis will show whether that prediction holds.
Why not just use a boundary-tolerant metric? Distance-tolerant Dice or Q-Dice would hide convention offsets in evaluation. They would not remove those offsets from a deployed contour feeding a radiotherapy plan, where a consistent 1.5 mm offset matters clinically. We report tolerant metrics alongside strict ones. If tolerant scoring alone makes $\rho_t$ vanish, the conventions are clinically negligible, and we will say so.
Why not suppress boundary pixels as CD-Net does? CD-Net treats disagreement at the boundary as random corruption. A site convention is systematic, so downweighting it discards a learnable signal and leaves the offset in place. We run CD-Net's noise-tolerant loss on the semi-synthetic control. We predict it will fail to recover the imposed dilation, while the convention code recovers it.
Decisive Checks
The work succeeds if three things hold. On the semi-synthetic control, the fitted code recovers the imposed contour offset to within 0.5 mm, while the image adapters stay invariant. On real held-out sites, $\rho_t$ is measurably above zero on at least some MRI sites, and code fitting closes that portion at $k \le 3$ with no loss against protocol masks. Code fitting also matches or beats few-shot LoRA at $k=1$. The work fails, and we will say so, if $\rho_t$ is indistinguishable from zero across sites, or if the code absorbs appearance, which would show up as improvement on site masks and degradation on protocol masks. Either outcome settles a question the domain-generalization literature currently assumes away.
References
- The Impact of Scanner Domain Shift on Deep Learning Performance in Medical Imaging: an Experimental Study
- Leveraging SAM for Single-Source Domain Generalization in Medical Image Segmentation
- Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation?
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Cross-denoising Network against Corrupted Labels in Medical Image Segmentation with Domain Shift
- Segment anything in medical images
- Efficient Semi-automatic Segmentation-Labeling of Any Volumetric Medical Image
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge