Matching Only the Scanner: Selective Sliced Transport for Frozen Medical Segmenters

Medical segmentation networks lose accuracy when they meet a scanner they were not trained on. On the M&Ms cardiac benchmark, unadapted models drop about 6% on unseen vendors. On AMOS, the drop on unseen MRI scanners exceeds 20%. The usual fix from domain adaptation is to make features from the new scanner look statistically like features from the training scanners, often by minimizing a Wasserstein distance. A Wasserstein distance is the minimum cost of moving one cloud of points onto another. That fix has two problems in the clinic. First, certified models often cannot be retrained. Second, there is evidence that forcing whole distributions to match is the wrong target. A clinical adapter that worked very well actually moved the data further from the source in Wasserstein distance (Adapt the Data, Not the Model). A new hospital also differs from the training data in real ways that should be kept, such as its mix of pathologies and therefore its mix of chamber and organ sizes. Matching everything erases that.

We propose to match only the part of the difference that is due to the scanner. The tool is sliced optimal transport, which compares two point clouds by projecting them onto many one-dimensional lines. On a line, the optimal matching is just "sort both sets and pair them up," so each comparison costs $O(n\log n)$. Because each line can be scored on its own, we can ask a question that full transport cannot answer cheaply: along which directions of the frozen network's features do two known vendors differ within the same anatomical label, while the labels themselves stay close together? Those directions carry scanner appearance, not anatomy. At deployment, the segmentation network stays frozen. We fit a small input-side transform on unlabeled scans from the new site, and it minimizes sliced transport distance only along those scanner directions. Everything else, including the site's class proportions, is left free.

Core Claim

A label-free, frozen-model adaptation that aligns target features to source features only along within-class, between-vendor sliced directions should give three results. It should beat both no adaptation and full sliced alignment on unseen vendors. It should stay stable when the target site's pathology mix differs from the source. It should help most on the structures where augmentation-based methods still fail, namely the right ventricle, the myocardium, apical and basal slices, and small abdominal organs. The claim is falsifiable. If a well-augmented nnU-Net leaves no residual gap on these structures, or if full alignment does just as well, the idea is wrong.

Why This Differs from Existing Work

The closest work is the Sliced Wasserstein Discrepancy method (Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation). It projects the outputs of two disagreeing classifiers onto $M$ random lines, with performance flattening near $M=128$. It uses the resulting 1D distances as a training loss, which requires retraining with target data. It reached 76.4% on VisDA 2017 versus 71.9% for Maximum Classifier Discrepancy. It aligns in all directions equally and was only tested on natural images. Domain Adaptive Semantic Segmentation by Optimal Transport replaces adversarial discriminators with entropic Sinkhorn transport on output maps. It reports 44.3 mIoU on GTA5→Cityscapes, also on driving scenes, and it also matches globally. We keep the sliced machinery from the first paper. We change three things: the network is frozen, the data are 3D medical volumes, and alignment is restricted to a learned set of directions.

The idea of choosing informative directions comes from max-sliced and projection-pursuit work. Principal Differences Analysis finds sparse 1D directions that maximize the Wasserstein distance between two distributions. Subspace Robust Wasserstein Distances restricts transport to the $k$-dimensional subspace where transport cost is largest, arguing that full high-dimensional transport mostly measures sampling noise. Statistical, Robustness, and Computational Guarantees for Sliced Wasserstein Distances gives a subgradient-ascent algorithm for the max-sliced direction, with stationary-point guarantees. It also shows that averaged slices have dimension-free estimation error. That matters here because we estimate distances from a few thousand subsampled voxels per scan. Both direction-finding methods are unsupervised about what the difference is. Our contribution is a class-conditional criterion that separates nuisance directions from anatomical ones.

The motivating negative result is the input-adapter finding in Adapt the Data, Not the Model. That adapter improved acute kidney injury prediction by +14.1 AUCPR on eICU while increasing Wasserstein distance to the source, because it repaired site-specific artifacts rather than matching global statistics. We read this as a constraint on design: an alignment objective must target the artifact, not the whole distribution. Their adapter needed target labels and worked on time series. Ours needs no target labels and works on images.

Method

Let $f$ be the frozen network's feature map at an early decoder stage. A voxel with label $c$ from vendor $v$ gives a feature $z\in\mathbb{R}^d$. For a unit direction $\theta$, let $W_\theta(P,Q)$ be the 1D Wasserstein distance between the projections $\theta^\top z$ under two distributions $P$ and $Q$, computed by sorting. On the labeled source vendors $a$ and $b$ (for example Siemens and Philips in M&Ms), we score each direction by how much it separates vendors within a class, relative to how much it separates classes:

$$s(\theta)=\frac{\sum_c \pi_c\, W_\theta(P_{a,c},P_{b,c})}{\sum_{c\neq c'} W_\theta(P_{c},P_{c'})+\epsilon},$$

Here $\pi_c$ is the voxel fraction of class $c$, $P_{c}$ pools both vendors, and $\epsilon$ is a small constant. We keep a bank $\Theta$ of $K$ high-scoring directions. We find them by projected subgradient ascent on $s$ from many random starts, following the max-sliced algorithm, and add random directions that pass a score threshold. With more than two source vendors, the numerator sums over vendor pairs.

At a new site we fit a lightweight input transform $g_\phi$. It is a monotone per-scan intensity curve plus a shallow residual convolution, applied before the frozen network. We fit it on unlabeled target scans by minimizing

$$\mathcal{L}(\phi)=\frac{1}{K}\sum_{\theta\in\Theta} W_\theta\big(f(g_\phi(x_{\text{tgt}})),\,f(x_{\text{src}})\big)+\lambda\,\|g_\phi(x)-x\|^2,$$

The source features $f(x_{\text{src}})$ come from a stored memory of subsampled source voxel features, and $\lambda$ keeps the transform close to identity. The frozen segmenter never changes. This is compatible with the locked-model setting.

Why should this be robust to pathology mix? By construction, directions in $\Theta$ separate classes poorly. So a change in how much of each class the target contains barely moves the target's marginal along $\theta$. Full-direction alignment has no such protection: it will push a dilated-cardiomyopathy-heavy site toward the source's ventricle sizes.

Research Program

The work has three parts. First, a diagnostic study on the source vendors. Does $s(\theta)$ find a small set of directions at all, meaning the score spectrum decays sharply? Does the per-direction distance along $\Theta$ between a held-out vendor and the source predict that vendor's Dice drop better than full sliced distance does? Second, the test-time adaptation experiments below. Third, a sensitivity analysis over the number of directions $K$, the choice of feature layer, and voxel subsampling. The Monte Carlo variance question raised in Asymptotic Guarantees for Learning Generative Models with the Sliced-Wasserstein Distance becomes an empirical question here: how many target voxels and scans are needed before the adapted Dice stabilizes?

Baselines

The naive reference is the frozen segmenter applied unchanged to the unseen vendor. On M&Ms, the challenge reports roughly a 6% drop for unadapted models (M&Ms Challenge). On AMOS, the reported drop on unseen MRI scanners exceeds 20% (AMOS). We rerun these numbers ourselves on the same splits rather than quoting them.

The strongest existing method is nnU-Net trained with heavy intensity augmentation, which is what the top M&Ms teams used. The winning team reached a left-ventricle Dice of about 0.88 to 0.92 across vendors and cut the unseen-vendor gap to 1.5% (M&Ms Challenge). Our frozen segmenter is this augmented nnU-Net, trained with the same pipeline and compute. Every adaptation method is therefore judged on top of the strongest baseline, not instead of it. Beating it means a statistically significant Dice gain on the unseen vendor for the right ventricle and myocardium, and on apical and basal slices, beyond seed-to-seed variation. We do not expect a gain on left-ventricle Dice, which is already near ceiling.

Histogram matching of each target scan to the source intensity distribution is the zero-parameter one-dimensional optimal transport baseline. It tests whether slicing in feature space adds anything over transport in raw intensity. Full sliced alignment, with the same transform $g_\phi$ and loss but $\Theta$ replaced by $K$ uniformly random directions, is the key ablation. It isolates the direction-selection ingredient. We also run an ablation that uses max-sliced directions between vendors pooled over classes, which isolates the class-conditional part of the criterion.

The Sliced Wasserstein Discrepancy method, retrained with M&Ms' unlabeled vendor-C scans, is included as a retraining upper reference. It uses the same backbone, $M=128$ projections, and the same tuning budget. Its reported VisDA 2017 number (76.4%) is not transferable, and we found no reported result on Lacuna for it on M&Ms or AMOS.

All adaptation methods get the same unlabeled target scans and the same number of adaptation steps. Hyperparameters are tuned only on a held-out source vendor treated as pseudo-target, never on the true target.

Why not just…?

Why not just rely on nnU-Net with intensity augmentation? This is the real threat. M&Ms shows it closes the left-ventricle gap to 1.5%. But that figure is for the easiest structure. The same challenge reports persistent failures on unseen Canon scans and on apical and basal slices, and AMOS reports its largest failures on small organs and unseen MRI scanners. Because our method runs on top of the augmented nnU-Net, the experiment directly measures whether any residual gap is left for adaptation to remove. If our method gives no gain over the augmented model on these structures, the result says augmentation suffices, and we will report that.

Why not just histogram-match the intensities? Histogram matching is exact 1D transport of raw intensity. It cannot correct shifts that show up only in texture, noise, or contrast between tissues, which is where proprietary reconstruction differs across vendors. It also assumes the target anatomy mix matches the source, the same flaw as full alignment. It is run as a baseline. If it matches our method, the feature-space machinery is unnecessary.

Why not align all directions, as the Sliced Wasserstein Discrepancy method does? Global matching penalizes legitimate differences between sites. The time-series evidence (Adapt the Data, Not the Model) shows successful adaptation need not reduce global Wasserstein distance. The controlled pathology-mix experiment below is designed to show exactly where full alignment breaks and selective alignment does not.

Why not use pooled multi-center training, as AMOS's joint CT and MRI training does (+2.14% mDice on MRI)? Pooling helps when more vendors can be collected. The unseen-vendor gap is by definition about the scanner not collected. Pooling is complementary: more source vendors give a better estimate of $\Theta$. We test this by varying the number of source vendors used for direction estimation.

Proposed Experiments

On M&Ms, the segmenter is trained on labeled vendors A and B, and $\Theta$ is estimated from A versus B. Adaptation runs on unlabeled scans of vendor C and of the strictly unseen vendor D, using a disjoint subset of each vendor's scans for adaptation and evaluation. On AMOS, we hold out one MRI scanner system as the target, contingent on the per-scan scanner metadata being usable. $\Theta$ comes from the remaining scanners, and we report per-organ Dice with emphasis on the adrenal glands, pancreas, and duodenum. Metrics are per-structure Dice and 95th-percentile Hausdorff distance, stratified by slice position for cardiac data, with bootstrap confidence intervals over patients.

The first decisive check is a controlled pathology-mix shift. From the vendor-D test pool we build target subsets that are heavy in dilated cardiomyopathy and subsets that are heavy in hypertrophic cardiomyopathy. The prediction is that full sliced alignment loses Dice as the mix departs from the source, while selective alignment stays flat. If both stay flat, class-conditional selection is not needed.

The second decisive check is predictive. Across held-out vendors and scanners, the sliced distance restricted to $\Theta$ should correlate more strongly with the frozen model's Dice drop than the full sliced distance does. If it does not, the directions are not capturing the harmful part of the shift. A third, cheaper check verifies that the adapted inputs leave between-class distances in feature space largely unchanged, which confirms that anatomy was not the thing being moved.