A segmentation model trained on labeled MRI scans of the heart usually fails badly on CT scans of the same organ. Unadapted, it reaches about 17% Dice overlap on the standard benchmark. Expert labels for the new modality are expensive, so the field relies on unsupervised domain adaptation: the model learns from labeled scans in one modality and unlabeled scans in the other. The most reliable ingredient in recent methods is class-conditional alignment. Each anatomical class gets a "prototype," an average feature vector, and target-modality features are pulled toward the prototype of their (guessed) class and pushed away from the others. This prevents a known failure in which simply matching the overall feature distributions of the two modalities maps myocardium onto blood pool. The problem is that these prototypes are a single vector per class for the whole body. Myocardium near the apex of the heart does not look like myocardium at the base, and a vestibular schwannoma (a small tumor near the inner ear) looks different where it fills the internal auditory canal than where it bulges outward. Collapsing every slice of an organ onto one point improves bulk overlap but blurs boundaries. That cost shows up in the numbers: the prototype method of Ye et al. raises Dice to 77.5% on whole-heart MRI→CT, yet its reported average surface distance is 5.1, worse than plain entropy minimization at 4.6.
We propose to make each class prototype a smooth function of position in an anatomical coordinate system, rather than a single vector. The coordinate is measured relative to structures that adapt reliably across modalities, not by raw slice number. Target features are then compared only to source features of the same class at the same anatomical location, and are pushed away from the competing classes found at that location. The same coordinate gives a label-free sanity check on pseudo-labels: a voxel labeled "aorta" at a position where the aorta never appears in any source scan is suspect. The intended result is a 3D cross-modality segmenter that keeps the Dice gains of class-conditional alignment without losing boundary accuracy. It should also degrade more gracefully when source labels are scarce, because the position-indexed prototypes carry anatomical prior knowledge that one global centroid cannot.
Core Claim
Class-conditional alignment in volumetric cross-modality segmentation has been formulated with the wrong unit of comparison. The right unit is "class $k$ at anatomical position $u$," not "class $k$." We claim that indexing prototypes by a modality-invariant anatomical coordinate does three things. It recovers the surface-distance accuracy that global prototypes lose. It makes the contrastive negatives local and therefore informative: myocardium is contrasted with the left-ventricle cavity at the same height, not with a distant aorta. It also yields a position-plausibility filter that cuts the confirmation bias of self-training. We further claim that these gains survive on top of a strong native-3D self-training pipeline, which is the test that matters. A gain over a 2D baseline alone would not be interesting.
This is distinct from three neighbors in the file. SDC-UDA attends across three adjacent slices during image translation to remove slice-to-slice flicker. Its pseudo-labels are refined with a local intensity-similarity heuristic, and its feature space contains no class-conditional structure. Ye et al. supply the method we build on, class prototypes with rolling memory dictionaries that handle classes missing from a slice, but their prototypes are 2D and position-blind. Positional contrastive learning shows that through-plane position is a strong self-supervised signal. It uses whole-slice positives indexed by the normalized slice number $m/n$, has no class structure, and its own limitations note that this index breaks when field of view varies across scans. Our coordinate is built specifically to avoid that failure.
Research Program
Backbone. The base model is a 3D segmentation network in the style of FPL+. It trains on CycleGAN-translated source volumes and on real target volumes. All convolution weights are shared across modalities, and each modality has its own batch-normalization statistics. Self-training uses FPL+'s size-aware uncertainty filtering. We choose this backbone deliberately. It is the strongest reproducible pipeline in the file, and building on it means any gain cannot be explained as "we finally used 3D." It also answers the volumetric-context direction with native 3D processing. The file contains evidence on both sides of the 3D-versus-slices question. The whole-heart challenge reports that deep networks trained on about 40 volumes were unstable on MRI. Meanwhile, the CrossMoDA winners used 3D nnU-Net self-training. We treat this as an empirical question and also run a 2D variant of the backbone.
Anatomical coordinate. For each volume we define $u \in [0,1]$ along the principal axis of an anchor structure. On the whole-heart data, the anchor is the union of all predicted cardiac foreground. Foreground-versus-background is far easier to adapt than the four-way split between substructures. On CrossMoDA, the anchor is the ipsilateral cochlea, which the challenge found adapts nearly as well as full supervision (median Dice 85.7% unsupervised versus 87.7% supervised). The tumor is located relative to it by signed distance along the canal axis. On source volumes the anchor comes from ground truth; on target volumes it comes from the current pseudo-labels. This choice follows the challenge's observation that rigid structures transfer while pathology does not, and it turns that asymmetry into a coordinate frame.
Position-indexed prototypes. Let $f_s$ denote the encoder feature of a labeled source voxel with class $y_s$ and coordinate $u_s$, drawn from a rolling memory bank as in Ye et al. The prototype of class $k$ at coordinate $u$ is a kernel-weighted average:
$$c_k(u) = \frac{\sum_{s:\,y_s=k} K_h(u-u_s)\, f_s}{\sum_{s:\,y_s=k} K_h(u-u_s)},$$
Here $K_h$ is a Gaussian kernel with bandwidth $h$. The bandwidth makes $c_k$ vary smoothly along the anatomy by construction; this smoothness is the volumetric constraint the flat prototypes lack.
Alignment loss. A target voxel with feature $f_t$, pseudo-label $k$ and coordinate $u_t$ receives a contrastive loss:
$$\mathcal{L}_{t} = -\log \frac{\exp(\cos(f_t, c_k(u_t))/\tau)}{\sum_{j \in A(u_t)} \exp(\cos(f_t, c_j(u_t))/\tau)}.$$
In this loss, $\tau$ is a temperature, and $A(u)$ is the set of classes with nonzero source kernel mass at $u$. Restricting the denominator to $A(u)$ is the mechanism by which negatives become local and anatomically meaningful.
Plausibility filter. If $k \notin A(u_t)$, the voxel's pseudo-label claims a class at a position where it never occurs in the source data. Such voxels are dropped from both this loss and the self-training cross-entropy. This filter is combined with FPL+'s existing uncertainty filter. The whole procedure adds no new networks, only a memory bank and a kernel average.
Analyses. We will measure how much the per-class prototype actually moves along $u$ in each modality. If it barely moves, the premise is false. We will also measure how often the plausibility filter removes voxels that are truly wrong versus truly right, using held-out target labels that are reserved for analysis only.
Baselines
The naive reference point is the source-only model: the same 3D backbone trained on labeled source data and applied directly to the target modality. On whole-heart MRI→CT, SIFA reports 17.2% Dice for its source-only model. We will rerun ours on our split rather than borrow that number.
The class-agnostic adaptation reference is SIFA itself, which combines image translation with adversarial alignment in the output space. It is reported at 73.0% Dice on whole-heart MRI→CT, 33.5% mean Dice on CrossMoDA in the SDC-UDA paper, and 67.59% Dice on bSSFP→LGE cardiac MRI in SGDR.
The class-conditional reference is Ye et al.'s global-prototype method: 77.5% Dice and ASD 5.1 on whole-heart MRI→CT. Alongside it we run entropy minimization, reported at ASD 4.6 in the same paper. Beating Ye et al. means exceeding their Dice while bringing surface distance at or below the entropy-minimization level. That would show the Dice-versus-boundary trade-off is an artifact of global prototypes.
SGDR, a content-style disentanglement method, reports the highest whole-heart number in the file at 80.18%. It used a different split (52 CT and 46 MR images), so we will rerun it on the standard 20/20 split before comparing.
The strongest competitor overall is the FPL+-style 3D self-training pipeline, which is also our backbone without the new loss and filter. FPL+ reports 91.98% Dice on hrT2→ceT1 vestibular schwannoma. On the standard ceT1→hrT2 CrossMoDA direction, the bar is the challenge winner's median Dice of 87.0% on the tumor and 85.7% on the cochlea, and SDC-UDA's 84.8% mean Dice.
Fair comparison. Every method uses the same translated volumes, the same splits, the same number of self-training rounds, and an equal hyperparameter-search budget of the same number of trials on the same validation scans. Rerun numbers are reported next to the published ones.
Ablations. Three ablations isolate the key ingredient. The first replaces $c_k(u)$ with a single global $c_k$ (Ye-style prototypes on the 3D backbone). The second keeps an equal number of prototypes per class but obtains them by unordered k-means clustering, with no coordinate. The third keeps everything but randomly permutes $u$ across voxels.
Why not just…?
Why not just feed the slice position or coordinate to the network as an extra input channel? This is the cheapest way to give the model positional information, and it is a real threat. A 3D network with a coordinate channel might learn position-dependent features on its own. However, an input coordinate does not change the alignment target: the network can still satisfy a global prototype loss by collapsing classes, and nothing tells it which classes are plausible where. We will run the global-prototype model with a coordinate input channel as its own baseline. If it matches our method on both Dice and surface distance, the contribution reduces to "use position somewhere," and we will report that honestly.
Why not just use several unordered prototypes per class? Multi-prototype clustering also captures within-class variation, and Ye et al.'s dictionaries could be extended this way at no conceptual cost. Unordered clusters, however, have no defined correspondence across modalities. A CT cluster and an MRI cluster of "myocardium" can pair the apex with the base. Unordered clusters also give no plausibility filter. The equal-count k-means ablation tests this directly. Our claim requires the coordinate-indexed version to win, especially on surface distance and on the scarce-label settings.
Why not just run the strong 3D translate-then-self-train pipeline and stop? CrossMoDA showed this recipe is hard to beat, with the winner within a few points of full supervision on the cochlea. Prototypes may therefore add nothing at the top end. Our response is to build on that pipeline rather than compete against it. We also expect the gains to concentrate where the pipeline is weakest: the pathological tumor class, which the challenge found highly variable across methods, and the regime with few source labels. If the added loss does not move the FPL+-style backbone on either, the honest conclusion is that class-conditional alignment is redundant once 3D self-training is strong.
Proposed Experiments
The primary benchmark is whole-heart MRI→CT and CT→MRI on the standard 20/20 subset, with four classes: ascending aorta, left atrium cavity, left ventricle cavity and myocardium. Here we report Dice, average surface distance and Hausdorff distance. The second benchmark is CrossMoDA ceT1→hrT2 (105 labeled source, 105 unlabeled target), on which we report Dice and average symmetric surface distance for tumor and cochlea separately, plus the rate of complete tumor failures (Dice below 10%). The third is bSSFP→LGE cardiac MRI, as an intra-MRI shift.
We add a slice-coherence measure: the mean absolute change in predicted per-class cross-sectional area between adjacent slices, normalized by structure size. This lets us test whether position-indexed prototypes reduce the flicker that SDC-UDA targets.
To model the annotation bottleneck, we repeat the whole-heart experiments with 20, 8, 4 and 2 labeled source volumes, echoing the two-patient regime of positional contrastive learning. No reported result was found on Lacuna for cross-modality adaptation at reduced source-label counts, so all such baselines will be run by us.
Four checks are decisive. First, on whole-heart MRI→CT, our method must exceed the global-prototype ablation in Dice and reach surface distance at or below entropy minimization under identical conditions. Failing that refutes the core claim. Second, the coordinate-permutation control must erase most of the gain. If it does not, geometry is not doing the work. Third, on CrossMoDA, gains on top of the FPL+-style backbone should appear mainly on the tumor and in the failure rate, with little change on the cochlea, which is already near its ceiling. Fourth, the advantage over global and unordered prototypes should grow as source labels shrink from 20 to 2 volumes. That growth is the prediction that ties the method to the annotation bottleneck. If it is absent, the method is a boundary-quality refinement rather than a label-efficiency tool, and we would present it as such.
References
- Unsupervised Domain Adaptation via Similarity-based Prototypes for Cross-Modality Segmentation
- SDC-UDA: Volumetric Unsupervised Domain Adaptation Framework for Slice-Direction Continuous Cross-Modality Medical Image Segmentation
- Positional Contrastive Learning for Volumetric Medical Image Segmentation
- FPL+: Filtered Pseudo Label-Based Unsupervised Cross-Modality Adaptation for 3D Medical Image Segmentation
- Evaluation of Algorithms for Multi-Modality Whole Heart Segmentation: An Open-Access Grand Challenge
- CrossMoDA 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation
- Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation
- SGDR: Semantic-guided Disentangled Representation for Unsupervised Cross-modality Medical Image Segmentation