Class-Conditioned Scale Selection for CT-to-MRI Segmentation

Segmentation networks trained on computed tomography (CT) scans lose a lot of accuracy when applied to magnetic resonance images (MRI) of the same anatomy. The AMOS benchmark reports drops of over 20% on unseen MRI scanner distributions (AMOS). Two architectural ideas have helped on hard single-modality targets. The first is adaptive receptive fields, in which the network picks, for each input, how wide a window of surrounding voxels each filter looks at. The second is adaptive feature fusion, in which the network weights shallow and deep features by how informative they look. Both decide their weights from the input's own statistics. Under a change of modality those statistics are exactly what changes: CT intensities follow tissue density, MRI intensities follow relaxation times. We suspect these input-driven gates are a hidden weak point under modality shift. Trained on CT, they learn to respond to CT texture and contrast, and on MRI they may select the wrong scale and the wrong layers.

We propose moving the adaptivity from the input to the class. The network ends in a prototype head instead of a learned linear classifier. A prototype is a stored reference feature vector for each organ, and each voxel is labeled by its similarity to these vectors. Each organ's prototypes are kept separately at several receptive-field scales and several network depths, and each organ carries its own learned weights over those scales and depths. The adrenal gland learns to be matched with narrow windows and mid-level features. The liver learns wide windows and deep features. Because these weights are fixed per class after training, an MRI scan cannot shift them. The claim is that this keeps the gains adaptive receptive fields bring on small and irregular structures while removing the modality sensitivity of input-conditioned gating.

Core Claim

The prior work splits along a clear line. Dynamic multi-scale convolutions such as DMC-Net weight parallel 3×3 and 5×5 branches by channel attention computed from the input, and reach 60.71% Dice on pancreatic tumors against roughly 54% for nnU-Net (DMC-Net). The authors note that it was validated only on CT. CFPNet-M fuses dilated branches hierarchically and is tested on five modalities, but on each one separately, never with transfer between them (CFPNet-M). These papers motivate the proposal: scale adaptivity matters for small structures, but no one has asked whether it survives a modality change.

Prototype segmenters handle scale on the classification side instead. ASGNet allocates a variable number of superpixel-derived prototypes over a fixed backbone (ASGNet). ALPNet-style local prototype pooling on DINOv2 works well within CT and within MRI separately, but was never trained on one and tested on the other (DINOv2 Few-Shot). Sim-MPNet keeps a memory bank of prototypes initialized by K-means, again per modality (Sim-MPNet). ProtoKD transfers pixel-to-prototype similarity maps between MRI sequences on BraTS, but its prototypes are single global class means and it never crosses physical modalities (ProtoKD). These papers supply the method, metric-based classification over local prototypes. None of them couples prototypes to receptive-field choice.

On the cross-modality side, the closest work is the causal single-source method of Chen et al. It trains on CT only. It fine-tunes Stable Diffusion with DreamBooth and ControlNet to render CT anatomy in other modality styles, then aligns features with a patch-wise contrastive loss (InfoNCE) feeding a parametric decoder head (SDG Causal). That approach changes the input. Ours changes where adaptivity lives in the model. The two are complementary, and we test them both separately and together.

The mechanism can be written in one line. Let $f_{l,s}(x)$ be the feature at voxel $x$ from decoder level $l$ passed through a depthwise branch with dilation $s$, and let $c_{k,l,s,m}$ be the $m$-th prototype of class $k$ at that level and scale. The class score is

$$z_k(x) = \sum_{l,s} \alpha_{k,l,s}\, \max_m \cos\big(f_{l,s}(x), c_{k,l,s,m}\big),$$

where $\alpha_{k,\cdot,\cdot}$ is a softmax over level–scale pairs learned per class and independent of the input. A softmax over $z_k/\tau$, with temperature $\tau$, gives the voxel label. The contrast we test is between $\alpha$ fixed per class and $\alpha$ computed from global pooling of the input, as in selective-kernel and DMC-Net gates.

Research Program

The backbone is a 3D U-Net, chosen because AMOS found it the strongest plain architecture (89.04% mDice on CT, above Swin-UNETR's 86.32%). At the three finest decoder levels we add depthwise branches with dilations 1, 2 and 4. Each class gets M=4 prototypes per level–scale pair. These are initialized from K-means on CT training features and then updated as exponential moving averages of features at the class's labeled voxels, which reduces the initialization sensitivity reported for ASGNet and Sim-MPNet. Training uses Dice plus cross-entropy on the scores $z_k$, CT data only, and one fixed intensity augmentation recipe shared by every method: random gamma, intensity inversion, and histogram remapping.

We build four variants on this shared scaffold. The first is the parametric 1×1 head on the single-scale backbone. The second is a parametric head on input-gated multi-scale branches, reproducing the DMC-Net gate. The third is a prototype head on input-gated branches. The fourth is the proposal, a prototype head with class-conditioned weights $\alpha$. This 2×2 design separates two questions: whether metric classification helps on its own, and whether conditioning the scale choice on class rather than input helps beyond that.

We also measure the mechanism directly. For input-gated models we record the gate distribution over scales inside each organ's ground-truth mask, on CT and on MRI, and compute the KL divergence between the two. If the hypothesis is right, the gate shift on MRI should correlate across organs with the Dice drop. The effect should be largest for small organs, adrenal glands and pancreas, where picking the wrong scale is most costly.

Baselines

The naive reference is a 3D U-Net trained on AMOS CT and applied unchanged to AMOS MRI. AMOS reports over 20% degradation for parametric models on unseen MRI scanner distributions, and its 89.04% in-domain CT mDice is the ceiling. We rerun this ourselves with the shared augmentation recipe so that every comparison uses the same data and preprocessing.

The strongest existing method is the causal single-source approach of Chen et al., which reports an average Dice of 86.20% on AMOS CT→MRI, within 1.13 points of target-supervised training. On lung CT→X-ray it raises Dice from 67.33% to 78.79% (SDG Causal). It is the strongest competitor because it attacks the same single-source setting with far more machinery: diffusion-generated style variants plus contrastive alignment. We run it with its released recipe if one is available. Otherwise we reimplement it with the same backbone, the same number of training iterations, and the same hyperparameter search budget of 20 trials per method on a CT-only validation split. No MRI labels are used for tuning by any method. Because 86.20% leaves little headroom on average, beating this baseline is defined in two ways. Our method must match its average Dice without diffusion augmentation, at a fraction of the compute. When combined with the diffusion augmentation, it must exceed it on the small-structure organs, adrenal glands, pancreas and gallbladder, by a margin larger than the seed-to-seed standard deviation over three seeds.

The scale-adaptive baseline is DMC-Net's input-gated multi-scale convolution with a parametric head (DMC-Net), trained on CT in the same pipeline. It reports 87.97% Dice on NIH pancreas and 60.71% on MSD pancreatic tumors in-domain, but has no reported cross-modality result. A static-pyramid baseline sums the same dilated branches with equal weights, isolating whether any adaptivity matters. The key ablation is variant three versus variant four, input-gated versus class-conditioned scale weights under an identical prototype head. If the class-conditioned version does not win on MRI, the core claim fails.

Why not just…?

Why not just swap in a prototype or cosine head and skip the scale machinery? This is the most serious threat. Metric heads are known to tolerate representation drift better than parametric heads (Prototypical Networks, Centroids Matching), and they may account for most of any gain. The 2×2 design settles this directly. If a prototype head on a single-scale backbone matches variant four on MRI, including on small organs, then the class-conditioned scale weights add nothing and we will report the simpler result. We expect the gap to appear on adrenal glands and pancreas, not on the liver.

Why not just train nnU-Net or a 3D U-Net with heavy intensity augmentation? AMOS shows that a plain 3D U-Net is hard to beat in-domain, so this baseline is real. Our naive reference already includes inversion and histogram remapping, which addresses the main CT-versus-MRI contrast differences. The evidence against relying on augmentation alone is the over-20% scanner-shift drop in AMOS and the pancreatic-tumor gap of roughly 54% versus 60.71% in DMC-Net. Whether augmentation closes the cross-modality gap is measured, not assumed.

Why not just use diffusion-based style translation? It works well on average (86.20%), but its authors report hallucinated or blurred fine structures such as the posterior vertebral arch, and it carries heavy compute cost. We do not compete with it on its own terms. We test whether class-conditioned matching repairs exactly the fine structures it damages, by stacking the two methods and reporting per-structure Dice and HD95.

Why not just use DINOv2 features with prototype pooling? The DINOv2 ALPNet-style model reaches 78.43 Dice on Abd-MRI and 73.72 on Abd-CT, but in-modality and in a few-shot protocol, so these numbers do not transfer to our setting. We include a DINOv2-LoRA encoder under our prototype head as an extra row. If foundation features already make input-gated scale selection modality-invariant, the gate-shift diagnostic will show near-zero KL divergence, and our claim narrows to non-foundation backbones.

Decisive Experiments

The primary test trains on the 500 AMOS CT scans and evaluates on the 100 AMOS MRI scans, reporting per-organ Dice and HD95 with three seeds. Secondary tests use the lumbar spine (CT→MRI and CT→X-ray) and lung (CT→X-ray) tasks from Chen et al.; for the spine task no reported Dice was found on Lacuna, so we report our own reruns of all methods. The decisive outcomes are three. First, variant four must beat variant three on MRI small-organ Dice. Second, in input-gated models, the CT-to-MRI gate KL must correlate positively with per-organ Dice drop. Third, CT in-domain Dice must not fall below the 3D U-Net reference, so that the invariance is not bought with source accuracy. If all three hold, class-conditioned scale selection is a cheap, generation-free way to make scale adaptivity survive modality shift. If only the first two fail, the result still tells the field that metric heads, not scale adaptivity, carry the cross-modality benefit.