A segmentation network trained on labeled CT scans usually breaks when it is applied to MRI. The anatomy is the same, but the image contrast is not. A common repair is to make the network's internal features look statistically alike across the two scanners. Most methods that do this compare features one 2D slice at a time, so they never check whether anatomy changes from slice to slice in the same way in both modalities. That slice-to-slice continuity is exactly what radiologists rely on when they scroll through a volume. We propose to align the joint distribution of each slice's features together with their change to the next slice, using sliced optimal transport. Sliced optimal transport is a cheap distance between distributions: it projects high-dimensional points onto many lines and compares the sorted projections on each line. We choose those lines adaptively, because the evidence that two modalities still differ is concentrated in a few slices and a few feature directions. Lines drawn uniformly at random almost never point there.
In two sentences: per-slice feature alignment can make CT and MRI slices look identical while leaving their depth structure mismatched, which produces organs that flicker in and out between adjacent slices. Matching the distribution of slice-to-slice feature increments, with projection directions searched for rather than sampled, should close that gap, and it should help most in the hard CT→MRI direction.
Core Claim
The claim has two parts, and each can be tested separately.
The first part concerns volumetric context. Matching marginal per-slice feature distributions does not constrain how features evolve along the depth axis. So the cross-slice mismatch survives alignment and shows up as fragmented target predictions. This fragmentation is the failure that Deep convolutional networks for pancreas segmentation in CT imaging documents for 2D slice-wise segmentation.
The second part concerns power decay, meaning the loss of a test's ability to detect a fixed difference as dimension grows. The depth-increment discrepancy is sparse: it lives near organ boundaries, the cardiac apex and the great vessels. Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics proves that averaging uniformly over projections loses this kind of signal exponentially, while maximizing over projections keeps it. A uniformly sliced alignment loss will therefore deliver a weak, noisy gradient on precisely the discrepancy that matters.
Concretely, let $u_z \in \mathbb{R}^C$ be a pooled encoder feature for slice $z$ of a volume. Let $\delta_z^{(s)} = u_{z+s} - u_z$ be its increment at depth stride $s$, and let $v_z = [u_z;\, \delta_z^{(1)};\, \delta_z^{(4)}]$. Write $P_S$ and $P_T$ for the distributions of $v_z$ over source (CT) and target (MRI) slices. The alignment loss is
$$ \mathcal{L}_{\text{align}} = \max_{U \in \mathrm{St}(D,k)} \frac{1}{k}\sum_{j=1}^{k} W_2^2\big(U_j^\top{}_\# P_S,\; U_j^\top{}_\# P_T\big) \;+\; \lambda\, \mathrm{SW}_2^2(P_S, P_T). $$
Here $\mathrm{St}(D,k)$ is the set of $k$ orthonormal directions in the $D$-dimensional joint space. $W_2$ is the one-dimensional Wasserstein distance, computed by sorting. $\mathrm{SW}_2$ is ordinary uniformly sliced Wasserstein with weight $\lambda$, which keeps a floor of global alignment.
The directions $U$ are updated by a few Riemannian ascent steps, as in Two-sample Test using Projected Wasserstein Distance. To avoid the overfitting that forces sample splitting in Neural Wasserstein Two-Sample Tests, the directions are fit on a memory bank of past batches and then evaluated on the current batch, which splits samples in time rather than discarding half the data.
Relation to Prior Work
The closest prior work, Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation, uses uniform radial slicing to measure disagreement between two classifier heads' softmax outputs on 2D natural images. We instead slice internal features, include depth increments, and choose directions adaptively. The two medical methods closest to ours represent the strongest current practice. Structure-Driven Unsupervised Domain Adaptation for Cross-Modality Cardiac Segmentation aligns 3D landmarks and Canny edges with adversarial discriminators. CISFA translates images and applies patch-wise and global contrastive losses on 2D slices. Neither compares depth dynamics as distributions.
The statistics literature supplies the method and motivates the adaptive part. Minimax-Optimal Two-Sample Test with Sliced Wasserstein shows that uniform slicing is rate-optimal only when the number of slices grows with the sample size, and only on simple geometries. Projected Wasserstein and Subspace Robust Wasserstein Distances supply the max-over-subspace construction.
One result constrains the design: Statistical Insights into HSIC in High Dimensions shows that fixed kernels degrade to linear covariance tests once dimension exceeds sample size. MM-WHS has only tens of volumes per modality, though each yields hundreds of slices, so we keep $k$ small (4 to 16) and pool features before projecting.
CARVE also motivates the sparsity assumption. It finds that adjacent slices are largely redundant and that critical evidence sits on one or two slices, but it handles this with training-free token pruning and does no cross-modality alignment.
Research Program
We build a 2.5D U-Net, an encoder applied per slice with a light depth-mixing layer, and train it on labeled CT plus unlabeled MRI with $\mathcal{L}_{\text{align}}$ applied at two encoder scales. The first study is a controlled measurement in the real feature space. We take source-only CT encoder features and build a "pseudo-modality" from CT volumes using a known intensity remapping plus slice-local perturbations at boundary slices. We then measure the detection power of uniform SW, adaptive SW, Gaussian-kernel MMD and an adversarial classifier as the channel dimension $C$ grows. This shows whether the power-decay argument transfers from synthetic hyperspheres to medical features.
The second study is the adaptation experiment itself. The third uses the adaptive sliced statistic, computed on held-out source and target features, as a label-free model-selection score. Unsupervised adaptation has no target labels for choosing checkpoints, and a detectability score that tracks target Dice would be useful on its own.
Baselines
The naive reference point is source-only training with no adaptation. Its reported lower bound on MM-WHS is 64.9% mean Dice for MRI→CT and 38.1% for CT→MRI, according to Structure-Driven UDA.
The strongest existing method on MM-WHS is that same structure-driven method, at 85.8% (MRI→CT) and 74.8% (CT→MRI) mean Dice. It is the strongest competitor because it already exploits 3D anatomical invariance, which is our core intuition, through a different mechanism. On the abdominal CT↔MRI task the strongest reported number is CISFA's 76.24% Dice for CT→MRI and 70.45% for MRI→CT. CISFA reports 68.87% on MM-WHS whole heart, under a protocol that evidently differs from the structure-driven paper's.
Because protocols differ across papers, we will rerun both methods from their descriptions on our exact splits and preprocessing, and report published numbers alongside. Beating them means a higher CT→MRI mean Dice on our rerun with matched tuning. Each method gets the same number of hyperparameter configurations, and checkpoints are chosen by the same label-free rule, with oracle selection also reported.
For SWD on medical data we found no reported result on Lacuna, so we implement it on our backbone.
Three ablations isolate the two ingredients, all on the identical 2.5D backbone. The first is adaptive slicing on per-slice features $u_z$ only, which removes the depth increments. The second is uniform SW on the joint vector $v_z$, which removes adaptivity; we give it $L$ proportional to the batch size, per the minimax paper, so it is not handicapped. The third replaces sliced transport on $v_z$ with Gaussian MMD and with an adversarial discriminator, which tests whether the distance itself matters.
Why not just…?
Why not just train a native 3D network and align its features? This is the most serious threat. A 3D encoder sees depth directly, and per-voxel feature alignment might capture continuity without explicit increments. The MM-WHS challenge reports high variance for 3D deep models trained on 40 volumes, and multi-atlas methods matching them on MRI, which hints at overfitting. That is a hint, not proof. We will run a 3D U-Net with both uniform and adaptive SW at matched compute. If it equals our method, we will conclude that depth increments are only a cheap surrogate for 3D capacity and report that.
Why not just use max-sliced Wasserstein on per-slice features? If the residual mismatch were purely in slice appearance, this would suffice. The per-slice ablation tests that directly. The prediction that would falsify our volumetric claim is equal Dice and equal slice-continuity error between the per-slice and joint variants.
Why not just translate CT into MRI-style images, as CISFA does? Translation attacks the appearance gap directly. However, its contrastive losses act on 2D patches, and unconstrained translation is known to warp fine boundaries. Our loss is also complementary rather than exclusive, so we additionally report CISFA-style translation followed by our alignment.
Why not just use an adversarial domain discriminator? Alippi et al. show that full-dimensional statistics lose detection power roughly as $1/d$. CISFA abandoned adversarial cycles because they distort organs. The discriminator-on-$v_z$ ablation measures whether this matters in practice.
Proposed Experiments
We evaluate on MM-WHS in both directions and on the CISFA abdominal CT↔MRI task. Metrics are Dice, average surface distance and Hausdorff distance, plus a slice-continuity error: the fraction of adjacent target-slice pairs where a structure's predicted area changes by more than a threshold that ground truth never exceeds.
Four checks are decisive. The first is whether the joint adaptive variant beats the per-slice adaptive variant on CT→MRI Dice and on continuity error, with gains concentrated on myocardium and the great vessels, which the challenge flags as hardest on MRI. The second is whether the adaptive joint variant beats the uniform joint variant, and whether the size of that gap tracks the power gap measured in the pseudo-modality study. The third is whether our method on the 2.5D backbone matches or beats a 3D encoder at equal compute. The fourth is whether the adaptive detectability score ranks checkpoints by target Dice better than uniform SW or MMD, measured by Spearman correlation across checkpoints.
If the second check fails, the power-decay framing is wrong for this regime and the contribution reduces to depth-aware alignment. If the first check fails, the volumetric hypothesis is wrong.
References
- Deep convolutional networks for pancreas segmentation in CT imaging
- Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics
- Two-sample Test using Projected Wasserstein Distance
- Neural Wasserstein Two-Sample Tests
- Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation.
- Structure-Driven Unsupervised Domain Adaptation for Cross-Modality Cardiac Segmentation
- Contrastive Image Synthesis and Self-supervised Feature Adaptation for Cross-Modality Biomedical Image Segmentation
- Minimax-Optimal Two-Sample Test with Sliced Wasserstein
- Subspace Robust Wasserstein Distances
- Statistical Insights into HSIC in High Dimensions
- CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understanding
- Evaluation of Algorithms for Multi-Modality Whole Heart Segmentation: An Open-Access Grand Challenge
- Change Detection in Multivariate Datastreams: Likelihood and Detectability Loss