Physical Coordinates for Few-Shot CT-to-MRI Segmentation

When a segmentation model trained on CT scans is adapted to MRI using only a handful of labeled MRI scans, its errors are usually blamed on appearance. Tissues have different brightness and texture, and noise looks different. This proposal tests a second, largely unmeasured cause: geometry. CT and MRI volumes are stored on voxel grids with very different spacing. A typical CT has fine, near-isotropic slices. A typical abdominal MRI has larger in-plane pixels and slices several millimetres thick, and its field of view has a different shape. Standard pipelines squeeze both into the same fixed grid, which is the medical version of the aspect-ratio distortion that vision researchers have documented for photos and documents. The geometric priors that make few-shot learning work then silently assume that a voxel means the same physical distance in the support scans and the query scans. Those priors include expected organ size, distance-to-boundary fields, and positional encodings. Across modalities, that assumption is false.

The proposal has two parts. First, it measures how much of the CT-to-MRI few-shot gap is geometric. The test resamples CT into MRI's geometry while keeping its appearance, and resamples MRI into CT's geometry, then checks how much accuracy each swap moves. Second, it builds a few-shot adapter whose coordinates, priors, and distances are expressed in millimetres on each scan's native grid, rather than in voxels on a resampled grid. If the geometric share turns out to be negligible, that is itself a useful negative result. It would tell the field that cross-modality few-shot work can safely ignore spacing and concentrate on appearance.

Core Claim

In few-shot cross-modality segmentation, a measurable part of the accuracy gap comes from mismatched voxel geometry between the support modality and the query modality. That part cannot be removed by resampling everything to one isotropic grid, because resampling either throws away in-plane detail or invents through-plane detail. It can be removed by keeping each scan on its native grid and making the model's notion of position, size, and distance physical.

The claim is narrower than the "native-aspect, parameter-efficient, cross-modality" synthesis the literature review suggests. That broad version would need a NaViT-scale pre-training corpus, which medicine does not have. This proposal instead changes only the units in which a frozen backbone's adapters and priors operate. It tests that change against the obvious fix of isotropic resampling.

What Prior Work Establishes and Leaves Open

Several papers motivate the geometric hypothesis without testing it. Patch n' Pack: NaViT processes images as variable-length patch sequences at native aspect ratio, using separate horizontal and vertical position embeddings. LLaVA-UHD slices high-resolution images into native-aspect tiles. Both show that removing aspect distortion helps, but only on natural images with web-scale training. We borrow NaViT's factorized per-axis position idea and index it by millimetres instead of patch counts. We do not attempt its pre-training. CARVE compresses slice-based 3D tokens for medical multimodal language models and reports that, at equal token budgets, in-plane resolution matters more than slice count. That finding is a direct argument against isotropic resampling, which typically lowers in-plane resolution. CARVE is training-free and does not study few-shot adaptation.

Two few-shot methods supply the components we modify. Transductive few-shot adapters for medical image segmentation (FSEFT) freezes a CT-pretrained Swin-UNETR and inserts small 3D convolutional "spatial adapters." At test time it penalizes query predictions whose organ proportions differ from those in the support set. The paper reports up to +0.9 Dice from this size prior, and shows that full fine-tuning collapses at one labeled scan. Its size prior is a voxel proportion, and it was evaluated only CT-to-CT. AGENet builds edge-aware geodesic distance fields, meaning distance measured along paths that slow down at image edges, computed with Fast Marching. It uses these fields to place prototypes densely near boundaries. AGENet reports 79.56% Dice at one shot and 81.67% at five shots, averaged over nine datasets. It computes distances on resampled square grids and evaluates each modality separately, never with CT supports and MRI queries.

Cross-modality methods constrain the design. Online Mutual Knowledge Distillation pairs a CycleGAN translator with two segmenters that teach each other. It reaches 0.9012 mean Dice on MM-WHS CT with MRI assistance, against a 0.8706 baseline, on fixed 20-volume splits and uniform resampled dimensions. AdaptDiff shows that GAN translation can damage thin structures, while weakly conditioned diffusion raises OCTA-500 vessel Dice from 73.16% to 81.94%. Both attack appearance and hold geometry fixed by resampling. Our work is complementary to them: if geometry matters, it should add to an appearance fix rather than replace one.

AMOS supplies the testbed and a key observation. Joint CT and MRI training helps MRI by +2.14 mDice. Models drop by more than 20% on MRI scanners unseen in training, and the worst-hit organs are small, thin ones such as the adrenal glands and duodenum. We hypothesize that part of that scanner drop is a spacing effect. We test this hypothesis; we do not assume it.

Research Program

The first component is a diagnostic study that splits the gap into geometric and appearance parts. On AMOS, which has the same 15 organs annotated in CT and MRI from eight scanner models, we fix a CT-pretrained frozen backbone and adapt it with K labeled scans, for K of 1, 3, and 5.

We build two synthetic conditions. "CT-in-MRI-geometry" resamples CT volumes to MRI spacing, slice thickness, and field-of-view aspect ratios drawn from the actual AMOS MRI distribution, while keeping CT intensities. "MRI-in-CT-geometry" does the reverse, as far as MRI's information content allows.

The geometric share is the accuracy lost going from CT to CT-in-MRI-geometry, divided by the accuracy lost going from CT to real MRI, at matched K. The same procedure is repeated on MM-WHS cardiac CT/MRI as a second anatomy. We report the share per organ, because the hypothesis predicts it will be largest for thin structures.

The second component is a physical-coordinate adapter, which we call PhysAdapt. Each scan stays on its native grid; we only normalize intensity. Three changes are applied to the FSEFT recipe.

First, the spatial adapters receive the per-axis voxel spacing. A small conditioning layer turns spacing into per-channel scale and shift, so the adapter learns how much to trust each axis. Each adapter's 3D convolution uses a per-axis dilation chosen so that its receptive field covers a fixed physical extent, roughly constant in millimetres across modalities. Factorized per-axis position features are indexed by millimetre coordinates, following NaViT's factorization.

Second, the transductive size prior is stated as physical organ volume rather than voxel proportion. With $v_c$ the support-set mean volume of organ $c$ in millilitres and $\hat{v}_c$ the predicted query volume, $\hat{v}_c = s_x s_y s_z \sum_i p_{i,c}$, the penalty is

$$ \mathcal{L}_{\text{size}} = \sum_c \left(\log \hat{v}_c - \log v_c\right)^2 . $$

Here $s_x, s_y, s_z$ are the query's voxel spacings and $p_{i,c}$ is the predicted probability that voxel $i$ belongs to organ $c$.

Third, for the 2D prototype track built on AGENet, the geodesic field is computed with Fast Marching under an anisotropic metric that weights each axis step by its physical length. Prototype placement density is then defined per square millimetre of boundary rather than per pixel.

The backbone stays frozen throughout. All gains therefore come from units and grid, not from extra capacity.

Baselines

The naive reference point is FSEFT's standard pipeline applied without modification to MRI. Every volume is resampled to the backbone's fixed grid, and adapters and the voxel-proportion size prior are used as published. This is what a practitioner would do today. A "no adaptation" arm, the frozen CT model with only its head retrained on the K MRI scans, sits below it.

The strongest competitor is FSEFT with physically isotropic resampling. All CT and MRI volumes are resampled to a common isotropic spacing before the adapters and the size prior. This is strong because it makes voxel counts proportional to physical volume, which removes most of the size-prior mismatch with no new machinery. If PhysAdapt only matches it, the method reduces to resampling. We run it at two spacings: one matching the MRI in-plane spacing, and one matching CT's finer spacing, which upsamples MRI through-plane. Choosing either spacing would otherwise look like cherry-picking.

For fairness, every arm uses the same frozen backbone, the same K supports drawn with the same seeds (at least five draws per K), and the same adapter parameter count within 5%. Every arm also gets the same hyperparameter search budget, tuned on a held-out MRI validation split, and the same GPU time per adaptation.

I found no reported result on Lacuna for FSEFT or any few-shot adapter on AMOS CT-to-MRI. The closest reported numbers are FSEFT's up to +0.9 Dice from its size prior on CT, AMOS's 89.04% mDice for 3D-UNet on CT with full supervision, and AGENet's 79.56% one-shot mean Dice across its nine datasets. So we will produce the baseline numbers ourselves.

On the 2D track, AGENet as published is the baseline, run with CT supports and MRI queries on its ABD-CT and ABD-MRI data. For the appearance comparison, Mutual Knowledge Distillation is run on MM-WHS at reduced K. I found no reported result on Lacuna for it below 20 volumes.

PhysAdapt counts as beating these baselines if it exceeds isotropic-resampled FSEFT by more than the seed-to-seed standard deviation in mean Dice at K of 1 and 3. A larger margin is expected on adrenal glands, duodenum, and esophagus, along with a lower 95th-percentile boundary distance.

Ablations remove each ingredient separately: millimetre conditioning and dilation, the physical size prior, and the anisotropic geodesic. One further ablation keeps the native grid but drops all spacing information. That separates "do not resample" from "know the spacing."

Why not just…?

Why not just resample everything to isotropic millimetres, as most clinical pipelines do? This is the real threat, and the experiments are designed around it. Resampling fixes unit mismatch, but it forces a choice: downsample MRI in-plane, which loses the detail CARVE found most valuable, or interpolate MRI's thick slices into thin ones the scanner never measured. Isotropic resampling is a main baseline at two target spacings. If it matches PhysAdapt, we will report that the geometric gap is solved by resampling and that native-grid processing adds nothing.

Why not just augment the few support scans with random anisotropic rescaling, so the adapter becomes robust to spacing? Augmentation teaches tolerance to geometric variation but gives the model no way to know the actual spacing of a query. It also cannot correct a size prior stated in voxels. With one or three supports, randomly rescaling them also multiplies a tiny dataset into a family of slightly wrong anatomies. We include "FSEFT + anisotropic scale augmentation" as an additional arm under the same tuning budget. If it closes the gap, the conclusion is that spacing conditioning is unnecessary.

Why not just fix appearance with translation, as AdaptDiff or Mutual Knowledge Distillation do? Both papers resample to fixed grids, so they leave geometry untouched by construction. The diagnostic study measures how much gap remains once appearance is held fixed: CT-in-MRI-geometry has CT appearance by design. A combined arm, translation plus PhysAdapt, tests whether the two fixes add up. If appearance translation alone closes nearly all the gap, the geometric share will show up as near zero and the proposal is falsified.

Decisive Experiments

Three checks decide the proposal. The first is the geometric share on AMOS and MM-WHS. If CT-in-MRI-geometry loses less than about a tenth of the CT-to-MRI drop, geometry is not a meaningful cause and the method half of the work is dropped. The second is the head-to-head against isotropic-resampled FSEFT across K and seeds, reported per organ. The hypothesis predicts gains concentrated on thin and small structures. A uniform gain or no gain would point to something other than geometry. The third, run on the AGENet track, compares one-shot boundary distance under the anisotropic metric against the resampled square-grid metric for MRI queries with thick slices. If prototypes placed per millimetre do not reduce the boundary error, the argument that distorted grids bias geodesic priors is wrong for this setting.

As a secondary analysis, we stratify AMOS MRI test scans by scanner. We then check whether PhysAdapt's gains track how far each scanner's spacing departs from the training spacing. If they do, part of the reported drop of over 20% on unseen scanners is geometric.