Separating Modality Shift from Site Shift in Medical Segmentation Transfer

When a segmentation model trained on one kind of scan (say, contrast-enhanced T1 MRI) breaks on another (T2 MRI, or OCT angiography), the field usually calls the drop a "modality gap" and builds image translators to close it. In nearly every public benchmark, though, the target scans also come from different hospitals, different machines, and different preprocessing scripts. So the measured gap mixes three things: the physics of the new modality, the idiosyncrasies of the new site, and differences in the data pipeline. We propose to pull these apart with a factorial transfer grid. For each anatomy we pick public datasets so that modality, acquisition site, and preprocessing pipeline can each be changed alone and in combination. Every run uses the same containerized code, so the pipeline factor is a controlled variable rather than hidden noise.

This matters because the remedies differ. If most of a reported "cross-modality" drop is site shift, diffusion translators are aimed at the wrong target, and simpler site harmonization or more diverse training data would do as well. The crossMoDA consortium's observation that heterogeneous multi-site training improved robustness (crossMoDA Challenge) hints at this, but its cohorts change sequence and site at once, so the effect cannot be attributed to either.

Core Claim

The drop in overlap score (Dice, the fraction of voxels shared between predicted and true masks, scaled to 0–100) under transfer can be decomposed on a two-by-two grid. One axis is source versus target modality. The other is source versus target site. Pipeline enters as a third, binary factor: shared versus site-native preprocessing. Write $D_{ms}$ for Dice when training on the source modality and site and testing at modality $m$ and site $s$, with $0$ meaning source and $1$ meaning target. The gap then splits as

$$ D_{00}-D_{11} = \underbrace{(D_{00}-D_{10})}_{\Delta_{\text{mod}}} + \underbrace{(D_{00}-D_{01})}_{\Delta_{\text{site}}} + \underbrace{(D_{10}+D_{01}-D_{00}-D_{11})}_{\Delta_{\text{int}}}. $$

Here $\Delta_{\text{mod}}$ is the modality-only drop, $\Delta_{\text{site}}$ the site-only drop, and $\Delta_{\text{int}}$ whatever the two do jointly beyond their sum.

We make two testable claims. First, $\Delta_{\text{site}}$ and the pipeline term make up a substantial share of the gaps that published cross-modality methods report. Second, adaptation methods differ systematically in which term they shrink. Translation-based methods mainly reduce $\Delta_{\text{mod}}$. Multi-site training mainly reduces $\Delta_{\text{site}}$, which is our reading of the "Heterogeneity Paradox." What is new here is the decomposition and the per-method attribution, not another adaptation method.

Research Program

The work has three parts: build the grids, attribute the gaps, and attribute each method's gains.

Retinal grid. This is the primary testbed because all its labels are public. OCTA-500 contains OCT and OCT-A scans of the same eyes from one device, so OCT to OCT-A is a nearly pure modality shift. DRIVE, STARE and CHASE_DB1 are all fundus photographs from different sources, which gives a nearly pure site shift within one modality. Fundus to OCTA-500 combines both, and is the setting AdaptDiff reports (AdaptDiff). This grid is not perfectly factorial, since no single site offers both fundus and OCT-A. We therefore estimate $\Delta_{\text{int}}$ only where a full square exists, and report the other cells as bounds.

Brain MRI grid. crossMoDA's cohorts supply ceT1 and T2 scans at London, Tilburg and the ten-site UK MC-RC collection (crossMoDA Challenge). This allows a true square: ceT1 London to T2 London, ceT1 London to ceT1 Tilburg, and ceT1 London to T2 Tilburg. Where target labels are held by the organizers, we will score through their evaluation service. If some cells cannot be scored that way, the study says so rather than filling them in.

Abdominal grid (secondary). We use AMOS22, which the AR-SSL4M paper lists among its CT and MRI sources (AR-SSL4M), paired against ABD-30 CT and CHAOS MRI as used in SSM-SAM. Whether AMOS22 provides same-site CT and MRI with matched organ labels must be verified before this grid is committed.

Pipeline factor. The open-science component supplies the method here. Following Denaxas et al. on modular, tested pipelines, and Ashton et al. on releasing full execution context because hidden settings change outcomes, every cell runs in one container with versioned preprocessing. The factor has two levels. One is shared resampling and intensity normalization. The other is each dataset's native, originally published preprocessing. Unit tests check split disjointness at the patient level and resampling invariants. We also include a small set of synthetic phantoms in the spirit of Estéban et al.: vessel trees and ellipsoidal organs with known masks. These let a pipeline bug that silently shifts or drops structure be caught before it contaminates a cell.

Attributing each method's gain. Each adaptation method is run on every cell. Its improvement is recorded separately on the modality-only, site-only and combined cells. This gives each method a signature: which term it shrinks and by how much.

Baselines

The naive reference is direct transfer: nnU-Net trained on source modality and site, applied unchanged. On the retinal combined cell, AdaptDiff reports 73.16% Dice for its unadapted baseline on OCTA-500(6M), against an in-target supervised oracle of 84.78%. We will reproduce this number in our container first. If we cannot, that failure is itself a pipeline-factor finding.

The strongest translation-based methods are AdaptDiff and DCDA. AdaptDiff is a conditional diffusion model trained on pseudo-labels to render source images in target style; it reports 81.94% Dice on OCTA-500(6M). DCDA separates image content from style and makes two expert segmenters agree (DCDA); it reports 79.28% Dice for OCT-A to OCT, against an 81.06% oracle. On MRI, the strongest pipeline is crossMoDA's bridge-and-refine recipe: unpaired GAN translation from ceT1 to T2, nnU-Net trained on the synthetic images, then self-training on target pseudo-labels. Top entries report median Dice above 87% for tumour and above 84% for cochlea, with a plateau near 88%.

These methods are the right competitors because they are what the field currently credits with closing modality gaps. For fairness, all methods share one segmentation backbone per anatomy, identical preprocessing within each pipeline level, the same GPU-hour cap, and the same number of hyperparameter trials, tuned only on source-side validation data.

We add two site-targeted baselines that involve no translation. The first is per-site intensity histogram matching to the source site. The second is multi-site source training, the crossMoDA heterogeneity recipe. The key ablation removes translation from the strongest method while keeping its self-training stage; this isolates how much of that method's gain is due to modality rendering specifically.

A translation method "beats" the site-targeted baselines only if it outperforms them on the modality-only cell by more than the variance across seeds and splits. Conversely, if histogram matching or multi-site training matches translation on the combined cell, the proposal's central claim is supported.

Why not just…?

Why not just stratify existing leaderboards by site? crossMoDA already reports three cohorts, but each changes sequence and site together. Stratifying cannot separate the two, because the missing cells were never scored: same-modality transfers across sites, and same-site transfers across modalities. The grid adds exactly those cells. If the organizers' data turn out not to permit them, the retinal grid carries the claim.

Why not just fix preprocessing to nnU-Net defaults and drop the pipeline factor? This is a real possibility. If shared versus native preprocessing moves Dice by less than seed variance on every grid, we will report a null and drop the factor. The case for keeping it rests on the reproducibility failures documented by Denaxas et al. and Ashton et al.. Its size in medical transfer is exactly what we measure.

Why not just probe features for site and modality information, as CFA does with separate orthogonal heads? CFA shows that encoders mix domain and class features on natural-image benchmarks. However, a probe that decodes site from features says nothing about how much Dice is lost because of site. We will use CFA-style orthogonal heads as an additional intervention on the grid, not as the measurement.

Decisive Checks

The first test concerns additivity. We fit $\Delta_{\text{mod}}$ and $\Delta_{\text{site}}$ from single-factor cells, predict the combined cell, and report the residual $\Delta_{\text{int}}$ with bootstrap intervals over patients. A large interaction would mean modality and site effects must be studied jointly, which is itself a result.

The second test concerns reinterpreting the heterogeneity paradox. If multi-site training helps through site diversity, it should shrink $\Delta_{\text{site}}$ while leaving $\Delta_{\text{mod}}$ essentially unchanged. If it also shrinks $\Delta_{\text{mod}}$, our reading is wrong.

The third test concerns attribution of published gains. The central claim is falsified if translation methods' gains on the combined cells come almost entirely through the modality-only term, and site-only baselines fail to close a meaningful fraction of the gap.

All containers, splits, phantoms and per-cell predictions will be released, so any new method can be scored on the full grid rather than on a single number.