Spend the Invariance Coarse, Spend the Labels Fine: Asymmetric Adaptation of Segmentation Cascades Across Scanners

Medical segmentation models often run as a cascade. A first network looks at a downsampled scan to find roughly where an organ is. A second network then examines a high-resolution crop around that location to trace the exact boundary. When such a cascade moves to a new scanner or modality with only a handful of labeled scans, the usual move is to adapt the whole model the same way everywhere. We argue the two stages fail for different reasons and should be treated differently. The coarse stage sees whole-body layout, which changes little between hospitals. Its failures are rare but catastrophic: if it misses the organ, the crop is empty and nothing downstream can recover. The fine stage sees small patches, where scanner-specific texture, contrast and noise dominate. Its failures are frequent but local.

The proposal is to make the coarse stage robust by training it only on source data under heavy style randomization, with a cropping rule that widens when the coarse stage is unsure. The few labeled target scans are spent only on the fine stage, through small adapters trained on crops. A crop-level learner gets many training patches from each labeled scan, so five target volumes are a much larger dataset for it than for a whole-volume model. The test is concrete. On abdominal CT→MRI (AMOS) and cardiac CT↔MR (MM-WHS) at a fixed budget of 1, 3 or 5 labeled target scans, does this asymmetric split beat adapting the whole volume, adapting both stages, and using no target labels at all? We also measure an error decomposition that says where cross-scanner damage actually lives.

Core Claim

Under cross-scanner or cross-modality shift with $K \le 5$ labeled target scans, segmentation error splits into two parts. Localization failures are whole-organ misses caused by a bad coarse crop. Boundary failures are local mistakes inside a correct crop. The two respond to different remedies. Target-free style invariance is enough for the first. Target labels pay off far more on the second, because cropping turns a few scans into many patch-level examples. A cascade built on this split should beat symmetric adaptation at equal label and compute budgets. It should also sharply reduce the rate of catastrophic misses that plague cascades: A 3D Coarse-to-Fine Framework for Volumetric Medical Image Segmentation reports 0% Dice cases on pathological pancreas when the coarse stage misses.

The closest existing work combines two of the three ingredients but never this allocation. COSTA builds a coarse network plus refinement network for multi-vendor vessel MRI. It applies one style-consistency loss uniformly across the network and trains on hundreds of fully labeled volumes, so it says nothing about where scarce labels should go. FAMNet handles cross-domain few-shot segmentation with a coarse prototype mask and frequency-band matching. It never crops and re-examines a high-resolution region, and it targets novel classes rather than a known organ on a new scanner. CoFi runs a 3-shot coarse mask → pruning → frozen SAM 2 pipeline, but its authors note it is untested under multi-site shift. Transductive few-shot adapters (FSEFT) adapts a frozen 3D foundation model with small 3D convolutional adapters from $K \in \{1,2,5,10\}$ scans. It does so on full volumes, within CT only. Our method reuses its adapter design and its size prior.

Research Program

The coarse stage is a low-resolution 3D U-Net trained on source CT. We use the classic U-Net because AMOS finds it beats Swin-UNETR on CT (89.04% vs 86.32% mDice). It is trained with adversarial style synthesis in the manner of Adversarial Consistency for Single Domain Generalization. That method perturbs image style with adaptive instance normalization, which re-scales feature channel statistics to mimic a new appearance. It also uses a patch-level mutual-information term so anatomy is not distorted while style changes. We borrow that method for the coarse stage only. The coarse stage never sees a target label.

The output is a per-organ probability map, plus an uncertainty estimate from a few test-time style perturbations. The crop box is not a fixed padding. It is the smallest box that contains a set fraction (e.g., 99%) of the coarse probability mass pooled across the perturbations, so disagreement across perturbations widens the crop. This replaces the hand-set margins that C2FTrans and CoFi rely on. It directly targets the empty-crop failure.

The fine stage is a single shared network for all organs. It follows Rap-Net in taking a two-channel input: the high-resolution crop plus the coarse prior for that crop. It is first trained on source CT crops. On the target domain it is frozen, and only 3D convolutional adapters of the FSEFT type are trained. They are trained on crops jittered around ground truth from the $K$ labeled target scans, with the coarse prior channel computed by the real source-only coarse stage, so the fine stage learns to correct realistic coarse errors. Transductive inference adds FSEFT's organ-size penalty, estimated from the support scans. That paper credits this penalty with up to +0.9% Dice at low data.

The analysis piece is an error decomposition run before any method comparison. For a source-trained cascade evaluated on target data, we swap in oracle crops (from ground truth) and oracle coarse priors one at a time. The change in Dice attributable to each swap tells us how much of the cross-domain gap is localization and how much is boundary.

That decomposition is the result that justifies or kills the design. If oracle crops recover most of the gap, label budget should go to the coarse stage and the asymmetric hypothesis is wrong for that shift. If oracle crops recover little, the fine stage is where labels matter. AMOS reports a drop of over 20% Dice on MRI from unseen scanners, so there is a large gap to divide.

Baselines

The naive reference is a 3D U-Net trained on source CT and applied unchanged to target MRI. AMOS documents a drop of over 20% Dice in this kind of setting, and we re-measure it on our exact split. A second naive point is the same network trained from scratch on only the $K$ target scans. Together these bracket "no transfer" and "no source."

The strongest target-free competitor is adversarial domain synthesis with mutual-information regularization (ADS). Adversarial Consistency for Single Domain Generalization reports 85.93% Dice on abdominal CT→MRI against 76.60% for plain training and 83.39% for GIN. That protocol uses its own organ set and split, so we rerun ADS on our AMOS split rather than quote it. The proposal must beat it, because using labels has to buy something over using none.

The strongest few-label competitor is FSEFT. Its paper reports an +11.8% Dice gain over self-supervised weights. No CT→MRI number for FSEFT was found on Lacuna, so we run it ourselves. It gets the same frozen backbone, the same $K$ target scans, full-volume adapters, and the size prior.

FAMNet is included as the published cross-domain few-shot method. It reports 65.79% Dice on abdominal CT→MRI and 71.58% on cardiac LGE→b-SSFP, but under a novel-class episodic protocol. We report it as a reference rather than as a same-protocol rival. A CoFi-style pipeline is the foundation-model competitor: source coarse mask, morphology pruning, then frozen SAM 2 prompts. CoFi reports 74.54% Dice on 3-shot microscopy, which is not directly comparable to our setting.

The key ablations hold the cascade fixed and move only the label allocation. One variant adapts only the coarse stage with the $K$ scans. A second adapts both stages (the symmetric variant). A third adapts neither and relies purely on style randomization. A further ablation replaces the uncertainty-widened crop with a fixed margin.

All methods share backbones where applicable, the same $K$ scans drawn from five random support draws, the same number of tuning trials (e.g., 20), and a matched GPU-hour cap. Beating a baseline means a higher mean Dice and Normalized Surface Dice across draws by more than the across-draw standard deviation. It also means a lower rate of catastrophic misses, defined as organ Dice below 10%. Draw-to-draw variance is reported because few-shot methods are known to swing with support choice.

Why not just…?

Why not just adapt the whole volume with FSEFT's adapters? This is the most serious threat. FSEFT already beats full fine-tuning at 1 shot, and the coarse-to-fine advantage might vanish once the backbone is strong. Our counter-argument is about data volume. A full-volume adapter gets $K$ training examples, while a crop adapter gets many jittered patches per scan focused on the region that matters. We also expect the cascade to train in less activation memory per step, which is roughly where A 3D Coarse-to-Fine Framework's 36% speedup came from. The matched-budget comparison at $K=1,3,5$ settles this directly. If FSEFT ties us, we report that crop-level adaptation adds nothing given a strong foundation model.

Why not just use style randomization and no target labels at all? ADS's 85.93% CT→MRI Dice shows target-free methods are already strong. The decomposition tells us whether the residual error is local boundary error that randomization cannot fix. DR-Adapter and the ADS authors both state that style-only methods fail when shape or content shifts. The "adapt neither" ablation measures the remaining gap exactly. If it is near zero, the label budget is unnecessary for that shift.

Why not just adapt both stages, or fine-tune the cascade end to end? It sounds strictly better to use labels everywhere. With $K \le 5$, though, the coarse stage gets only $K$ whole-volume examples and can overfit to the few target scanners. That would raise the catastrophic-miss rate, which is the exact failure the cascade must avoid. The symmetric-adaptation ablation tests this. We also report miss rate on held-out target scanners not represented in the support set.

Why not just hand the coarse box to SAM? CoFi itself shows SAM needs carefully filtered prompts and hallucinates boundaries without them. It is also frozen, so it cannot absorb MRI-specific appearance from the support scans. The CoFi-style baseline measures whether that prior beats a few labeled target crops.

Proposed Experiments

The main setting is AMOS: train on the 500 CT scans, then adapt to MRI with $K \in \{1,3,5\}$ labeled scans. Evaluation uses held-out MRI scans, split by scanner where metadata allows, with mean Dice and Normalized Surface Dice over 15 organs. We separately report small organs (adrenals, gallbladder), where the fine stage should matter most.

The second setting is MM-WHS cardiac CT↔MR at the same $K$, reporting Dice and average surface distance. It is comparable in spirit to Meta-hallucinator, which reports a 6.7% Dice gain over MT-UDA at 4 shots, though with labels on the source side. A third, harder check is COSTA cross-vendor TOF-MRA. There the structures are thin vessels, and the boundary error should dominate even more.

The decisive checks are three. First, the oracle-crop decomposition on AMOS should show that most of the remaining target-domain gap of the style-randomized cascade lies inside correct crops. Second, at matched budget, adapting only the fine stage should beat both FSEFT on full volumes and the symmetric variant. The margin should be largest at $K=1$, where crop-level data multiplication helps most. Third, the uncertainty-widened crop should cut the catastrophic-miss rate relative to fixed margins without lowering mean Dice. If the decomposition instead shows localization dominates, or FSEFT matches the cascade, the asymmetric claim is falsified for that shift. That negative result would still tell practitioners where few target labels belong.