Hospitals that deploy an image segmentation model, software that outlines organs in a scan, keep running into the same problem. A model trained at one site loses accuracy when a new site with a different scanner adopts it. The usual fix is to fine-tune the model on the new site's data, and that fix causes catastrophic forgetting: the updated model gets worse at the sites it already served. It also breaks the regulatory status of a certified "locked" device. We propose to leave the certified 3D segmenter completely untouched. Each new site instead gets a small learned image-to-image corrector that reshapes its scans so the frozen model reads them correctly. The correctors are kept in a bank, and at test time a scan is sent to the right one using stored feature statistics, never stored patient images.
Forgetting of old sites then cannot happen in the network's weights, so the failure moves to two measurable places. The first is whether a correction applied only to the input can close the cross-scanner gap at all. AMOS reports a drop of more than 20% in segmentation accuracy on MRI from unseen scanners (AMOS). The second is whether the system picks the right corrector for an incoming scan. The scientific question underneath is how much of real multi-site medical domain shift can be repaired at the input, and how much requires changing the model. The answer decides whether locked-model continual deployment is viable for dense 3D segmentation.
Core Claim
We claim the following. For scanner and protocol shift (not new anatomy or pathology), a supervised input-space corrector learned from a handful of labeled volumes per new site, routed by privacy-safe feature statistics, matches weight-updating continual methods on new-site accuracy. It also keeps backward transfer near zero by construction. It should clearly beat the cheap input fix everyone tries first, intensity standardization. If histogram-style normalization recovers most of the gap, the learned corrector is unnecessary and we will report that.
The closest work splits into three groups. The first group adapts inputs but leaves the model frozen. VPTTA learns a tiny frequency-domain prompt per test image by aligning batch-normalization statistics. EDCP learns a deformable convolutional prompt with an offset bank that skips adaptation for familiar domains. Both are unsupervised, run per instance, and rely on batch-normalization statistics. That last point matters because nnUNet-style 3D segmenters use instance normalization, so the alignment loss these methods use is not directly available. Neither method accumulates durable per-site knowledge.
The second group is INPUTADAPTER, which supplies our core method. It trains a supervised adapter in front of a frozen clinical predictor and finds that the adapter succeeds by repairing institutional artifacts rather than matching the source distribution. It reports +14.1 AUCPR on eICU kidney-injury prediction. That work covers time series only, a single target hospital, and no sequence of sites. We move the idea to 3D dense segmentation and to a stream of sites.
The third group updates weights. FR²Seg replays stored low-frequency Fourier "styles" from past sites. SMG-Learning replays diffusion-generated scans. ACS disentangles content from style but needs two sites at the start and then freezes its encoder. All three alter the deployed predictor. We instead start from a single-site backbone, which is the cold-start case ACS cannot handle.
Research Program
The backbone is a 3D nnUNet trained once on a single initial site and then frozen. We choose the convolutional UNet over a larger transformer partly because AMOS found the 3D-UNet ahead of Swin-UNETR on CT, with 89.04% versus 86.32% mDice.
For each new site $s$, we train a corrector $g_s$ with the objective
$$ \min_{\phi_s} \sum_{(x,y)\in D_s} \mathcal{L}_{seg}\big(f(x + g_{\phi_s}(x)),\, y\big) + \lambda \,\lVert g_{\phi_s}(x)\rVert_1 . $$
Here $f$ is the frozen segmenter, $D_s$ is a small labeled set of $k$ volumes from site $s$ (we test $k$ from 2 to 10), $g_{\phi_s}$ is a shallow 3D convolutional encoder-decoder with parameters $\phi_s$ that outputs a residual image and starts at identity, $\mathcal{L}_{seg}$ is nnUNet's Dice-plus-cross-entropy loss, and $\lambda$ limits how far the corrected scan may drift from the original.
The corrector is deliberately anisotropy-aware. It uses separate in-plane and through-plane kernels, because 3D MRI slice thickness varies across protocols. The residual bound guards against the corrector "painting" anatomy for the frozen model.
For routing, we store for each site only the mean and covariance of the frozen backbone's first-stage features, pooled per volume. An incoming scan goes to the corrector whose site has the lowest Mahalanobis distance. If every distance exceeds a threshold calibrated on held-out sites, the scan goes to a shared fallback corrector. This fallback is trained by averaging corrector weights, so it costs nothing extra. No patient images leave a site or persist.
The analysis then has two parts. We break the remaining error into a part due to wrong routing and a part due to the limits of correction itself, by comparing against an oracle that knows the true site. We also ask which shifts are "input-repairable" by contrasting scanner-only shifts (prostate vendors, AMOS MRI scanners) with shifts that include population differences (hippocampus datasets mixing healthy and ADNI subjects).
Baselines
The naive reference points are the source-only frozen model with no adaptation and plain sequential fine-tuning. FR²Seg reports a backward transfer (BWT, the change in accuracy on earlier sites after later training) of −25.26% for sequential fine-tuning on six-site prostate MRI. Lifelong nnUNet reports the first site's Dice collapsing from about 85% to about 21% (Lifelong nnUNet). These anchor the scale of forgetting.
The strongest existing weight-updating competitor is FR²Seg. It is the most direct privacy-preserving continual multi-site segmenter, and it reports 88.30% average Dice with −1.43% BWT on the six-site prostate benchmark. We will reimplement its Fourier style replay and consistency regularization inside our 3D nnUNet pipeline, using the same site order, the same $k$ labeled volumes per new site, and the same tuning budget. Its published 2D number is a reference only, since its protocol uses full site training sets.
EWC is included because it is the standard regularizer. FR²Seg reports −8.18% BWT for it.
The most dangerous competitor is per-site LoRA inside the backbone, in the spirit of SD-LoRA. Each site gets its own low-rank weight update, routed exactly like our correctors. This makes it equally free of forgetting, but it changes the certified weights. It isolates whether "input-only" costs accuracy relative to "small weight change."
The key ablations are a pointwise intensity-only corrector, which is a learned monotone intensity mapping with no spatial context, plus classical histogram matching to the initial site. Together these isolate whether learned spatial correction matters at all.
Beating FR²Seg means average Dice within one point or higher with BWT closer to zero under oracle routing, and an end-to-end gap from oracle routing below one Dice point. Matching per-site LoRA within about one point would show that the locked-model constraint is nearly free.
Why not just…?
Why not just normalize intensities or histogram-match each scan? This is the real threat to the proposal, because much scanner shift is global contrast. FR²Seg's own premise is that site "style" lives in low frequencies. The pointwise and histogram ablations test this directly. If they recover most of the gap on prostate and AMOS MRI, we report that the learned corrector is unnecessary there and restrict our claim to shifts where it fails, such as through-plane resolution and coil inhomogeneity.
Why not just add per-site LoRA modules to the backbone? It is equally resistant to forgetting under routing, and it likely has more capacity. Its cost is regulatory: it modifies the certified predictor, which González et al. and INPUTADAPTER identify as triggering re-validation (González et al.). Our head-to-head comparison measures the accuracy price of avoiding that. If LoRA wins by a wide margin, the locked-model route is weak, and we will say so.
Why not just keep a 1–3% rehearsal buffer? ITL shows that 3–5% exemplars approach joint training, and Lifelong nnUNet finds rehearsal best overall. We include rehearsal as a non-deployable upper reference. The comparison it supports is whether privacy-safe isolation gets close to it, since retaining raw scans across institutions is what governance forbids.
Why not just use unsupervised test-time prompting like VPTTA? It needs no labels. However, its alignment loss depends on batch normalization, and weight-updating test-time methods like TENT degrade over rounds, losing 5.12 Dice in VPTTA's tests. We will adapt VPTTA with an instance-norm statistics substitute and run it on the same stream to measure the value of the few labels.
Proposed Experiments
The site streams are the six-site prostate MRI benchmark (RUNMC, BMC, HCRUDB, UCL, BIDMC, HK), the AMOS MRI subset split by scanner, and the three-dataset hippocampus benchmark used by ACS. Each stream runs in three random site orders, because Lifelong nnUNet found strong sensitivity to order. Metrics are per-site Dice, average Dice, BWT, routing accuracy, and Dice on held-out unseen sites under the fallback corrector.
There are three decisive checks. The first is whether the oracle-routed corrector beats histogram matching by a clear margin on AMOS MRI, where the reported cross-scanner drop exceeds 20%. The second is whether the end-to-end routing loss stays under one Dice point. The third is whether the corrector's advantage shrinks on the hippocampus stream, as the input-repairability hypothesis predicts for population shift. If the advantage instead holds there too, the corrector is doing more than fixing scanner artifacts, and its residual images need closer clinical inspection.
References
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Each Test Image Deserves A Specific Prompt: Continual Test-Time Adaptation for 2D Medical Image Segmentation
- Efficient Deformable Convolutional Prompt for Continual Test-Time Adaptation in Medical Image Segmentation
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors
- FR²Seg: Continual Segmentation Across Multiple Sites via Fourier Style Replay and Adaptive Consistency Regularization
- Towards Synchronous Memorizability and Generalizability with Site-Modulated Diffusion Replay for Cross-Site Continual Segmentation
- Adversarial Continual Learning for Multi-domain Hippocampal Segmentation
- Lifelong nnUNet: a framework for standardized medical continual learning
- SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning
- Regulating radiology AI medical devices that evolve in their lifecycle
- Incremental Learning Meets Transfer Learning: Application to Multi-site Prostate MRI Segmentation