Style-Factored Adapter Merging for Constant-Size Multi-Site Medical Segmentation

A segmentation model trained at one hospital often breaks at the next one, because a different scanner vendor, field strength or acquisition protocol changes how the same organ looks. When hospitals cannot pool their data, the model has to learn the new sites one at a time. The cleanest way to avoid forgetting earlier sites is to freeze the model and attach new parameters for each site. That makes the stored model grow with every hospital that joins, and when the site is unknown at test time, it also forces a choice of which attachment to use.

We propose a way to keep the deployed model at a fixed size however many sites arrive. The method separates each site's change into two parts. The first is a tiny input-level "style" code that captures intensity and contrast differences. The second is an anatomy-level correction, and all sites share one low-rank adapter for it. Our central bet is that merging many sites into one adapter fails mainly because site-specific style contaminates the shared update. Once style is removed at the input, the anatomy corrections from different sites largely agree and can be merged with little loss.

Core Claim

Two recent results point at each other without meeting. Merge before Forget (SLAO) keeps a single low-rank adapter (LoRA, a pair of small matrices $A$ and $B$ whose product is added to a frozen weight) at constant memory across a stream of language tasks. It does this by re-initializing $A$ from a QR basis of the previous adapter and merging task-specific $B$ matrices with a $1/\sqrt{i}$ weight. Separately, VPTTA shows that much of cross-site medical shift sits in the low-frequency amplitude of the image. In 2D fundus and polyp segmentation, a prompt of only 27 to 75 parameters in that frequency band recovers a large part of the gap, from 65.86% to 71.93% mean Dice on optic disc and cup.

Our claim joins the two in three parts. First, continual merging of one adapter works in multi-site 3D medical segmentation only if the low-frequency style is factored out first. Second, after factoring, the per-site $B$ updates show much lower directional conflict than without it. Third, the resulting model's stored size is

$$P(T) = P_0 + L(m+n)r + T\,s,$$

where $P_0$ is the frozen backbone, $L$ the number of adapted layers of shape $m\times n$ at rank $r$, $T$ the number of sites seen, and $s$ the size of one style code, a few hundred numbers. The only term that grows with $T$ is negligible next to everything else. This differs from LoCo-PVT, which adds a fresh set of adapters per dataset and so grows as $O(T)$, though slowly (16.7% parameter increase over three steps). LoCo-PVT also targets new organs, not the same organs under new scanners.

Research Program

The backbone is a convolutional 3D U-Net. We choose it because the AMOS benchmark found the classic 3D U-Net ahead of Swin-UNETR on CT (89.04% vs 86.32% mean Dice). Almost every parameter-efficient continual method in the file instead assumes Transformer layers, including OSFT, CONEC-LoRA and DoT. We therefore define low-rank adapters for 3D convolutions by reshaping each kernel into an output-by-(input × kernel volume) matrix and adding a rank-$r$ product.

When site $t$ arrives with labeled data, training proceeds in two stages. The first stage fits a 3D low-frequency amplitude code that maps the site's volumes toward the source site's feature statistics. This extends VPTTA's 2D frequency prompt to volumes, which VPTTA lists as open, and uses the same batch-normalization statistics alignment loss. The style code is stored per site. The second stage trains a fresh $B_t$ on the style-normalized volumes, with $A$ initialized from the QR basis of the current merged adapter as in SLAO. $B_t$ is then merged into the running $B$ with weight $1/\sqrt{t}$, and $B_t$ is discarded.

At test time no site label is needed. We pick the stored style code whose induced feature statistics best match the incoming volume, or fit a new code for a few steps if none matches well. The test volume then goes through the single merged adapter. This lookup costs $T$ forward statistics passes over a few-hundred-parameter bank, not a learned site classifier like CONEC-LoRA's.

The analysis has three parts. First, we test whether SLAO's asymmetry, a stable $A$ with task-specific $B$, holds for convolutional adapters under scanner shift; this had only been shown in language models. Second, we measure conflict between sites' $B_t$ updates, as the pairwise cosine similarity between them, with and without style factoring. Third, we test whether small structures absorb the merging loss. AMOS reports that cross-domain failures concentrate in small organs such as the adrenal glands and duodenum, so we report per-organ Dice and not only means.

Baselines

The naive reference points are the frozen source model applied to every site and plain sequential fine-tuning. The latter shows the forgetting being prevented: ACS reports that a standard U-Net loses over 35% Dice on its first hippocampus dataset after training on the third. Offline joint training on pooled sites is the upper bound.

The strongest existing method on these benchmarks is SMG-Learning. It replays synthetic past-site images from a diffusion generator with learned per-site prompts and aligns their gradients with the incoming site. On six-site prostate MRI it outperforms the second-best continual method by 2.13% overall Dice and has the best backward and forward transfer. On four-vendor cardiac MRI it trails joint training by only 0.58% Dice. The file gives no absolute Dice for SMG-Learning, so we will rerun it under its own site orders, splits and preprocessing.

We count our method as beating SMG-Learning if it matches its average Dice within 0.5 points and matches or exceeds its backward transfer. It must do so while storing no generative model and adding under 1% of backbone parameters across the stream. We would also report outright accuracy wins, but they are not the claim.

The capacity reference is per-site LoRA stacking in the style of LoCo-PVT, run two ways: with an oracle site label, and with a learned site classifier. The oracle version shows the accuracy that unbounded growth buys. The classifier version shows what routing errors cost. Single-adapter SLAO without style factoring is the ablation that isolates our key ingredient. A second ablation keeps style factoring but merges $B$ by plain averaging, isolating the merge schedule. We also include a 3D VPTTA-style prompt with no adapter, TENT, and MAS/EWC-regularized full fine-tuning.

All methods share the same 3D U-Net, training epochs per site, augmentation and site orders. Each gets the same hyperparameter search budget, tuned on the first two sites only, so no method sees future sites during tuning.

Why not just…?

Why not just store a separate LoRA per hospital, since storage is cheap? Disk is cheap, but routing is not. CONEC-LoRA needs a separate multi-depth classifier to pick the domain adapter, and on natural-image benchmarks it gets within 1% of an oracle. Scanner differences inside a single vendor family may be far subtler, so routing errors could be worse here. The oracle-versus-classifier comparison measures exactly this cost. If the classifier version matches the oracle and both beat our merged adapter clearly, per-site stacking is simply better, and we would report that.

Why not just use a per-site input prompt or per-site normalization statistics, with no adapter at all? This is the most serious threat. If scanner shift were purely low-frequency style, the style code alone would close the gap and the shared adapter would be dead weight. VPTTA's numbers say style correction helps substantially, but its 71.93% fundus Dice still sits well below what supervised multi-site models reach. On the related fundus benchmark, the personalized ensemble P-MoLE reports 94.01% Dice. That gap suggests structure beyond style, but the comparison crosses settings and is not proof. The prompt-only baseline settles this directly. If it matches the full method on prostate and cardiac, we narrow the claim to say adapters matter only for harder shifts, such as AMOS's cross-vendor MRI, where generalization drops exceed 20% Dice.

Why not just fine-tune everything with an importance penalty like EWC? This adds zero parameters. However, ACS reports that a MAS-regularized baseline is beaten by +0.127 Dice retention on sequential hippocampus scanners, and OSFT argues that such soft penalties degrade steadily over long sequences. We include these baselines anyway, because short medical streams may not expose that long-sequence weakness.

Why not just replay synthetic images, as SMG-Learning does? Replay works, but it requires training and keeping a diffusion model plus a prompt for every site, which is the overhead we are trying to remove. Head-to-head comparison against SMG-Learning tells us whether that overhead buys accuracy that the constant-size model cannot reach.

Proposed Experiments

The primary streams are six-site prostate MRI (RUNMC, BMC, HCRUDB, UCL, BIDMC, HK), four-vendor cardiac MRI and three-cohort hippocampus MRI. Each is run in at least three site orders, with one site held out entirely to measure generalization to unseen sites.

The 6-site, 4-vendor and 3-cohort streams are too short to stress constant-size merging, so we build longer streams two ways. The first splits AMOS by its eight scanners, covering both CT and MRI. The second runs repeated cycles over the prostate sites, following VPTTA's multi-round protocol, where TENT lost 5.12 Dice while VPTTA lost only 0.42. Metrics are mean and per-structure Dice, surface distance where the benchmark reports it, backward and forward transfer, stored bytes as a function of $T$, and test-time latency.

Three results would decide the claim. First, single-adapter merging without style factoring should fall measurably below the oracle per-site stack, while factoring should recover most of that gap. Second, the update-conflict measure should drop under factoring, linking the accuracy recovery to the proposed mechanism and not to extra tuning. Third, along the long AMOS and cyclic streams, the factored model's Dice on the earliest sites should stay flat while its parameter count stays fixed.

If the first result holds and the second does not, the gain is real but our explanation is wrong. If merging without factoring already matches the oracle stack, the style factoring is unnecessary and SLAO transfers directly to medical imaging. That would be a simpler and still useful finding.