Medical segmentation models trained on scans from one hospital often fail badly at another. In one vestibular schwannoma study, a network that scored about 95% overlap on its home data dropped to about 16% on routine scans from other UK hospitals. Faint tissue edges are where this failure should concentrate. Inside a tumor, the evidence is strong and redundant. At its edge, the model must separate a weak intensity change from noise, and the noise pattern belongs to a particular scanner. We propose a narrow fix for a common deployment case: one center's labeled data, and one unlabeled scan from a new center at test time.
The method keeps the trained network's confident interior prediction. It re-solves only the thin uncertain band around the predicted boundary, using sparse dictionary coding: each small image patch is written as a combination of a few learned template patches, called "atoms." The anatomy atoms, each paired with a label patch, are learned once at the home center and then frozen. At test time, a small set of extra "acquisition atoms" is fitted by alternating optimization to that one new scan. They are pushed to absorb what the new scanner does differently, such as blur, noise texture, or contrast shape, without absorbing anatomy. Boundary labels are then read off the anatomy atoms alone. In two sentences: keep what the network is sure of, and at the uncertain edge ask which home-center edge patterns, plus a scan-specific model of the new scanner's distortions, best explain the pixels. That question is answered by per-scan iterative optimization rather than by a frozen single-pass network.
Core Claim
We claim three things. First, a large, measurable share of single-center failure lies in the boundary band rather than in missed structures. Second, that share can be recovered at test time without target labels or multi-center training data. Third, recovery depends on two ingredients existing methods lack: iterative, non-amortized sparse inference, and target-fitted atoms kept separate from source anatomy atoms. The first claim is an empirical diagnosis we test before building anything. If it fails, the method has little to fix, and we say so.
The closest prior work is Cross-Domain Visual Recognition via Domain Adaptive Dictionary Learning. That paper splits a dictionary into a common part and domain-specific parts, keeps them mathematically dissimilar ("incoherent"), and applies this to whole-image classification of faces and objects. It needs several labeled or observed domains at training time. Our setting differs in three ways: only one domain is available for training, the per-domain atoms are fitted after deployment on a single scan, and the output is a dense boundary label rather than a class.
Discriminative dictionary learning for abdominal multi-organ segmentation supplies the idea of predicting labels from sparse codes of local patches. It also shows that local choices beat global ones on a low-contrast organ: pancreas Dice rose from 65.5% to 71.1%. That work runs on one 150-scan cohort, depends on atlas registration, and has no notion of scanner-specific atoms. Coupled Dictionary Learning for Multi-contrast MRI Reconstruction supplies the coupled-dictionary device: shared structural atoms plus contrast-specific atoms. It uses the device for reconstruction quality, not segmentation, and within one repository.
Research Program
Source phase. A standard segmentation network, nnU-Net, is trained on the single source center. From source scans we extract small 3D intensity patches centered on voxels within a few millimetres of the true boundary. Each patch is locally normalized to zero mean and unit norm, which removes local gain and offset before any dictionary sees it. We learn an anatomy dictionary $D_A$ jointly with a label dictionary $L$, so that an image patch $y$ and its label patch $\ell$ share one sparse code $x$ ($y \approx D_A x$, $\ell \approx L x$). Training alternates sparse coding and atom updates. We use the parallel batched solvers of DB-KSVD: Scalable Alternating Optimization for Disentangling High-Dimensional Embedding Spaces so this is cheap at millions of patches.
Test phase, per target scan. The network produces a probability map. We define the ambiguous band as voxels with probability between 0.1 and 0.9, or within $r$ mm of the predicted boundary. Voxels with probability above 0.95 or below 0.05 form a confident anchor set $\mathcal{C}$, and their thresholded predictions serve as pseudo-labels $\hat\ell$. With $D_A$ and $L$ frozen, we fit a small target dictionary $D_T$ of roughly 8 to 32 atoms by alternating minimization of
$$\min_{D_T, X, Z} \sum_{i} \|y_i - D_A x_i - D_T z_i\|_2^2 + \lambda\|x_i\|_1 + \mu\|z_i\|_1 + \beta \sum_{i \in \mathcal{C}} \|\hat\ell_i - L x_i\|_2^2 + \gamma \|D_A^\top D_T\|_F^2 .$$
The terms are defined as follows. The index $i$ runs over patches from both the band and the anchor set. $x_i$ and $z_i$ are the sparse codes on anatomy and target atoms. $\lambda$ and $\mu$ set sparsity. The fourth term makes the anchor patches, whose labels we trust, decode correctly through anatomy codes alone. That forces $D_T$ to explain the scanner-specific residual rather than the anatomy. The last term is the incoherence penalty from the domain-adaptive dictionary work, which discourages $D_T$ from copying anatomy atoms.
Band voxels then receive the label $L x_i$, averaged over overlapping patches. This label is fused with the network probability using a weight chosen once on source validation data. Because $D_T$ is fitted per scan, the method needs no target cohort. If several scans from one site are available, the same objective can pool them into a site-level $D_T$.
Why iterative inference matters here. Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders proves that a single-layer encoder cannot match iterative sparse coding at recovering codes, and reports iterative coding beating such encoders by over 10% in feature recovery on synthetic data. That result motivates the design; it does not guarantee it works on MRI patches. We therefore test it directly with an ablation that replaces the iterative solver with a trained feedforward encoder.
RecycleNet: Iterative Decision Refinement via Latent Feature Recycling shows that repeated passes sharpen segmentation. Its refinement, however, stays inside the network's fixed weights and has nothing that can absorb a new scanner's statistics. That is exactly the gap $D_T$ fills.
Known risk. The additive model $D_A x + D_T z$ is an approximation. Scanner blur is closer to a convolution, and slice thickness changes the patch geometry. Local normalization handles gain and offset. For thick slices, we additionally learn $D_A$ from source patches degraded by random through-plane downsampling. If residual error in the band on the target stays high after fitting $D_T$, the additive model has failed, and the reconstruction error is directly measurable to tell us so.
Baselines
The naive reference is the source-only nnU-Net applied to target scans with no adaptation. On the vestibular schwannoma single-center-to-routine transfer, Deep learning for automatic segmentation of vestibular schwannoma: a retrospective study from multi-center routine MRI reports that a model trained on single-center Gamma Knife data (SC-GK) fell from about 95% to as low as about 16% Dice on routine multi-center scans (MC-RC). We retrain this model ourselves on the same splits to get per-case numbers. The same paper supplies an oracle ceiling we do not expect to reach: training on multi-center routine data gives a median of 89.4% Dice on T2-weighted scans.
The strongest simple competitor is histogram matching of each target scan to the source intensity distribution before segmentation. In AtrialGeneral: Domain Generalization for Left Atrial Segmentation of Multi-Center LGE MRIs it raised unseen-center left atrial Dice from 0.678 to 0.772, beating mutual-information disentanglement (MID-Net, 0.741) and random style transfer (RST-Net, 0.756). We run it on identical networks and data. Because our method is a post-hoc step, we also run it on top of histogram matching. Beating this baseline means a statistically significant gain in boundary metrics when our method is added to histogram-matched inputs, not merely when compared against raw inputs.
The strongest boundary-specific method is the hardness-weighted Dice loss with supervised spatial attention of Automatic Segmentation of Vestibular Schwannoma from T2-Weighted MRI by Deep Spatial Attention with Hardness-Weighted Loss. It reported 0.87 Dice and 0.43 mm average symmetric surface distance (ASSD) on single-center T2 scans. We retrain it on the same source-only data and evaluate it cross-center, where no reported result found on Lacuna exists. For the cross-modality regime we compare against the translation-plus-self-training pipelines of the crossMoDA challenges. crossMoDA Challenge: Evolution of Cross-Modality Domain Adaptation Techniques for Vestibular Schwannoma and Cochlea Segmentation from 2021 to 2023 reports top median Dice above 87% for tumor and 84% for cochlea. Weakly Unsupervised Domain Adaptation for Vestibular Schwannoma Segmentation reports mean Dice 0.77 and ASSD 0.61. Those pipelines use a whole unlabeled target cohort, whereas we use one scan at a time, so we report both conditions separately.
Test-time network adaptation must also be a baseline, run with the same per-scan data and a matched wall-clock budget. Concretely, this means recomputing normalization statistics on the target scan and minimizing prediction entropy for a few steps. No reported result found on Lacuna for this on these cohorts. RecycleNet-style recycling, trained on source, serves as the "iterative but no dictionary" competitor.
Three ablations isolate our ingredients. Setting $D_T$ to zero leaves source atoms only and tests whether target-fitted atoms matter. Replacing iterative coding with a trained amortized encoder tests the amortization argument. Dropping the anchor term and the incoherence term tests whether $D_T$ stays anatomy-free. All hyperparameters for every method are tuned only on held-out source-center data, with an equal search budget of about 20 configurations each. No method sees target labels.
Why not just…?
Why not just histogram-match the target scan? This is the most serious threat, because it already beat deep disentanglement in AtrialGeneral. However, it applies one global intensity remapping. It cannot model scanner-specific noise texture or blur at a faint edge, and AtrialGeneral itself notes it struggled with post-ablation scar. It is also complementary rather than competing, so our main comparison stacks our method on top of it. If histogram matching alone closes most of the boundary-band error, the dictionary stage is not worth its cost, and the band-error decomposition described below will show that directly.
Why not just adapt the network's normalization layers or minimize entropy at test time? This is also a real threat. It is cheap and per-scan, like ours. The difference is that it changes weights that affect the whole image, including interiors the network already gets right. It also has no explicit mechanism that stops it from sharpening toward wrong but confident boundaries. We run it under a matched compute budget. If it matches our boundary-band gains, we will report that the dictionary structure adds nothing beyond per-scan adaptation.
Why not just train with heavy intensity augmentation or image translation? Augmentation did not prevent the roughly 16% collapse reported by Kujawa et al. Translation pipelines plateau near 88% Dice even with whole target cohorts, and Hardan et al. needed extra augmentation to suppress artifacts introduced by the translation networks. Neither changes inference on an individual unseen scan. We nonetheless apply nnU-Net's default augmentation to every method, so any gain we report is on top of it.
Why not just add a boundary loss? Hardness weighting fixes the network after training. Wang et al. also document a narrow working range beyond which it overfits to noise and artifacts. Across centers, the most "hard" voxels are likely to be scanner artifacts, so we expect it to help in-center and less cross-center. The retrained baseline tests exactly this.
Proposed Experiments
The primary regime is vestibular schwannoma on T2-weighted MRI. We train on SC-GK and test on MC-RC routine scans, which include 1.0T to 3.0T scanners and varied slice thickness, and on the independent Tilburg set, as described by Kujawa et al. and the crossMoDA 2023 cohort. The second regime is left atrial LGE-MRI from AtrialGeneral's three centers: we train on one center and test on each of the other two. It is worth stating plainly that AtrialGeneral's 0.772 figure comes from its own protocol, so we rerun histogram matching under ours. A third, in-center sanity check uses the 2018 atrial challenge data. There, Combating Uncertainty with Novel Losses for Automatic Left Atrium Segmentation reports 92.24% Dice; our method should not hurt performance in that setting.
Metrics are Dice and ASSD. We also compute Dice restricted to the boundary band. Crucially, we split each failed case into a detection error (a missed or spurious connected component) and a boundary error (a correctly detected component with a wrong outline), because our method can only address the latter.
The first decisive check comes before any method-building: on source-only nnU-Net predictions for MC-RC, what fraction of lost Dice sits in correctly detected components? If most of the drop to about 16% comes from missed tumors, the proposal's scope shrinks accordingly and we report boundary gains only on detected cases.
The second check is whether our method improves boundary-band Dice and ASSD over histogram matching plus test-time normalization, with paired tests across scans and stratified by scanner field strength and slice thickness. The third check is whether the gain vanishes when $D_T$ is removed or the solver is amortized; if it does not, our mechanism is not the cause. Finally, we check whether $D_T$ really stays anatomy-free. We measure the similarity between $D_A$ and $D_T$ atoms, and we verify that decoding target patches with $D_T$ alone yields near-chance labels. If $D_T$ carries label information, it has absorbed anatomy and the separation claim fails. We also report small post-operative residuals (under 30 mm³) and dark tumors with weak T2 signal separately, since those are where the crossMoDA plateau lives.
References
- Cross-Domain Visual Recognition via Domain Adaptive Dictionary Learning
- Discriminative dictionary learning for abdominal multi-organ segmentation
- Coupled Dictionary Learning for Multi-Contrast MRI Reconstruction
- DB-KSVD: Scalable Alternating Optimization for Disentangling High-Dimensional Embedding Spaces
- Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders
- Abstract: RecycleNet - Latent Feature Recycling Leads to Iterative Decision Refinement
- Deep learning for automatic segmentation of vestibular schwannoma: a retrospective study from multi-center routine MRI
- AtrialGeneral: Domain Generalization for Left Atrial Segmentation of Multi-center LGE MRIs
- Automatic Segmentation of Vestibular Schwannoma from T2-Weighted MRI by Deep Spatial Attention with Hardness-Weighted Loss
- crossMoDA Challenge: Evolution of Cross-Modality Domain Adaptation Techniques for Vestibular Schwannoma and Cochlea Segmentation from 2021 to 2023
- Weakly Unsupervised Domain Adaptation for Vestibular Schwannoma Segmentation
- Combating Uncertainty with Novel Losses for Automatic Left Atrium Segmentation