A Closed-Form Site Correction for Locked Medical Segmenters

Medical image segmenters trained at one hospital often fail at another, mostly because a different scanner renders the same tissue with different intensities and contrast. Regulators make the obvious fixes hard. A model certified as a medical device cannot have its weights changed. Patient scans from the original training site usually cannot leave that site. And the methods that adapt the model while it runs add per-image optimization, which is too slow for live imaging. We propose a small linear correction that is fitted once per new site, in seconds, from a handful of unlabeled scans. It is inserted as a 1×1 convolution after the first block of the frozen network, and it needs only a compact summary stored at the source site, not source images.

The summary comes from projective dictionary learning: a family of methods that learns, for each class, a pair of matrices. One matrix encodes a feature vector into a code in a single multiplication. The other decodes it back. The pair is trained so that the encoder for one class sends features of every other class to nearly zero. We learn such a pair for each anatomical class (background, left ventricle, myocardium, right ventricle) on the frozen network's own early features. At a new site we then solve for a linear map that makes target features look "explainable" by this class-structured summary again. The two-sentence version is this: each anatomical structure gets a fast linear encoder–decoder fitted on source-site features, and a new scanner's features are corrected by the one linear map under which every pixel is again well reconstructed by exactly one structure's pair. Because the map has a closed-form update and runs as a single matrix multiply per pixel, adaptation adds almost no latency and leaves the certified weights untouched.

Core Claim

Scanner shift in MRI mostly corrupts the relations between tissue contrasts in early feature channels. This kind of corruption pushes features out of the per-class subspaces that the source network relies on. A class-structured analysis–synthesis summary can detect that departure without labels, through reconstruction residuals and null-space leakage, and undo it in closed form. Our specific bet is that the class structure matters. Projecting onto a single source subspace (essentially PCA), or matching stored per-channel statistics, should recover less than a correction that also requires each pixel to be cleanly claimed by one anatomical class. If that bet fails, the method reduces to known baselines, and the experiments are built to show that.

Relation to Prior Work

The encoder–decoder mechanism comes from Projective dictionary pair learning for pattern classification. That paper learns class-specific encoder/decoder pairs whose encoders send other classes to a near-null space, and it classifies by minimal reconstruction error. It reports 98.3% on AR Face with training in 11.3 seconds. The block-diagonal and graph-regularized variant in Scalable Block-Diagonal Locality-Constrained Projective Dictionary Learning reports 78.0% versus 73.5% for the original method on Caltech 256. Both are fully supervised, and both name unsupervised target distributions as an open problem. We use their pairs only at the source, where labels exist, and add a label-free target step.

The idea of a linear bridge between two domains' codes comes from Discriminative Semi-Coupled Projective Dictionary Learning for Low-Resolution Person Re-Identification and Cross-View Projective Dictionary Learning for Person Re-Identification. Those methods learn view- or resolution-coupled pairs, but they require identity-matched pairs across domains. We learn the bridge from unpaired, unlabeled target pixels.

The label-free target constraint, that each target sample should be reconstructed by only one class block, is borrowed from Domain-Shared Group-Sparse Dictionary Learning for Unsupervised Domain Adaptation. That method uses iterative sparse coding and needs source and target data together. Cross-Domain Visual Recognition via Domain Adaptive Dictionary Learning shares the same limitations. Neither was tested on medical scanners.

The closest medical work is Single-Domain Generalization in Medical Image Segmentation via Test-Time Adaptation from Shape Dictionary. It learns an ordinary dictionary over label masks and adapts batch-normalization parameters per image, at 56 ms per 2D slice. It reaches 81.1% Dice on unseen prostate sites. We differ on three counts: our dictionary lives in feature space rather than shape space, we never touch network weights, and we fit once per site rather than per slice.

The regulatory motivation, that the predictor stays locked and only the input is transformed, is set out by Adapt the Data, Not the Model. Its neural input adapter needs target labels and a source exemplar bank. We need neither.

Two papers constrain the design. The scanner-shift study The Impact of Scanner Domain Shift on Deep Learning Performance in Medical Imaging finds MRI far more fragile than CT. It also finds that generic noise augmentation does not help, so we target MRI and do not expect augmentation-style fixes to substitute. The M&Ms Challenge finds that intensity-type augmentations beat spatial ones for cross-vendor robustness. That supports placing our correction in early, intensity-sensitive features.

Research Program

At the source site, we run the frozen segmenter on labeled training scans and collect per-pixel feature vectors $f \in \mathbb{R}^C$ after the first encoder block. For each class $k$, we learn an encoder $P_k$ and decoder $D_k$ with the dictionary-pair objective of Gu et al., using labels downsampled to that resolution. We also store the network's existing per-channel normalization statistics. Only these matrices leave the site. Their size is a few thousand numbers per class for $C$ around 64, and they contain no images. Whether feature-space atoms leak patient information is a question we will probe with a simple inversion attack, not assume away.

At a target site, we collect unlabeled target features $f_t$ from a few scans. We then fit a channel-mixing matrix $W \in \mathbb{R}^{C\times C}$ together with per-pixel class assignments $z_t$ by solving

$$ \min_{W,\{z_t\}} \sum_t \Big( \|W f_t - D_{z_t} P_{z_t} W f_t\|_2^2 + \beta \sum_{j \neq z_t} \|P_j W f_t\|_2^2 \Big) + \gamma \|W - I\|_F^2 . $$

In this objective, $\beta$ weights leakage into other classes' encoders. The $\gamma$ term anchors $W$ to the identity, which rules out the trivial solution $W=0$ and keeps the correction small. We will also test an added constraint that corrected features match the stored normalization means.

The problem is solved by alternation. Each assignment step picks the class with the smallest penalized residual. Each $W$ step is a linear least-squares problem in the $C^2$ entries of $W$, so it has a closed-form solution. At deployment the frozen network receives $W f$, which is a single 1×1 convolution. The decoder reconstruction $D_{z}P_{z}Wf$ is used only for fitting and as a per-pixel reliability score, never as the network input. Feeding the reconstruction instead is an ablation.

Three analyses accompany the method. The first measures how adaptation quality depends on the number of unlabeled target volumes (1, 5, 20). The second compares the placement depth of the correction. The third tests whether the residual flags pixels where the frozen model is wrong, which would give a calibration signal for free.

Baselines

The naive reference is the frozen source model applied unchanged. On M&Ms this is the model whose unseen-vendor drop the challenge measured at about 6% Dice for baselines. On the scanner-shift study's prostate task, a Siemens-trained classifier fell from 0.791 to 0.505 AUC.

The strongest existing competitor that also leaves weights untouched is OptTTA. OptTTA searches augmentation policies at test time to minimize prediction entropy and match stored normalization statistics, and it uses no source data. It reports a 22.5% average Dice gain over the unadapted model on M&Ms and 9.2% on spinal cord grey matter. It is the right bar because it satisfies the locked-model and source-free constraints. We will run its public code on the same frozen backbone and the same target volumes, with its default search budget. We will record both Dice and wall-clock adaptation cost. Beating it means matching its Dice gain within one point while adapting more than an order of magnitude faster. Clearly exceeding its gain at any speed would be a stronger result.

For the weight-updating regime we include the shape-dictionary method TASD, which reports 81.1% average Dice on unseen prostate sites and 85.8% on fundus. It serves as a reference ceiling rather than a fair competitor, since it modifies weights. Approaching it without modifying weights would be meaningful.

Three ablations isolate the key ingredient. In the first, $W$ is fitted to project onto a single unstructured source subspace learned by PCA on all classes pooled. In the second, a diagonal $W$ plus bias is fitted only to match stored normalization statistics. This is AdaBN recast as an external adapter. In the third, we use the full objective but with $\beta=0$, so there is no null-space leakage term. All variants receive the same target volumes, the same frozen network, and the same tuning budget, with $\beta$ and $\gamma$ selected on one held-out vendor and then fixed.

Why not just…?

Why not just histogram-match target scans to a stored source intensity profile? This is a real threat for MRI, because the M&Ms results show intensity-driven fixes carry much of the cross-vendor gain. It needs only a stored histogram, which raises little privacy concern, and it costs nothing. We include it both as a baseline and as preprocessing beneath our method. If our correction adds nothing on top of histogram matching, then the shift is essentially monotone in intensity and the claim fails. A gain on top of it, especially on the unseen Canon vendor, would show that cross-channel contrast relations matter beyond a single intensity curve.

Why not just match per-channel feature statistics, or project onto a source PCA subspace? Both are cheaper relatives of our method, and either may capture most of the effect. We do not argue around this. The diagonal moment-matching ablation and the PCA ablation answer it directly. The proposal stands only if the class-structured objective beats both on Dice at unseen vendors, and if the advantage grows with shift severity: larger for Canon than for seen vendors, and larger for MRI than for CT on AMOS.

Why not just fine-tune on 10–20% labeled target data, which the scanner-shift study found recovers much of the loss? That approach changes certified weights and requires annotation at every new site, so it is ruled out by the deployment constraint rather than by accuracy. We still report it as an upper bound. This shows how much of the fine-tuning recovery a label-free, weight-free linear correction can reach.

Proposed Experiments

The primary benchmark is M&Ms. We train the backbone (an nnU-Net-style 2D U-Net) on Vendor A, then adapt to Vendors B and C and to the unseen Canon Vendor D, reporting left ventricle, myocardium, and right ventricle Dice. The second benchmark is multi-site prostate MRI: training on NCI-ISBI13 and testing on the I2CVB and PROMISE12 sites, which allows direct comparison with TASD's reported 81.1%. Spinal cord grey matter provides a comparison point against OptTTA's reported gain. AMOS serves as a modality contrast. The scanner-shift study predicts small CT gains and large MRI gains, and AMOS reports drops above 20% on unseen MRI scanners. Our method should follow that ordering, and failure to do so would indicate that it is fitting something other than scanner shift.

For each method we report Dice, adaptation wall-clock time per site and inference overhead per slice, and the number of unlabeled target volumes used. We also measure expected calibration error of the frozen model's outputs when they are modulated by our residual. That measurement is set against the post-hoc calibration results of Ouyang et al., who report 14.48% to 12.78% on ACDC with simulated artifacts.

The decisive check is the gap between the full objective and the PCA and moment-matching ablations on Vendor D. If that gap is under one Dice point, the anatomical class structure is not doing work, and we would report the simpler adapter instead.