Treating Shifted Feature Tokens as Missing Values in Frozen Medical Segmenters

A segmentation model trained at one hospital often fails at another because the new scanner produces images the model has never seen. Under a regulatory lock, the hospital is not allowed to retrain the certified model. It often has few or no labeled local scans anyway. This note proposes a fix that leaves every weight untouched. Inside the frozen network, find the small image regions whose internal feature vectors fall outside anything the model saw during training. Treat those vectors as missing data. Fill them in from similar regions of the training scans, using the same nearest-neighbor imputation that clinicians' data teams already use for missing laboratory values. The repair is local and needs no target labels. It can also decline to act: regions with no trustworthy training analogue are flagged rather than filled.

The bet is that scanner shift does not damage a model's internal representation uniformly. It damages it in patches, such as low-quality slices, coil-shading regions, and thin low-contrast structures. Global fixes like intensity normalization or recomputing feature statistics cannot target those patches, but a per-location missingness test can. The evidence for patchy damage is indirect but suggestive. In the FeTA 2024 fetal brain challenge, image quality moved Dice by about 0.10, more than acquisition site did at about 0.075 (FeTA 2024). In AMOS, cross-scanner failures concentrate on small structures like the adrenal glands and inferior vena cava (AMOS).

Core Claim

In a frozen 3D segmentation network, each spatial location at an intermediate layer has a feature vector, or token. Under scanner shift, the tokens far from every source token are the ones where the prediction is wrong, and they form a minority of locations. Replacing only those tokens with a locally linear reconstruction from their nearest source tokens should recover more Dice and topology on small structures than three alternatives: global feature-statistics recalibration, plain k-nearest-neighbor averaging, and nearest-neighbor label transfer. It should also roughly match low-rank fine-tuning on a handful of labeled target scans, without changing a single weight. The method must abstain where source support is absent, which is the expected regime for CT-to-MRI transfer. That abstention is part of the claim, not a failure of it.

How it relates to prior work

Two papers already combine nearest-neighbor retrieval with frozen models and supply the motivating evidence, but neither does dense prediction under imaging shift. Borrowing Treasures from Neighbors retrieves complete-modality training examples by global image-level similarity. It feeds them through a trained cross-attention module on a frozen ViLT backbone to compensate for missing modalities. At 1% of training data it gains 6.1% over a multi-task baseline and beats full fine-tuning. It works only on classification, in distribution, with one global vector per example. We retrieve per-location tokens and train nothing.

INPUTADAPTER adapts target inputs for a regulatory-locked predictor, optionally using a k-NN source memory bank. It reports +14.1 AUCPR points on eICU acute kidney injury. It operates on 1D ICU time series, however, and trains its adapter with target labels. Our proposal is label-free, operates on volumetric feature maps, and has no trainable adapter.

The imputation method itself comes from the tabular literature. Locally Linear Reconstruction computes donor weights by solving a small constrained least-squares problem rather than averaging. It won 242 of 336 UCI conditions against mean, hot-deck, and expectation-maximization imputation. Parimputation argues that only "predictable" gaps, those in dense donor neighborhoods, should be filled, and that isolated gaps should be left flagged. We use both ideas, moved from tabular attributes to deep feature tokens.

Two results constrain the design. Nearest Neighbor Sampling for Covariate Shift Adaptation proves that 1-NN is optimal for imputing target labels. The proof assumes target support lies inside source support, an assumption that heavy acquisition shift breaks. We therefore treat 1-NN as a baseline to beat and add abstention for out-of-support tokens. Missing Value Imputation for Diabetes Prediction found k-NN imputation robust across a wide range of k (50 to 1000) on tabular EHR. In a feature space with hundreds of dimensions, distances may concentrate and break that robustness, so k must be tested rather than assumed.

Method

Train a source segmentation network once and freeze it. We will use an nnU-Net-style 3D U-Net, because AMOS found 3D-UNet ahead of Swin-UNETR (89.04% versus 86.32% mDice on CT). Choose a layer, which might be the bottleneck or one decoder stage. Pass the source training scans through the network and store a subsampled bank of their tokens at that layer. The bank is stratified by the source model's predicted class so that small organs are not swamped. It is indexed by an approximate nearest-neighbor graph, with dimensions reduced by PCA fitted on source tokens only.

At test time, for each target token $z$, retrieve its $k$ nearest source donors $d_1,\dots,d_k$. Let $\rho(z)$ be the mean distance from $z$ to its donors. A token counts as missing if $\rho(z)>\tau$. Here $\tau$ is the 99th percentile of $\rho$ computed leave-one-scan-out on source scans, so it is fixed without any target data. A missing token counts as predictable if the reconstruction residual below is under a second source-calibrated threshold. Otherwise it is unpredictable: it is left untouched and reported in an uncertainty map. Predictable missing tokens are replaced by

$$\hat z=\sum_{j=1}^{k} w_j d_j,\qquad w=\arg\min_{w}\Big\|z-\sum_{j=1}^{k} w_j d_j\Big\|^2+\lambda\|w\|^2\ \ \text{s.t.}\ \sum_j w_j=1,$$

where $w_j$ are reconstruction weights and $\lambda$ is a small ridge term for numerical stability. This is the locally linear reconstruction objective applied to observed token coordinates. The repaired feature map continues through the frozen remainder of the network. The approach is parameter-efficient in the strictest sense: it trains no parameters. Its only hyperparameters are $k$, the layer, and two thresholds, all set on source data.

Research Program

The work has three stages, each able to kill the idea early.

The first stage is diagnostic. Before any imputation, we measure whether the missingness mask predicts errors. We take the area under the ROC curve of $\rho(z)$ as a detector of voxel-level segmentation error, upsampled to image resolution. We also measure the fraction of tokens flagged, in-domain and under shift. The method only makes sense if flagged tokens are a minority and are enriched for errors. If shift corrupts almost every token, global recalibration is the right tool and we stop.

The second stage is the main comparison under scanner shift with no target labels. Here selective locally linear reconstruction is compared against the baselines below, with all hyperparameters fixed from source data. The third stage tests the small-sample axis. We shrink the source cohort to 10, 25, and 50 scans, consistent with the under-100-sample regime described by Bria et al.. This asks whether a thinner donor bank makes imputation fail sooner than it degrades the frozen model itself.

Baselines

The naive reference is the frozen source model applied unchanged, with standard intensity normalization and the source model's own augmentation already baked in. On AMOS this is the regime reported to lose more than 20% mDice on unseen MRI scanners. Every method must beat it, and its gap to in-domain performance defines how much can be recovered.

The strongest label-free competitor is feature-statistics recalibration. This means re-estimating normalization-layer statistics on the unlabeled target scans, a standard test-time adaptation step. It is also label-free and parameter-free, which is exactly what our method claims to be. If shift is mostly a global change of feature mean and variance, it will match us. It uses the same frozen model and the same unlabeled target pool. No reported result found on Lacuna for normalization-statistics recalibration on AMOS or FeTA, so we will produce it ourselves. Histogram matching of target volumes to a source template is a second label-free baseline, motivated by its partial failure in the COVID-19 small-data study.

The strongest method that does use target labels is low-rank adapter fine-tuning of the source model on five labeled target scans. It stands in for "just adapt it," with the regulatory lock set aside. Full fine-tuning on the same five scans is a second variant. Both receive the same GPU budget and a tuning sweep of equal size, using held-out target labels that our method never sees. The closest reported reference on AMOS is joint CT-MRI training, which gave a +2.14% mDice gain on MRI (AMOS). On FeTA, the reported low-field result is Dice 0.86, HD95 1.69, ED 6.26, and the winning method's Euler characteristic difference was 20.9 (FeTA 2024). Those figures come from models trained with full multi-site data. They are ceilings for context, not like-for-like comparisons.

Four ablations isolate the key ingredient. The first replaces all tokens instead of only flagged ones, testing whether selection matters. The second uses a plain mean over the k donors instead of reconstruction weights, testing whether locally linear reconstruction matters. The third uses 1-NN replacement, the choice suggested by the covariate-shift theory of Portier et al.. The fourth drops abstention, so every flagged token is filled.

A final non-parametric competitor is nearest-neighbor label transfer. Each target token takes the class of its nearest source tokens, and the decoder is skipped, as in atlas-style segmentation.

Beating the field means three things. Selective reconstruction must exceed feature-statistics recalibration by at least 2 mDice points on small structures (adrenal glands, inferior vena cava, esophagus, duodenum) on AMOS cross-scanner MRI, across three seeds of source training. It must not lose on large organs. And it must come within 1 point of five-scan adapter fine-tuning.

Why not just…?

Why not just recompute normalization statistics on the target scans? This is the most serious threat because it is equally cheap and label-free. It applies one correction to every location, though. If FeTA's dominant driver is image quality, which varies within and between scans, a single global correction will under-fix corrupted regions and may disturb clean ones. Our diagnostic stage settles this directly. If flagged tokens are a small, error-enriched minority, local repair has room to win. If recalibration matches us on small structures, we will report that the simpler method suffices.

Why not just fine-tune with a few labeled target scans? In low-data regimes, frozen-backbone methods have outperformed full fine-tuning in Borrowing Treasures from Neighbors and INPUTADAPTER. In practice, the regulatory lock forbids weight changes. Still, low-rank fine-tuning with five scans may simply be better. We include it so that the cost of staying frozen is measured, not asserted.

Why not just harmonize the images with histogram matching or heavy augmentation? Histogram equalization did not stop dataset-specific clustering in the COVID-19 small-data study. Top FeTA entries already used heavy augmentation and still lost about 0.075 Dice to site. Both are run as baselines and stacked under our method, to test whether the gains add.

Why not just transfer labels from nearest source tokens? It is simpler and in the spirit of 1-NN label imputation. It discards the decoder's spatial reasoning, however, and likely fragments thin structures, which the topology metric will expose.

Proposed Experiments and Decisive Checks

On AMOS we use two settings. The main one trains on MRI from a subset of scanner models and tests on held-out models. The stress test transfers CT to MRI, where we predict that most tokens are flagged as unpredictable and the method mostly abstains. On FeTA we train on KISPI and test on Vienna and on the 0.55T King's College cohort. We report per-structure Dice, HD95, and Euler characteristic difference, because FeTA showed that overlap metrics can plateau while topology errors persist.

Three results would be decisive. First, the error-detection AUROC of the missingness mask must exceed 0.75 under shift. Second, selective locally linear reconstruction must beat both its replace-everything and plain-mean variants. Without that, the imputation framing adds nothing over generic retrieval. Third, the method must not erase pathology. On FeTA pathological cases, we measure whether repaired tokens pull atypical anatomy toward the healthy source manifold, using biometry error against the reported 7.72% MAPE of the best FeTA biometry model. If imputation improves Dice on typical anatomy but worsens pathological biometry, the method is unsafe as proposed. We would then restrict it to abstention-only use, where the missingness map serves as a quality-control flag rather than a repair.