Pre-Declared Sparse Masks for Test-Time Adaptation of Medical Segmenters

A segmentation model trained at one hospital often degrades sharply at the next, because a different scanner, field strength or reconstruction setting changes how the same anatomy looks. Test-time adaptation (TTA) addresses this by updating the model on unlabeled images as they arrive at the new site. TTA means adjusting the model during deployment using only the incoming images, without any labels. The difficulty is that these label-free updates are noisy. They can rewrite the parts of the network that encode anatomy, which damages exactly the cases that matter most, such as tumors and malformations.

We propose to decide, before deployment and using only source data, a very small fixed set of weights (about 0.1–0.5%) that respond to acquisition changes. All test-time updates would be confined to that set. The set is found by measuring which weights become more important to the loss when labeled source images are passed through physically plausible acquisition corruptions: bias fields, k-space spikes, motion ghosting and Fourier amplitude swaps. The bet is that scanner-driven shift concentrates in a stable subset of weights. If so, a mask computed from simulated shifts should transfer to real cross-site shifts, reduce the cases where adaptation makes things worse, avoid the memory cost of estimating importance online, and give regulators a pre-declared, auditable region of the model that is allowed to change.

Core Claim

The claim has two parts, and both are testable. The first concerns which weights to update. Existing parameter-selective TTA protects the weights most important to the source task and pushes updates into the unimportant ones. We argue the better choice for acquisition shift is the opposite rule: update the weights whose importance rises under acquisition corruption, since those are the ones mediating the shift. The second part concerns where the selection comes from. It can be computed once, offline, from simulated shifts on labeled source data, with no target data and no online bookkeeping. If either part fails, the experiments below will show it. A random or inverted mask that matches ours would refute the first part. A mask derived from simulation that does far worse than one derived from real target data would refute the second.

Relation to Prior Work

The proposal reuses a measurement idea from the sparse-adaptation literature. Training Neural Networks with Fixed Sparse Masks approximates the Fisher information, a per-weight measure of how sensitive the loss is to that weight, from as few as 32 samples. Updating the top 0.5% of weights by this score matched dense fine-tuning on GLUE (82.6 vs 82.5). The same paper constrains our design: performance broke down sharply at 0.02%, so we sweep sparsity rather than assume ultra-sparse masks work. Language Models are Super Mario and Localize-and-Stitch show that 90–99% of fine-tuning updates can be dropped. They motivate the premise that useful adaptation lives in a small subset, but they study supervised task updates rather than unsupervised shift correction.

Two papers sit closest to our idea. Personalized Federated Learning with Residual Fisher Information for Medical Image Segmentation computes a "residual" Fisher score: the gap in weight sensitivity between local data and cross-site data synthesized with Fourier amplitude swaps. It keeps the top domain-sensitive weights local. It reaches 85.42% Dice on six-site prostate MRI, about 1% over FedBN, and argues that layer-level choices like normalization-only tuning are too blunt. That paper motivates us and supplies our scoring rule. It differs from our setting in that it is supervised, federated and multi-round. We take its residual idea into source-free, label-free deployment and test whether the selected weights transfer across shift types.

Parameter-Selective Continual Test-Time Adaptation (PSMT) does select weights at test time. It does so in the opposite direction from us, regularizing Fisher-critical weights and updating the rest. It computes Fisher information online, which raised memory from about 1.2GB to about 5.3GB, and it was evaluated only on CIFAR-C and ImageNet-C classification. We treat PSMT as the direct conceptual competitor to beat.

On the medical side, the strongest TTA methods change the training signal, not where it is applied. Gradient Alignment Improves Test-Time Adaptation for Medical Image Segmentation (GraTa) aligns the gradients of the entropy and consistency losses and scales the learning rate by their agreement. Progressive Test Time Energy Adaptation for Medical Image Segmentation (PTTEA) minimizes a learned shape energy. Both update the network broadly. Our mask works alongside these objectives rather than replacing them, so we run GraTa's loss inside our mask. A Large Scale Benchmark for Test Time Adaptation Methods in Medical Image Segmentation (MedSeg-TTA) finds no method wins across modalities and reports negative transfer from shape priors on pathological anatomy. Our negative-transfer metric is designed around that finding.

Finally, Adapt the Data, Not the Model argues that regulatory lock forbids changing a certified predictor's weights, and adapts inputs instead (with target labels, on time series). Our proposal is a middle position between that view and unrestricted TTA: weights may change, but only within a region declared and audited before deployment.

Research Program

The core object is a per-weight acquisition-sensitivity score computed on labeled source data. Let $\theta_i$ be a weight, $L$ the segmentation loss on source image $x$ with label $y$, and $t$ a random acquisition corruption drawn from a family $\mathcal{T}$. The score is

$$s_i = \mathbb{E}_{x,t}\left[\left(\frac{\partial L(f(t(x)),y)}{\partial \theta_i}\right)^2\right] - \mathbb{E}_{x}\left[\left(\frac{\partial L(f(x),y)}{\partial \theta_i}\right)^2\right],$$

which is the diagonal Fisher information under corruption minus the clean diagonal Fisher. Weights with large positive $s_i$ matter more once the image is corrupted. We keep the top $k$ of them as a fixed binary mask $m$, with $k$ swept from 0.02% to 2%.

The corruption family $\mathcal{T}$ is built from the TorchIO-style artifacts used in Improved post-hoc probability calibration for out-of-domain MRI segmentation: bias field, motion, ghosting and k-space spikes. We add the Fourier amplitude swaps of the ResFIM paper, plus modality-specific variants: gamma and blur for ultrasound and X-ray, and stain jitter for pathology.

At deployment, any TTA objective is applied with gradients multiplied by $m$, and the other 99%+ of weights stay frozen. The objectives we test are entropy minimization, GraTa, and a mean-teacher variant. Because the mask is fixed, memory cost equals that of ordinary TTA restricted to $k$ weights. A second backbone family checks that the effect is not specific to small UNets: a DinoV2 encoder with Rein-LoRA, following Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation?. There, the mask is computed over the adapter and decoder weights.

The analysis component asks where the masks land. We check whether they concentrate in normalization layers, early convolutions or decoder heads. We measure how much masks computed from different corruption subsets overlap, and how much they overlap with an oracle mask computed from labeled real target data using the same residual score.

Baselines

The naive reference is the source model with no adaptation, run on every target domain under the same preprocessing. Every other number is reported as a change relative to it, together with the fraction of target cases it makes worse. GraTa reports a 5.87% mean Dice improvement over no adaptation across five optic disc/cup domains, and +12.77% on Domain D (GraTa). That is the bar on that benchmark.

The strongest existing method is GraTa itself, updating the parameters it normally updates. It is the most recent medical TTA method in the file with a cross-domain reported gain, and its improvement comes from a better objective, which our mask is orthogonal to. We run it with its published optimizer settings on the same source checkpoint and target streams. Our variant uses an identical loss, step count and batch size, and differs only by the mask. Beating it means a higher mean Dice and a lower HD95 (the 95th-percentile boundary distance, which catches errors Dice misses), each outside three-seed variability, without a higher negative-transfer rate.

On cardiac MRI we also run PTTEA, which reports about a 9% Dice gain over the second-best TTA method on ACDC→LVQuant. The comparison will reveal whether shape priors or sparse masking help more under pathology.

PSMT, ported to segmentation, is the direct test of our selection rule. It uses the same unlabeled stream and online Fisher estimation, and we report its memory alongside its accuracy.

Tent-style normalization-only tuning is the minimal parameter-subset competitor. We run it both at its natural size and with its parameter count matched to our $k$.

The input-level alternative is SFDA-FSM frequency alignment, which MedSeg-TTA reports raised ultrasound Dice from 0.4811 to 0.8333 on TN3K→DDTI. MedSeg-TTA gives no numbers on Lacuna for the other six modalities, so we reproduce SFDA-FSM ourselves there.

Three ablations isolate the key ingredient. The first uses a random mask of equal size. The second uses an "inverted" mask containing the top-$k$ weights by clean source Fisher, which is the weight set PSMT protects. The third uses an oracle mask computed from labeled target data, as an upper bound on what selection can achieve. All methods share a fixed hyperparameter search budget per method, tuned on one held-out source-side corrupted split rather than on target data.

Why not just…?

Why not just tune the normalization layers? This is the most serious threat. Intensity-driven scanner shift is exactly what normalization statistics absorb, and Tent-style tuning is cheap. The evidence against it as a complete answer is indirect. ResFIM found that domain sensitivity is spread across individual weights within layers and beat FedBN on prostate MRI (85.42% vs about 84.45%), but that result was supervised. GraTa reports that it beats TENT on optic disc/cup. Our analysis settles the question directly. If the residual-Fisher masks concentrate almost entirely in normalization parameters, and normalization-only tuning at matched size performs as well, then the idea reduces to a principled justification for normalization tuning, and we would report it that way. If the masks place substantial mass in convolution and attention weights and outperform normalization tuning on the structural shifts (CT→MRI spine, BraTS-GLI→SSA), the claim stands.

Why not just apply a random sparse mask, or PSMT's rule? Restricting how much of the network can change may be the whole benefit, regardless of which weights are chosen. The random-mask and inverted-mask ablations at equal $k$ test exactly this. Our claim requires that the residual mask beat both, especially on negative-transfer rate. A tie with random masking would mean the selection score adds nothing.

Why not just align the inputs and leave the model untouched? Frequency alignment is simple and clinically attractive, and it is excellent on ultrasound. However, MedSeg-TTA concludes there is no silver bullet, and that stylistic alignment fails on structural and cross-modality shifts. The spine CT→MRI shift collapses a UNet to 1.22 Dice (Cekmeceli et al.), a failure unlikely to be fixed by amplitude matching. We expect input alignment to win or tie on ultrasound and dermoscopy and to lose on MRI and cross-modality shifts. We also test combining the two methods, since they are complementary.

Why not just fine-tune everything at test time? Without labels, full updates amplify pseudo-label noise and allow anatomy-encoding weights to drift. The results that sparse adaptation matches full tuning (FISH, SparseAdapter) suggest little expressiveness is lost by restricting updates. Full-network entropy TTA is included as the $k=100\%$ endpoint of our sparsity sweep.

Proposed Experiments

The main testbed is the seven MedSeg-TTA shifts, covering MRI, CT, retinal imaging, ultrasound, pathology, dermoscopy and chest X-ray. It is supplemented by the five-domain optic disc/cup benchmark used by GraTa and by the cardiac shifts used by PTTEA (ACDC→LVQuant, M&M). The DinoV2+Rein-LoRA backbone is evaluated on the VFM spine and prostate shifts. AMOS unseen-scanner MRI, where drops exceed 20% mDice, serves as a stress test.

We report Dice, HD95, expected calibration error (the mismatch between confidence and accuracy), peak GPU memory, and the negative-transfer rate: the fraction of target volumes whose Dice falls below no-adaptation. We also report Dice separately on pathological cases (BraTS-SSA tumors, LiTS-derived lesions).

Four checks decide the outcome. The first is whether the simulation-derived mask recovers most of the oracle mask's gain. We count it as a success if it recovers at least two-thirds, and we also report the overlap between the two masks. The second is whether it beats both random and inverted masks at matched sparsity on at least five of the seven MedSeg-TTA shifts. The third is whether GraTa-inside-mask improves on GraTa's 5.87% mean gain on optic disc/cup while lowering the negative-transfer rate. The fourth is whether performance holds over a long continual stream that cycles through target sites, where unrestricted TTA is expected to drift. A further sweep traces the sparsity curve, to find whether the abrupt breakdown FISH reported near 0.02% also appears for medical segmentation. Failure on the first check would mean simulated acquisition corruptions do not identify the weights real scanner shift acts on, which is itself a useful negative result for anyone relying on augmentation as a proxy for site shift.