Adapting Locked Medical Segmenters to New Scanners from a One-Shot Private Feature Summary

A medical image segmentation model that works at the hospital where it was built often fails at the next hospital, because a different scanner produces images with different contrast, noise and texture. Two constraints make the obvious fixes hard. Regulators treat a certified model as locked, so its weights cannot be retrained. The developer's training scans are patient data, so they cannot be shipped to the new site to guide adaptation. We propose to keep the model frozen and train a small image-to-image "adapter" at the new hospital. The adapter rewrites incoming scans so that the frozen model's internal features look like the features it produced on its own training data. The developer describes those features once, by releasing a noisy statistical summary that is differentially private. Differential privacy is a formal guarantee that limits how much any single patient can change what is released.

The summary is computed and noised a single time, and everything afterwards counts as post-processing. The adapter therefore trains against a fixed target. It does not chase a noisy discriminator, the moving opponent network in adversarial training that becomes unstable under privacy noise. That is the main bet of this note. The known way private adversarial training fails is to put noise on a target that keeps moving. The known way medical models fail is scanner shift on a model nobody is allowed to touch. One fixed private summary in the model's own feature space addresses both problems at the same time.

Core Claim

A class-conditional, multi-layer kernel summary of a frozen 3D segmenter's internal features can be released once under patient-level differential privacy. An input adapter trained only against that summary and unlabeled target scans will recover much of the cross-scanner accuracy loss. It will do so while the certified model stays unchanged, with no raw source data leaving the developer, and with stable optimization at budgets where privately trained adversarial adapters diverge. The new part is not private distribution matching in general. It is three specific choices made together: the frozen predictor's own features serve as the matching space, the summary is conditioned on organ classes so small structures are not swamped by the liver, and the adaptation happens in input space so the source model has no backward transfer by construction.

Where This Sits in Prior Work

Our design starts from the evidence that adversarial training and differential privacy do not mix. The standard private training method is DP-SGD, stochastic gradient descent with per-sample gradient clipping and added Gaussian noise. Private Post-GAN Boosting shows that putting this noise into discriminator updates erases the directional signal the generator relies on, so the final checkpoints are often worse than earlier ones. A Unified View of Differentially Private Deep Generative Modeling attributes the failure to privatizing a non-stationary component. DPAF describes a batch-size conflict: privacy wants large batches, while discriminators destabilize on them. Together these papers rule out the most natural design, a domain-adversarial adapter with a privately trained domain discriminator. We keep that design only as a baseline.

Our mechanism reuses the "privatize the target once" idea. DP-MERF releases a noisy random-Fourier-feature estimate of the data's kernel mean embedding, a fixed-length vector summarizing the distribution. A generator then fits that vector with no further privacy cost. The slicing mechanism adds noise to random low-dimensional projections, and even reports an MNIST-to-USPS adaptation experiment. Both were tested on digits, tabular data or 32×32 images, and both match raw data rather than a downstream model's features. DP-Sinkhorn stabilizes private generation with optimal transport but uses pixel-space L1/L2 costs, which it acknowledges miss semantic structure. We move the matching into the feature space of the clinical model itself, the space whose shift actually causes the errors.

Three closer papers do private domain adaptation, and each differs from us in a specific way. Differentially Private Optimal Transport privatizes projected source features and computes a transport map for classification on Office-Caltech, Office-Home and VisDA. PRIMA privately aligns covariance matrices between domains. Differentially Private Domain Adaptation with Theoretical Guarantees reweights samples but needs target labels. All three adapt or retrain the predictor, all work on whole-image classification, and none was tested on medical volumes. PRIMA in particular matches only second-order statistics, which is a weakness we test directly.

On the medical side, Adapt the Data, Not the Model supplies the input-adaptation template for frozen predictors. It is time-series only, has no privacy guarantee, and its best variant stores raw source exemplars. Our released summary replaces that memory bank. Regulating AI Adaptation supplies the motivation in three findings: only 6 of 416 FDA-cleared devices were retrained, site shift costs up to 0.18 AUC, and local fine-tuning damages the original site by 0.176 AUC on average. AMOS supplies the benchmark and the gap: more than a 20% mDice drop on unseen MRI scanners, with small structures such as the adrenal glands and duodenum hit hardest. Differentially private federated deep learning for multi-site medical image segmentation adds a constraint on design. A deeper ResNet-18 U-Net failed to converge under DP-SGD. This is one more reason to keep every iterative optimization outside the privacy mechanism.

Research Program

On the developer side, the frozen segmenter $f$ is run over each source patient's volume. At a few chosen encoder and decoder layers $l$, each voxel's feature vector is mapped through a fixed random Fourier feature map $\phi_l$. The voxel is assigned to its ground-truth organ class $c$, which is available at the source. For patient $i$, the per-class mean $m_{i,l,c}$ is computed, and the concatenation over all $(l,c)$ is clipped to norm $C$. The released summary is

$$\hat\mu = \frac{1}{N}\sum_{i=1}^{N} \mathrm{clip}_C\big([m_{i,l,c}]_{l,c}\big) + \mathcal{N}\!\left(0, \tfrac{\sigma^2 C^2}{N^2} I\right),$$

where $N$ is the number of source patients and $\sigma$ is set by the Gaussian mechanism for the chosen $(\epsilon,\delta)$. Because each patient's whole contribution is clipped as one unit, the guarantee is at the patient level rather than the voxel level. A noisy per-class voxel-count vector is released under the same budget split so that classes can be reweighted. Noise shrinks as $1/N$, which is why this suits large developer cohorts and why we expect a cliff for small cohorts. Differentially Private Optimal Transport needed $\epsilon=20$ on the small DSLR subset.

At the target hospital, an adapter $g_\theta$ is trained on unlabeled local scans. It is a lightweight 3D residual U-Net, initialized to the identity, with $\theta$ its weights. Its objective is

$$\min_\theta \; \sum_{l,c} w_c \big\| \hat\mu_{l,c} - \bar\phi_{l,c}\big(f(g_\theta(x))\big)\big\|^2 + \lambda\, \mathcal{R}(g_\theta(x), x).$$

Here $\bar\phi_{l,c}$ is the target-side mean embedding. Voxels are grouped by the frozen model's own predicted class, refreshed periodically. The weight $w_c$ boosts rare classes, and $\mathcal{R}$ is a fidelity term that limits how far the adapter may move intensities and edges, in the spirit of INPUTADAPTER's range and fidelity losses.

Training the adapter consumes no source privacy budget. It also keeps target data on site, because only $\hat\mu$ crosses the institutional boundary.

The main risk is that the adapter could learn a perturbation that fools the frozen model's features without restoring correct anatomy. Class-conditioning on the model's own predictions makes this risk worse, because it lets the model's errors feed back into the target statistics. We address the risk in two ways: with the fidelity term, and by always scoring against true target labels that the method never sees during training.

The analysis component asks when a fixed private summary beats per-step private training. We plot recovered mDice against $\epsilon$ and against $N$, and we track training curves across learning rates, which is the stability test DP-Sinkhorn used against GS-WGAN.

Baselines

The naive reference point is the frozen segmenter applied directly to the unseen scanner. On AMOS, a classic 3D-UNet reaches 89.04% mDice on CT, and AMOS reports drops of over 20% on unseen MRI scanners (AMOS). We reproduce this split ourselves so that every method sees identical held-out vendors.

The strongest privacy-respecting simple competitor is per-organ intensity histogram matching. The developer releases Gaussian-noised per-class intensity histograms under the same $(\epsilon,\delta)$. The target applies a monotone intensity remap using the frozen model's predicted masks. It is cheap, low-dimensional and therefore light on noise, and it directly targets the MRI-units problem AMOS highlights. To count as beating it, our method must exceed it by a margin larger than seed variance, which we expect to be about 1 mDice, overall and especially on the small organs.

The strongest existing private alignment method is a PRIMA-style release of noisy, PSD-corrected per-layer feature covariances (PRIMA). We pair it with our exact adapter, privacy split and tuning budget, so the only difference is second-order versus full kernel-embedding statistics. PRIMA's published results are on Office-Caltech10 and Amazon reviews. It reports a 1–2% accuracy gain over DPDA, and DPDA reports 91.2% on Office-Caltech against a non-private 93.0% (Differentially Private Optimal Transport). No reported result was found on Lacuna for any private adaptation method on AMOS or any 3D medical segmentation shift, so all private baselines there are run by us.

The adversarial baseline is the same adapter trained against a domain discriminator on frozen features. The discriminator is trained with DP-SGD on source features, which requires the developer to run training rather than release a summary. We count its privacy cost over all tuning runs, because each run spends budget, and we give it the same number of configurations. Beating it means higher mean mDice and, more importantly, lower variance across learning rates and seeds.

The upper reference is a non-private version of our method, using exact source embeddings. Alongside it we run an INPUTADAPTER-style adapter with a raw source exemplar bank (Adapt the Data, Not the Model). Together these define how much accuracy privacy costs. A separate, non-locked reference is full target fine-tuning with target labels. It measures the ceiling, and its backward transfer on the source site is reported against the 0.176 AUC figure from Regulating AI Adaptation.

Ablations isolate the key ingredients. The first runs the same objective with pixel-space embeddings instead of frozen-model features, which tests the DP-Sinkhorn critique. The second drops class-conditioning and uses one pooled embedding. The third uses a single layer instead of several. The fourth swaps random features for k-dimensional slices as in the slicing mechanism.

Why not just…?

Why not just do private histogram matching of intensities? This is the most serious threat. If AMOS MRI shift were mostly a monotone remap of intensities per tissue, a few noisy histograms would solve it at almost no privacy cost. Our method only earns its complexity if scanner shift also changes texture, noise and contrast relations that a per-voxel remap cannot undo. The head-to-head comparison at matched $\epsilon$ settles it. If histogram matching comes within about 1 mDice overall and on the adrenal glands and duodenum, we will report that the deep-feature summary is unnecessary for this shift and restrict our claim to the shifts where it is not, such as CT-to-MRI or the chest X-ray sites.

Why not just train a domain-adversarial adapter with a private discriminator? The adversarial generative literature (Private Post-GAN Boosting, DPAF) predicts divergence and sensitivity to hyperparameters. However, a domain discriminator is a simpler binary task than a GAN discriminator and may behave better. We therefore run it as a baseline rather than dismiss it, and we report learning-rate sweeps. If it matches us at equal total privacy spend, the stability argument fails for this setting.

Why not just generate private synthetic source scans with a diffusion model and adapt toward them? Private 3D latent diffusion reaches FID 26.77 at $\epsilon=10$ with public pre-training, but quality collapses at $\epsilon\le1$. It also requires a public pre-training cohort and much more compute. Our summary targets only the statistics the frozen model is sensitive to. We include one synthetic-data arm at $\epsilon=10$ and $\epsilon=1$ for cardiac MRI if compute allows. If synthetic data wins at $\epsilon=10$, we would position our method as the low-budget alternative.

Why not just fine-tune the model with DP-SGD at the new site? It is ruled out by the locked-model constraint, and the federated DP study shows deeper U-Nets may not converge under DP-SGD (multi-site DP segmentation). We still report it as a non-compliant reference.

Proposed Experiments

The primary testbed is AMOS. The frozen 3D-UNet is trained on CT plus MRI from a subset of vendors, and the remaining MRI vendors are held out as target sites. We sweep $\epsilon \in \{0.5, 1, 3, 10\}$ with $\delta$ below one over the cohort size, and we subsample the source cohort to vary $N$. Metrics are per-organ Dice and mDice, with small structures reported separately, plus an empirical membership-inference audit on the released summary.

The second testbed is pneumothorax detection across NIH, SHC and BID with a frozen DenseNet-121, using a 2D adapter. It tests whether the approach transfers to classification. There, the reported reference points are the 0.18 AUC shift drop and the 0.23 AUC recovered by local fine-tuning, the latter at a 0.176 AUC average cost to the original site (Regulating AI Adaptation).

Four checks are decisive. The first: at $\epsilon=1$, does our adapter close a substantial fraction of the unseen-scanner gap while beating private histogram matching and private covariance alignment beyond seed noise? The second: is its variance across learning rates markedly lower than the private adversarial adapter's? The third: does the pixel-space ablation lose most of the gain, showing that the frozen model's feature space is the essential ingredient? The fourth: does performance on the source site stay exactly unchanged, which is guaranteed because the model is untouched, while the target gain approaches the non-private exemplar-bank upper bound?

A failure on the first check against histogram matching would be the most informative negative result. It would mean that, for MRI vendor shift, private adaptation needs only a handful of intensity statistics.