A medical image classifier that works well at the hospital where it was built often loses much of its accuracy at a new hospital, because scanners, staining, cameras and patients differ. The usual fix is to retrain the network on local data, and that fix causes two problems. Retraining damages performance at the original site. A study of a pneumothorax detector across three hospitals found that local fine-tuning recovered up to 0.23 area under the ROC curve (AUC) at the new site but cost the original site 0.176 AUC on average Regulating AI Adaptation: An Analysis of AI Medical Device Updates. Retraining also breaks the regulatory status of a cleared device, which helps explain why only 6 of 416 FDA-cleared AI devices were retrained within two years of approval (same paper). We propose a different arrangement. The feature extractor is trained once and never touched again. Each site gets its own classification head, built by averaging the features of a few dozen labeled local cases per class. No gradient step is taken at deployment.
Whether this works depends on what the frozen features look like, and that is the scientific question. We hypothesize that the right training objective for a locked backbone pushes class centers apart but deliberately does not squeeze each class into a tight cluster. Squeezing discards feature directions that a per-site head would need to re-anchor shifted classes. Separation keeps shifted classes from landing on top of each other. If the hypothesis is right, a single training run plus a closed-form per-site calibration should recover most of the cross-site loss, leave the original site exactly as it was, and outperform both standard training and the compactness-heavy centroid losses currently used for medical domain shift.
Core Claim
The claim is narrow and falsifiable. Under scanner, camera and site shift, the accuracy that a frozen backbone can recover through head-only recalibration is governed by inter-class separation, not intra-class compactness. An objective that maximizes the angle between class centers while putting a floor under within-class spread therefore yields backbones whose per-site nearest-centroid heads beat those built on cross-entropy or centroid-contrast backbones at equal labeled budget.
Let $f_\theta$ be the feature extractor producing unit-norm features $z$. Let $\mu_c$ be an exponential-moving-average center of class $c$ over $C$ classes, and let $s_d$ be the batch standard deviation of feature dimension $d$ among samples of the same class. The training loss is
$$ \mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \lambda \sum_{c \neq c'} \max\!\big(0,\ \cos(\mu_c,\mu_{c'}) - \cos\theta^\ast\big) + \beta \sum_{d} \max\!\big(0,\ \sigma_0 - s_d\big), $$
Here $\theta^\ast$ is the target separation angle, set to the equiangular value $\arccos(-1/(C-1))$ (the angle at which all class pairs are maximally and equally spread), and $\sigma_0$ is a floor on within-class spread. The second term repels centers only. There is no term that pulls samples toward their own center. The third term prevents the within-class collapse that cross-entropy with weight decay drives toward.
At a new site, the deployed classifier assigns $z$ to the class whose locally re-estimated center $\hat\mu_c^{\text{site}}$ has the highest cosine similarity. The centers are computed from $k$ labeled cases per class. The original site keeps its own centers, so its predictions are bit-identical before and after any other site is added.
Where this sits in prior work
The closest work is Contrastive Centroid Supervision Alleviates Domain Shift in Medical Image Classification, referred to here as FCCL. FCCL keeps moving-average class centroids and adds a contrastive loss that pulls each sample toward its own centroid and away from others. With no target data it improves EyePACS cross-camera performance from 0.734 to 0.767. FCCL motivates us, because it shows that centroid geometry matters under medical shift. It differs from our proposal in two ways. First, it bets on compactness. Second, it deploys one shared end-to-end classifier and never asks what a frozen representation allows after shipment.
Two papers constrain FCCL's bet on compactness. Learning with Fantasy shows that aggressive within-class compression shrinks the occupied feature space. It finds that widening gaps between classes matters more than tightening them: removing its reserved "virtual class" component costs nearly 9% on miniImageNet. On the Robustness of Neural Collapse and the Neural Collapse of Robustness shows that fully collapsed class geometry is brittle under input perturbation, with points jumping between class vertices. Both findings come from settings other than clinical site shift. Testing whether they transfer is part of the contribution.
The decision to adapt only the head comes from Decoupling Representation and Classifier for Long-Tailed Recognition. That paper finds that class imbalance lives mostly in the classifier, so a frozen backbone with a retrained or rescaled head beats joint rebalancing tricks; LWS reaches 49.9% on ImageNet-LT. We borrow the decoupling but change what it is applied to. That paper decouples to fix label imbalance on one distribution. We decouple across sites, and we change the first stage so the backbone is built to be recalibrated.
Unsupervised Continual Domain Shift Learning with Multi-Prototype Modeling, referred to here as MPM, argues that optimal decision boundaries differ across domains, so forcing domain-invariant features is self-defeating. That argument supports per-site centers over the adversarial invariance approach. MPM, however, achieves this with graph propagation modules and a replay buffer, evaluated on natural-image benchmarks. We get per-domain prototypes with no extra modules at all.
Medical adaptation pipelines such as Labeled-to-Unlabeled Distribution Alignment add teacher-student loops and dual prototype sets. That added operational weight is what the multi-stage training literature warns about Rethinking and Improving Multi-task Learning for End-to-end Speech Translation. In our design, "multi-stage" shrinks to one training run plus an averaging step that has no hyperparameters beyond $k$.
Research Program
The work has three parts.
The first part builds and trains the separation-first backbone. We use ResNet-50 for the classification tasks, matching FCCL's setting. We sweep $\lambda$, $\beta$ and $\sigma_0$ on source-site validation data only.
The second part is a controlled study of geometry versus recoverability. We train four backbones with identical architecture, data, augmentation and epochs. The first uses plain cross-entropy. The second adds centroid pull plus push, which is the FCCL objective. The third adds center repulsion only. The fourth is the full proposal, repulsion plus the variance floor. For each backbone we measure three things on source data: the within-class collapse statistic NC1, the minimum pairwise angle between class centers, and the effective rank of class-conditional features. At each target site we measure how often target samples sit nearer to the wrong re-estimated center. This error separates two failure modes. One is a shift that a head can fix, such as a class center moving as a whole. The other is a shift that no head can fix, where classes mix. The quantity of interest is the curve of target performance against $k$, with $k$ ranging from 5 to 100 labeled cases per class, under nearest-centroid recalibration.
The third part is an unlabeled variant for sites that cannot provide labels. Centers start at the source values and are refined by a few iterations of soft k-means on target features. This tests whether separation also makes label-free recalibration safe. The concern comes from the research file's warning that biased centroids can cascade into errors under severe shift.
If the classification results hold, we extend to dense prediction. On AMOS, whose authors report a drop of more than 20% when moving to MRI from unseen scanners, voxel features from a frozen 3D-UNet encoder-decoder get per-scanner organ centers. The aim is to test whether small, low-contrast organs, the low-margin structures highlighted by Learnability-Driven Knowledge Assimilation, benefit most from separation.
Baselines
The naive reference is the source-trained cross-entropy model deployed unchanged. FCCL reports 0.799 for this model on the source site and 0.734 on the target camera domain of EyePACS FCCL. The three-hospital pneumothorax study reports external AUC drops of up to 0.18 Regulating AI Adaptation. Any method must first beat this reference at the target site.
The strongest existing method for shift without target data is FCCL. Its reported EyePACS target score of 0.767 is the number to beat in the zero-label column. We will rerun FCCL with our backbone, augmentation and an equal tuning budget, so the comparison does not rest on a reported figure alone. Our method only counts as beating FCCL if both of the following hold: (a) our $k=0$ model is not worse than FCCL, and (b) with $k$ labeled local cases, our recalibrated head exceeds FCCL's backbone given the same per-site centroid head.
The strongest adaptive competitor is full fine-tuning on local data. The pneumothorax study shows it recovers up to 0.23 AUC locally at an average cost of 0.176 AUC at the host site. We run it on the same $k$ cases with early stopping on a held-out local split. Our method beats it if it reaches comparable local AUC with zero host-site loss.
The decoupled-classifier baseline uses a cross-entropy backbone with a per-site linear head retrained on the $k$ cases, following Decoupling Representation and Classifier. We found no reported result on Lacuna for decoupled per-site heads on medical site shift, so we report our own numbers.
The ablation that isolates the key ingredient is the four-backbone comparison described in the second part of the research program, with the head fixed to nearest centroid. If repulsion-only and repulsion-plus-floor do not beat pull-plus-push at matched $k$, the separation-over-compactness claim fails.
All methods share data splits, ResNet-50 or 3D-UNet capacity, epoch budget, and a fixed number of hyperparameter trials selected on source validation data only.
Why not just…?
Why not just retrain a linear head per site on an ordinary cross-entropy backbone? This is the most serious threat. It is cheap, it is also frozen-backbone, and the decoupling literature says heads carry much of the problem. We do not argue it away. It is baseline three, and the four-backbone ablation measures exactly the gap between it and our method. If cross-entropy features recalibrate as well as separation-first features at every $k$, we will report that the training objective is irrelevant for locked deployment. That would still be a useful negative result for regulators considering frozen-backbone change plans. The reason to expect a gap is that cross-entropy with weight decay drives within-class collapse, and the neural-collapse robustness study shows collapsed geometry mixes classes under perturbation, which no head can undo.
Why not just use FCCL, which already uses class centroids? FCCL tightens clusters and ships one shared classifier. The tension we test is whether that tightening throws away the directions a site-specific center needs. Learning with Fantasy gives reason to suspect it does. The pull-plus-push arm of our ablation answers the question directly.
Why not just fine-tune with LoRA adapters, or keep a separate fine-tuned copy per site? Per-site copies do avoid forgetting at the host site. They do so by multiplying the number of validated models and gradient-trained artifacts, which is the regulatory burden that keeps devices locked. They also risk overfitting when $k$ is small. We include per-site full fine-tuning as a baseline. If it dominates at small $k$, that weakens our case to an operational argument only, and we will say so.
Why not just augment harder, for example with Bézier intensity curves or edge inputs? Train Once, Deploy Anywhere shows that such augmentation helps single-source generalization, reaching 64.78% Dice on BraTS'19. Augmentation and separation are not exclusive, so we apply the same augmentation to every arm. The question we ask is about what remains once augmentation has done its share.
Proposed Experiments and Decisive Checks
The classification experiments use three benchmarks. EyePACS provides cross-camera shift, HAM10000 provides male-to-female patient shift, and WSSS4LUAD provides cross-hospital shift from GDPH to TCGA. These follow the FCCL protocol, though the file reports no numbers for the last two. The fourth benchmark is a three-hospital pneumothorax setting modeled on the NIH, Stanford and BIDMC study. In that setting we train at one site and add the other two sequentially. We report target AUC, host-site AUC after each addition, and the recovery curve over $k$. Host-site AUC must be unchanged by construction; any drift would indicate a pipeline bug.
The segmentation extension uses AMOS across its eight scanners and the six-site Prostate collection. It reports per-organ Dice before and after per-scanner center recalibration.
Three outcomes would decide the claim. First, at $k$ between 20 and 50, the separation-first backbone should close a clearly larger fraction of the source-to-target gap than both the cross-entropy and the FCCL backbones under the identical head. Second, on source data, the minimum angle between class centers should predict recoverability across backbones better than NC1 does. Third, on HAM10000, where FCCL's authors concede that demographic shift is not purely geometric, gains should shrink. This marks the boundary of the method rather than contradicting it. If the first check fails, the hypothesis is wrong. If only the second fails, separation helps for reasons other than the mechanism we propose, and the paper must say so.
References
- Regulating AI Adaptation: An Analysis of AI Medical Device Updates
- Contrastive Centroid Supervision Alleviates Domain Shift in Medical Image Classification
- Learning with Fantasy: Semantic-Aware Virtual Contrastive Constraint for Few-Shot Class-Incremental Learning
- On the Robustness of Neural Collapse and the Neural Collapse of Robustness
- Decoupling Representation and Classifier for Long-Tailed Recognition
- Unsupervised Continual Domain Shift Learning with Multi-Prototype Modeling
- Labeled-to-unlabeled distribution alignment for partially-supervised multi-organ medical image segmentation
- Rethinking and Improving Multi-task Learning for End-to-end Speech Translation
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Learnability-Driven Knowledge Assimilation for Class-Incremental Semantic Segmentation
- Train Once, Deploy Anywhere: Edge-Guided Single-source Domain Generalization for Medical Image Segmentation