Medical imaging models are usually trained on a few dozen labeled scans from one hospital and then deployed at another hospital, with a different scanner, staining protocol, or patient mix. They fail there in two ways at once. With so few labels they memorize the training set, and they are also brittle to the change of site. Hospitals often have plenty of unlabeled scans from the deployment site. The standard way to use them is pseudo-labeling: let the model label the target scans and train on its own guesses. That works only when the model's guesses are already decent, and with tiny labeled sets under site shift they often are not.
We propose a different use of the unlabeled target scans. Use them to measure, without any labels or guesses, how fragile the model's predictions at the deployment site are to small changes in its weights, and train the model to make that fragility small. Concretely, we find the weight direction along which target-site predictions change fastest. We then take the supervised step on the few source labels at the weights perturbed along that direction, and we also penalize the change itself. The core claim is that flatness should be measured where the model will be used, not where its labels came from. If that is right, a regularizer built only from the target site's unlabeled scans should help most exactly when labels are scarcest, which is the regime where pseudo-labeling and label-driven flatness methods degrade.
Core Claim and Mechanism
Several existing methods already use unlabeled data to steer training toward "flat" solutions, meaning weights around which the model's behaviour changes little. They choose the direction of weight perturbation in different ways, and that choice is what separates them from us.
FlatMatch finds the worst-case weight perturbation from the labeled loss and penalizes prediction disagreement on unlabeled data. Its authors report that with about one label per class, labeled gradients become noisy and the perturbations become useless. Under a few-dozen-label clinical regime, this is a warning that the direction should not come from labels.
Inconsistency-Aware Minimization (IAM) supplies the tool we reuse. It defines a label-free "local inconsistency": the worst-case change, measured by KL divergence, in the model's output probabilities under a weight perturbation of radius $\rho$. It computes this by power iteration on the Fisher information matrix. IAM was tested only with unlabeled data drawn from the same distribution as the labels, on CIFAR. There it lowered FixMatch's CIFAR-10 error with 250 labels from 6.26% to 5.30%.
DiM is the closest medical work. It applies sharpness-aware entropy minimization only to confident unlabeled samples and adds $f$-divergence logit alignment across sites. Our proposal differs from DiM in three ways. It filters nothing by confidence, which matters because confidence is least trustworthy under shift. It minimizes no entropy, so it does not push the model toward overconfidence. It needs no multi-site alignment, so it applies with a single labeled source site.
The objective is the following. Let $\theta$ be the network weights, $D_S$ the small labeled source set, and $U_T$ the unlabeled target-site pool. Let $p_\theta(x)$ be the predicted class distribution, which for segmentation is per pixel, averaged over pixels. The target inconsistency is
$$S_\rho^T(\theta)=\max_{\|\delta\|\le\rho}\ \mathbb{E}_{x\sim U_T}\,\mathrm{KL}\big(p_\theta(x)\,\|\,p_{\theta+\delta}(x)\big)\approx \tfrac{1}{2}\rho^2\,\lambda_{\max}(F_T),$$
where $F_T$ is the Fisher information of the model's own predictive distribution on $U_T$. The approximation is the second-order expansion of KL, and $\lambda_{\max}$ is its largest eigenvalue, whose eigenvector $v_T$ gives the worst direction. Because $F_T$ samples outputs from $p_\theta$ itself, no label or committed pseudo-label is involved.
Training minimizes
$$\mathcal{L}(\theta)=\mathcal{L}_{\text{sup}}(\theta+\rho v_T;\,D_S)+\lambda\, S_\rho^T(\theta).$$
The first term reverses FlatMatch's construction. FlatMatch perturbs along a labeled-sharp direction and penalizes on unlabeled data. We perturb along the target-sharp direction and require the source labels to still be fit there.
Research Program
We will build the method as an add-on that works with any segmentation or classification network and any semi-supervised base loss. It has three pieces: a target-only Fisher power iteration, the perturbed supervised step, and the direct inconsistency penalty.
The analysis has three threads. The first is whether the target-site Fisher top eigenvalue at the end of training predicts target error across methods and seeds better than the source-site eigenvalue does. This is the mechanism check: if target flatness does not track target error, the premise fails.
The second thread is how gains scale with the labeled-set size. We will sweep the labeled source set from very small, a handful of slides or images, up to moderate sizes. We expect the advantage over label-driven methods to grow as labels shrink.
The third thread is a degenerate solution we must rule out. A model can make target predictions insensitive to its weights by making them nearly constant on target scans. We will track the target prediction marginal and the per-image entropy. If collapse appears, we will add a marginal-diversity term and report its effect separately.
Compute is a real cost. IAM roughly doubles per-epoch time. All comparisons will therefore be matched on GPU-hours as well as on epochs.
Proposed Experiments
We use three benchmark families from the file, chosen to span shift severity and task type.
The first is Camelyon17-WILDS with its curated unlabeled target data from Extending the WILDS Benchmark for Unsupervised Adaptation. We subsample labeled source patches at the slide level to create small-sample regimes. We will also report the full-label setting so results connect to published numbers: ERM reaches 70.8% out-of-distribution accuracy and Noisy Student reaches 86.7% in that paper.
The second is the multi-site fundus and prostate MRI segmentation settings used by DiM and Dual-supervised Asymmetric Co-training (DAC). Here DiM reports 87.28% Dice on fundus with 20 labels, compared with 85.85% for MiDSS, and 87.45% Dice on prostate MRI against a fully supervised 88.32%. DAC reports 76.35% optic-cup Dice at 10% labels.
The third is echocardiography, TMED-2 to Unity cross-hospital transfer, as in InterLUDE. No reported result was found on Lacuna for InterLUDE on TMED-2 to Unity, so we will run it ourselves.
As a stress test we will use AMOS with training on CT plus one MRI scanner and testing on held-out MRI scanners. The paper reports a drop of over 20% in that setting. We expect the regularizer to narrow, not close, that gap.
Metrics are accuracy for classification, and Dice with Hausdorff distance for segmentation. We also report expected calibration error, because entropy-minimizing competitors may trade calibration for accuracy and a weight-stability regularizer should not. Model selection uses only source validation data. Target-oracle selection is reported separately, so that no method benefits from peeking at target labels.
Baselines
The naive reference is empirical risk minimization (ERM), meaning plain supervised training, on the labeled source subset with standard augmentation, using the same backbone. A second reference, ERM with aggressive augmentation, addresses the WILDS finding that robust objectives failed to beat ERM on 8 of 10 datasets. A third, SAM on the source labels alone, separates flatness itself from where flatness is measured.
The strongest competitors split by benchmark. On Camelyon17 the strongest existing method in the file is Noisy Student self-training, which reports 86.7% out-of-distribution accuracy with full labels (Extending WILDS). On fundus and prostate the strongest is DiM, at 87.28% and 87.45% Dice. On both we also run FlatMatch and FixMatch with the same target unlabeled pool.
Fairness rules apply to every method. All use the same unlabeled pool, the same backbone, an equal hyperparameter trial budget of about twenty configurations each, and matched GPU-hours, with non-SAM methods given extra epochs to offset our extra passes. Beating these baselines means two things. On the reported full-label points we must not fall below their numbers by more than seed variance. In the small-label regime, at 20 or fewer labeled images, we must exceed both Noisy Student and DiM by a margin larger than three-seed variance. Adding our term on top of Noisy Student or FixMatch must also improve target error, since the method is meant to work alongside self-training.
Two ablations isolate the key ingredient. The first computes the Fisher direction from an unlabeled source pool instead of the target pool; this is IAM as published. The second replaces the weight-space perturbation with input-space adversarial smoothing on the target pool, using Virtual Adversarial Training. If the source-Fisher variant matches ours, the "measure flatness at the deployment site" claim is false. If VAT matches ours, the weight-space machinery is not worth its cost.
A frozen self-supervised backbone with a k-NN or linear head, in the style of ALAN, serves as the no-fine-tuning reference. ALAN reports 90.78% TMED-2 view accuracy and about 80% left-ventricle Dice.
Why not just…?
Why not just freeze a self-supervised backbone and fit a k-NN or linear probe? For classification this is a serious threat: ALAN reaches 90.78% on TMED-2 without updating the backbone. It is weaker for dense segmentation, where ALAN plateaus near 80% Dice. Frozen representations also still degrade under large scanner shift, as the over-20% drop on unseen MRI scanners in AMOS shows. We include the probe on every benchmark. If it beats us on echo classification, we will report that and narrow the claim to segmentation and severe-shift settings.
Why not just self-train with pseudo-labels, as Noisy Student does? On full-label Camelyon17 it lifts out-of-distribution accuracy from 70.8% to 86.7%, so it may well win there. Its teacher, however, is trained on the labels. When labels shrink to a few slides, the teacher's errors on the target site become the training signal. Our term has no such dependence. The labeled-set sweep decides the matter: we predict the gap favors our method at small label counts and closes at large ones. We also test the combination, in case the two are simply complementary.
Why not just use ERM with strong augmentation, or SAM? WILDS shows ERM is hard to beat, and DISAM shows that SAM's perturbation is biased toward whichever domains converge slowest. Neither ever looks at target-site scans. If source-only SAM matches target-Fisher regularization, the unlabeled pool adds nothing. That is a direct comparison in our table, not an assumption.
Why not just use DiM, which already does sharpness-aware training on unlabeled medical scans? DiM is the closest prior work, and we compare against its reported numbers directly. Its sharpness term acts only on confidently predicted target samples and pairs this with entropy minimization. Under strong shift, the confident subset is biased toward easy target images, and entropy minimization tends to hurt calibration. Expected calibration error, measured alongside Dice on fundus and prostate, will show whether this distinction matters in practice. If DiM matches us in both accuracy and calibration, our contribution reduces to single-source applicability.
Decisive Checks
The proposal stands or falls on four results.
First, across all trained models, the target-site Fisher top eigenvalue must rank-correlate with target error more strongly than the source-site eigenvalue does.
Second, the source-Fisher ablation must underperform the target-Fisher version on target error, with the gap widening at smaller label counts.
Third, our method must beat FlatMatch, Noisy Student and DiM in the 20-or-fewer-label regime on at least two of the three benchmark families under matched compute and source-only model selection.
Fourth, target predictions must not collapse toward a constant class marginal.
If the first or second check fails, the idea is wrong, whatever the headline accuracy shows.
References
- FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning
- Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Data
- DiM: f-Divergence Minimization Guided Sharpness-Aware Optimization for Semi-supervised Medical Image Segmentation
- Extending the WILDS Benchmark for Unsupervised Adaptation
- Dual-supervised Asymmetric Co-training for Semi-supervised Medical Domain Generalization
- InterLUDE: Interactions between Labeled and Unlabeled Data to Enhance Semi-Supervised Learning
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
- Unlocking the Heart Using Adaptive Locked Agnostic Networks
- Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts