When a medical imaging model moves to a new hospital, a common way to adapt it is self-training. The model labels the new hospital's unlabeled images itself, keeps the labels it is confident about, and retrains on them. Recent methods set a separate confidence cutoff for each class, so that hard or rare classes still contribute training labels. Those methods assume that low confidence means a class is hard. That assumption breaks when the source model has learned a shortcut, meaning a non-biological cue such as a chest drain that is correlated with the diagnosis. In a hospital where the drain no longer tracks the disease, the model is overconfident on images that contain the drain and underconfident on the clinically important cases without it. A per-class cutoff cannot tell these two situations apart. It either admits confident drain-driven errors or starves the drain-free positives it most needs.
We propose to set pseudo-label cutoffs per class and per inferred shortcut stratum rather than per class alone. A deliberately shortcut-prone auxiliary model assigns each unlabeled target image to a stratum, according to whether a shortcut alone could explain its prediction. Selection is then balanced across the resulting (class, stratum) cells with a single acceptance rate. That rate is chosen without target labels, using class-stratified pseudo-target validation. The aim is a self-training method whose pseudo-labels repair hidden stratification instead of amplifying it, with only one knob, and that knob can be tuned without labeled target data.
Core Claim
The claim is that confidence-driven class-adaptive thresholding fails under shortcut shift in a predictable direction, and that grouping by an auxiliary shortcut detector fixes most of the failure more cheaply than adding more annotation.
Three findings motivate the claim. Hidden stratification is real and large: pneumothorax AUROC is 0.94 with a chest drain present and 0.77 without one on ChestX-ray14 (Hidden stratification causes clinically meaningful failures in machine learning for medical imaging). Binary-label models collapse to a robust AUROC of 0.46 on SIIM-ACR when the drain correlation is broken (Reducing Reliance on Spurious Features in Medical Image Classification with Spatial Specificity). No class-adaptive thresholding method has been tested under either condition.
The methods we build on are all confidence- or weight-driven. DyConfidMatch maps class-average confidence through a concave function. AllMatch uses classifier weight norms as a signal of how well a class has been learned. JointMatch uses two peer networks with adaptive local thresholds. All three were evaluated in-distribution, on CIFAR, point clouds or text. The weight-norm signal is especially suspect here, because a class can gain a large weight norm by fitting an artifact.
Our proposal differs from these methods in one respect. The quantity that decides which pseudo-labels are admitted is not how confident the model is in a class. It is whether that confidence is also shared by a model that only knows shortcuts.
Research Program
The auxiliary "shortcut model" is trained only on the labeled source data. We build it to be biased on purpose, using either early stopping after one or two epochs or a low-capacity network on downsampled input. The motivation is that networks fit simple, low-dimensional cues first (Problèmes associés au déploiement des modèles fondés sur l'apprentissage machine en santé).
For each unlabeled target image, the main model gives a predicted class $\hat y$ and a confidence. The shortcut model gives its own probability for $\hat y$, which we bin into $K$ strata; $K=2$ or $3$ by default. Within each (class, stratum) cell we accept the top fraction $p$ of images ranked by main-model confidence. This is the class-balanced percentile rule of Class-Balanced Self-Training (CBST), extended from classes to cells. The consequence is that drain-free pneumothorax images, where the shortcut model disagrees, compete only with each other. They are no longer outranked by overconfident drain-bearing images. The student is then retrained with the usual weak/strong augmentation consistency.
The acceptance rate $p$ is the only new hyperparameter. We choose it with the pseudo-target validation of Pseudo-Label Validation for Unsupervised Domain Adaptation, which splits the source data, trains an imputer on one half, and scores candidate models against the imputer's soft labels on target. We modify it in one way: the surrogate loss is averaged within each (class, stratum) cell and then across cells, rather than pooled. This addresses the gap the file identifies, where a pooled score rewards checkpoints that do well on majority cells. The imputer is itself trained on source data and may carry the shortcut. We therefore also test whether the stratified score's ranking of checkpoints agrees with oracle ranking on the shortcut-free cells specifically.
The analysis component comes before the method is built. We first measure whether existing adaptive thresholds admit shortcut-driven errors. For AllMatch, DyConfidMatch and fixed-threshold FixMatch, we log pseudo-label precision and recall separately for target images with and without the shortcut. The shortcut status comes from ground-truth metadata, such as chest-drain annotations, that is used for evaluation only. We also track weight norms and class thresholds over training. If the shortcut class's weight norm grows fastest while its drain-free precision falls, the failure of the weight-norm signal is confirmed rather than just argued.
Baselines
The do-nothing reference is source-only supervised training (ERM) with standard augmentation, applied directly to the target. On SIIM-ACR with binary labels its reported robust AUROC is 0.46 (Reducing Reliance on Spurious Features in Medical Image Classification with Spatial Specificity). On Camelyon17 its out-of-distribution accuracy is 70.8% (Extending the WILDS Benchmark for Unsupervised Adaptation).
The strongest existing unsupervised adaptation method in the file is Noisy Student self-training. It raised Camelyon17 out-of-distribution accuracy to 86.7% in the same WILDS 2.0 study. It is the competitor to beat because it already exploits unlabeled target data well on a real hospital shift, using a global threshold. We found no reported result on Lacuna for any unsupervised adaptation method on the SIIM-ACR or ISIC robust splits, so those numbers will be produced here.
Class-adaptive competitors are DyConfidMatch's concave-mapping thresholds and AllMatch's weight-norm thresholds, both transplanted to the adaptation setting. We also include a plain CBST per-class percentile rule, which is the direct ablation: it is our method with $K=1$ and no shortcut model.
As an annotated reference point, not an equal competitor, we report Saab et al.'s mask-supervised model: robust AUROC 0.82 on SIIM-ACR and 0.73 on ISIC. It shows how much of the annotation-driven gain label-free adaptation recovers.
Fairness of comparison works as follows. All methods share the backbone, the source and target splits, and the training budget. Every method receives the same number of hyperparameter trials. Every method is tuned by the same label-free protocol, meaning the stratified pseudo-target validation. For each method we also report oracle-tuned results and the spread across its grid, so that sensitivity is measured and not hidden.
Beating the baselines means two things. The first is higher worst-cell AUROC or accuracy on the shortcut-decorrelated target than CBST and Noisy Student, with aggregate performance no lower than Noisy Student. On Camelyon17 that means at least 86.7%, where we expect parity because the shift there is staining rather than a localized shortcut. The second is a smaller gap between label-free-tuned and oracle-tuned performance than the competitors show.
Why not just…?
Why not just use CBST's per-class percentile selection? This is the most serious threat. Per-class percentiles already stop easy classes from monopolizing the pseudo-labels. If drain-free positives rank high enough within the positive class, strata add nothing. The $K=1$ ablation settles this directly. We will also report where drain-free positives fall in the within-class confidence ranking. If they are not concentrated at the bottom, we will concede that stratification is unnecessary on that dataset.
Why not just run Noisy Student with a global threshold, since it already works on Camelyon17? Its 86.7% comes from a staining shift. There, low confidence plausibly does reflect difficulty, and there is no confounder whose correlation with the label flips. Under a flipped drain correlation, a global threshold admits exactly the confident drain-bearing images. The per-stratum precision logging in our analysis tests this directly. Camelyon17 serves as the control where we predict no advantage.
Why not just recalibrate confidences with isotonic regression, then threshold? When is Multicalibration Post-Processing Necessary? finds isotonic regression cheap and competitive. However, it needs labeled data from the distribution being calibrated, which the target hospital lacks. Calibration fitted on source reproduces the source correlation. Multicalibration across groups needs the group labels we are trying to avoid, and the same paper calls those algorithms "hyperparameter divas". We include source-fitted isotonic recalibration followed by a fixed threshold as a baseline.
Why not just add spatial supervision or audit saliency maps? Masks work, but they cost annotation at every site. Saliency audits are unreliable against expert-labeled regions (Benchmarking saliency methods for chest X-ray interpretation; Agreement Between Saliency Maps and Human-Labeled Regions of Interest). Explanation-based unlearning cost five points of accuracy in Unlearning Spurious Correlations in Chest X-ray Classification. We use the mask result as an upper reference, not as something to argue away.
Proposed Experiments
The primary testbeds are SIIM-ACR pneumothorax, where the chest drain is the shortcut, and ISIC melanoma, where skin markings are. On both we build source splits where the shortcut is correlated with the label and unlabeled target splits where it is decorrelated, following the robust-split construction of Saab et al. For external transfer, a SIIM-ACR-trained model is adapted to ChestX-ray14, scored on the drain-annotated subset. The multi-source Covid-19 chest X-ray set, where bone age acts as a shortcut, is a secondary testbed, and Camelyon17 is the no-shortcut control. The metrics are robust AUROC, worst-(class, stratum) AUROC using the held-out metadata, aggregate accuracy, pseudo-label precision per stratum, and the gap between label-free and oracle tuning.
Three results would be decisive. First, the analysis must show that DyConfidMatch and AllMatch have lower pseudo-label precision on drain-free positives than a global threshold. If they do not, the motivating tension is false and the project stops there.
Second, the method must beat the $K=1$ CBST ablation on worst-cell AUROC on SIIM-ACR and ISIC while tying on Camelyon17. A gain everywhere would suggest a generic regularization effect rather than a shortcut-specific one.
Third, when the shortcut model is replaced by a random stratifier, the gain must disappear. When the shortcut model is replaced by the true metadata stratifier, the result gives the ceiling that a perfect shortcut detector would reach, which shows how much our inferred strata leave on the table.
References
- Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging
- Reducing Reliance on Spurious Features in Medical Image Classification with Spatial Specificity
- DyConfidMatch: Dynamic thresholding and re-sampling for 3D semi-supervised learning
- AllMatch: Exploiting All Unlabeled Data for Semi-Supervised Learning
- JointMatch: A Unified Approach for Diverse and Collaborative Pseudo-Labeling to Semi-Supervised Text Classification
- Problèmes associés au déploiement des modèles fondés sur l’apprentissage machine en santé
- Pseudo-Label Validation for Unsupervised Domain Adaptation
- Extending the WILDS Benchmark for Unsupervised Adaptation
- When is Multicalibration Post-Processing Necessary?
- Benchmarking saliency methods for chest X-ray interpretation
- Agreement Between Saliency Maps and Human-Labeled Regions of Interest: Applications to Skin Disease Classification
- Unlearning Spurious Correlations in Chest X-Ray Classification