When the Mask Carries the Shortcut: Auditing and Hardening Two-Stage ROI Pipelines Across Centers

A common defense against shortcut learning in medical imaging is to segment the anatomy first and let the diagnostic classifier see only that region. Shortcut learning here means a model latching onto features that happen to correlate with the label, such as a chest drain or a hospital watermark, instead of the disease itself. Region-of-interest (ROI) masking of this kind has worked well in controlled settings. Masking out subcutaneous fat raised lung-ultrasound severity AUC from 0.6553 to 0.7742 (The Role of Pleura and Adipose in Lung Ultrasound AI). Pixel-level masks lifted worst-group pneumothorax AUROC from 0.46 to 0.82 (Reducing Reliance on Spurious Features in Medical Image Classification with Spatial Specificity). Both results, however, rest on masks that are either hand-drawn or produced in the same hospital the classifier is tested in. In deployment the mask comes from a segmentation network trained at one center, and that network can learn its own shortcuts, such as calipers, centering, and zero-padded borders (Shortcut Learning in Medical Image Segmentation). Masking may therefore not remove the shortcut. It may move the shortcut into the mask, whose shape, size, and failures at a new hospital are themselves correlated with the label and with the site.

This note proposes to measure that relocation directly and to fix the part of it that matters. The claim has two parts. First, the robustness of a two-stage pipeline at a new center is bounded by how the classifier behaves under the segmenter's target-site errors, not by its accuracy on clean masks. Second, those errors can be approximated without any target data. We stress the source-center segmenter with synthetic acquisition shifts and train the classifier on the resulting imperfect masks. In two sentences: segment-then-classify pipelines inherit a hidden failure mode, because the mask is learned from biased single-center data and breaks or leaks label information at new sites. We build an audit that exposes this, and a training scheme that exposes the classifier to realistic, shift-induced mask errors so that it does not depend on mask quality it will not get.

Core Claim and What Is New

The closest work treats each half of the pipeline in isolation. Saab et al. show that spatial supervision breaks reliance on spurious features, but they use ground-truth masks and single-dataset spurious splits. Wenger et al. show that segmenters learn shortcuts, but they stop at segmentation Dice and never follow the corrupted mask into a diagnosis. The research file's own gap analysis notes that no study has evaluated this cascade. Single-source generalization methods for segmentation (Domain Game, FAN-Net, SynthSeg) make the mask itself more robust, but they report only segmentation scores.

Our contribution is to treat the segmenter's error distribution under shift as the object of study. One quantity organizes the work, the propagation gap at a target center $t$:

$$\Delta_t = \mathrm{AUROC}_t(\text{classifier} \mid \text{oracle masks}) - \mathrm{AUROC}_t(\text{classifier} \mid \text{predicted masks}),$$

Here $\mathrm{AUROC}_t$ is the worst-group AUROC on center $t$. Worst-group means the clinically hard stratum, such as pneumothorax without a chest drain. We hypothesize that $\Delta_t$ is large for single-center pipelines even when segmentation Dice at $t$ looks acceptable, because a few failures in exactly the hard stratum dominate the classification error. The disrupted pleural line in severe disease, which Gare et al. report breaks their segmenter, is one example of such a failure.

Research Program

The work has three parts.

The audit. For each pipeline we compute three quantities: $\Delta_t$ across centers, Dice broken down by label and by stratum, and a mask-leakage probe. The probe trains a small network on the binary predicted mask alone, with no image intensities, to predict the diagnosis and, separately, the source center. If the mask by itself predicts the label above chance in the hard stratum, or identifies the hospital, then shortcut information is reaching the classifier through mask geometry. Image-level masking cannot remove that leakage. A cheap bias check like this was not available before, and it requires no saliency maps. That matters because saliency maps align poorly with expert regions (Benchmarking saliency methods for chest X-ray interpretation).

The method: stress-derived mask errors. We train the source segmenter normally. We then generate a family of shifted versions of each source training image. One shift type is low-frequency amplitude swaps and rescaling, following the observation in FAN-Net that the amplitude spectrum carries site style while the phase spectrum carries boundaries. A second is per-tissue random intensity remapping in the style of SynthSeg, which mimics contrast inversions between modalities. A third is removal or inpainting of calipers and text, and a fourth is off-center cropping, following Wenger et al. Running the frozen segmenter on these stressed images produces masks whose errors are realistic: they fail where the segmenter's shortcuts fail, not as random blobs. The downstream classifier is trained on images masked with a mixture of these stressed masks and clean masks. Mask-weighted pooling uses a soft, dilated weighting in the spirit of mask-weighted GAP, so that tissue next to the lesion stays partly visible and a slightly misplaced boundary does not delete the pathology. A consistency loss ties the classifier's predictions under clean and stressed masks for the same image.

The cross-modality stress test. Physics shifts such as MR to CT produce the most severe mask failures. The file does not contain a cross-modality dataset with diagnostic labels, so we build a controlled testbed on MM-WHS 2017. The diagnostic label is derived from ground-truth anatomy, for example left-ventricle cavity volume above a cohort percentile. A spurious cue is injected at a controlled correlation in the source modality only. This is explicitly a probe of mechanism, not a clinical claim. It tests whether stress-derived training transfers when segmenter errors come from a genuine change of imaging physics.

Baselines

The naive reference is a single-stage classifier trained on raw images with binary labels. Its known behavior gives an anchor. Saab et al. report a robust AUROC of 0.46 on SIIM-ACR for this setting, and Hidden Stratification reports AUC 0.94 with drains versus 0.77 without drains on CXR14.

The strongest existing competitor is Saab et al.'s spatial-specificity classifier, which uses a supervised pixel-level similarity loss with segmentation masks plus semi-supervised learning. It reports robust AUROC 0.82 on SIIM-ACR with segmentation masks and 0.73 on ISIC. It is the strongest competitor because it already uses masks to suppress shortcuts and needs no test-time segmenter, so its errors cannot cascade. We retrain it on exactly our source center's data and mask budget. It receives the same backbone, the same augmentation pool minus our stressed masks, and an equal hyperparameter search budget of a fixed number of trials on a source-only validation split. We evaluate it on the same held-out centers. Beating it means a higher worst-group AUROC at unseen centers with non-overlapping bootstrap intervals. Matching it in-distribution while losing at external sites would count as failure.

The second strong competitor is the standard two-stage pipeline with Wenger et al.'s mitigations applied to the segmenter: random cropping, reflection padding, and annotation inpainting. A further variant applies SynthSeg-style or FAN-style randomization to the segmenter. Both test whether simply making the mask robust is enough. They receive the same compute and tuning budget.

Two ablations isolate the key ingredient. In the first, the classifier is trained on generic morphological mask jitter (random dilation, erosion, and shifts of the same magnitude as the stressed-mask errors) instead of stress-derived masks. If this matches our method, the "realistic errors" argument fails. In the second, hard binary masking replaces soft dilated pooling, to separate the gain from preserving nearby tissue from the gain from error exposure. Oracle-mask results define the ceiling used to compute $\Delta_t$.

Why not just…?

Why not just fix the segmenter with random cropping, alternative padding, and annotation removal? This is the most serious threat. Wenger et al. show these fixes restore fetal-ultrasound Dice to roughly 90–98% from 70–80%. If hardened segmenters drive $\Delta_t$ to near zero at external centers, our classifier-side training is unnecessary, and we would report that as the main finding. We expect a residual gap for two reasons. First, these fixes target spatial artifacts, not failures caused by disease severity, such as the disrupted pleura Gare et al. describe. Second, the leakage probe can reveal label information in mask geometry that survives a well-performing segmenter. The combination of hardened segmenter and stress-derived classifier, compared against the hardened segmenter alone, settles it.

Why not just skip test-time masking and use masks only as training supervision, as Saab et al. do? This removes the cascade entirely, and it is our main baseline. Its limitation is that robustness at a new center then depends on the single-stage model ignoring background it can still see. Its reported 0.82 robust AUROC comes from a split within one dataset, not an external hospital. If it matches our pipeline under leave-one-center-out evaluation, the conclusion is that two-stage pipelines are not worth their failure mode, which is a useful negative result.

Why not just pool data from several hospitals? Pooling helps when it is available. In the PSMA-PET study, leave-one-center-out training reached a median DSC of 0.74 against 0.76 for mixed training (The impact of multicentric datasets). Our setting assumes a single source center, and confounded pooling can teach site-as-severity. We include a pooled-centers upper reference wherever several sources exist, the COVID CT cohorts in particular, to quantify how much of the pooled benefit single-source stress training recovers.

Why not just use target-domain adaptation such as SIFA? SIFA raised MR-to-CT cardiac Dice from 17.2% to 73.0% (SIFA), but it needs unlabeled target scans. In the MM-WHS testbed we run SIFA as a privileged reference that is allowed target access, not as a competitor.

Proposed Experiments

The primary experiment trains on SIIM-ACR (source) and tests on CXR14 as an external center, with the chest-drain stratum as the worst group. The decisive comparisons are three: worst-group AUROC against the reproduced Saab classifier, $\Delta_t$ for each pipeline, and mask-leakage AUROC for the label and for the dataset of origin.

The second experiment uses the multi-source COVID CT cohorts (LUNA16, MosMedData, and a private cohort) in a leave-one-source-out design. It tests whether the mask-weighted GAP pipeline, at 96.5% sensitivity and 87.8% specificity as reported, holds when its pulmonary segmenter is trained on one source only.

The third experiment uses ISIC with surgical markings and the ISICCROP off-center shift. Here the segmenter's centering shortcut is known to fail, so it measures directly how much of that failure reaches melanoma classification.

The fourth is the MM-WHS cross-modality testbed described above.

The findings that would confirm the thesis are threefold: a substantial $\Delta_t$ for standard pipelines despite reasonable external Dice, non-trivial leakage from the mask alone, and a reduction of both under stress-derived training that generic mask jitter does not reproduce. Any of these failing, especially hardened segmenters alone closing the gap, would refute the central claim, and we would report it as such.