Chest X-ray classifiers often learn from the wrong pixels. They pick up hospital-specific markings, support cables, positioning cues from bedridden patients, and body-habitus signals that correlate with age, sex, or race. When a model is trained at one hospital and deployed at another, these cues change and accuracy drops, often unevenly across patient groups. The common fix is to segment the organ first and let the classifier see only the lungs and heart. That fix has a hidden cost. The masks fail more often on the sickest patients, and a mask drawn around "normal" anatomy can cut away the dense consolidation or effusion the classifier needs to see. Masking can therefore move bias from the classifier into the segmenter instead of removing it.
We propose a middle path. The classifier starts from a fixed anatomical mask, but it is allowed to learn its own soft extension of that mask. A spatial smoothness penalty called total variation, which charges the model for every abrupt jump between neighbouring mask values, forces that extension to be made of a few contiguous blobs. The idea rests on an asymmetry. Real pathology beyond the anatomical boundary tends to form large connected regions, while most acquisition shortcuts are small, scattered, high-frequency marks. A smooth mask should be able to grow to cover the first kind of signal while being structurally unable to select the second. We test this in the regime where it should matter most: training on a small single-center cohort and evaluating on four external centers, broken down by patient subgroup and acquisition view.
Core Claim
Anatomical priors already give strong structure on their own. CheXmask supplies 657,566 masks produced by a landmark-graph model across ChestX-ray8, CheXpert, MIMIC-CXR, Padchest and VinDr-CXR. Each mask comes with a per-image quality estimate from Reverse Classification Accuracy, a method that scores a predicted mask without needing a ground-truth mask for that image (CheXmask). What is missing is a way to use these masks in diagnosis without losing pathology the masks exclude.
Our claim has two parts. First, a classifier whose pooling is weighted by a learned, total-variation-regularized soft mask, anchored to the anatomical prior, keeps the out-of-distribution robustness of hard masking. Second, the same classifier recovers the sensitivity that hard masking loses on anteroposterior (AP) views and severe disease, and it reduces the gap between worst-off and best-off subgroups at external centers. The smoothness term is the key ingredient, not decoration. Without it, a learned mask becomes a pixel-level selector that can re-admit exactly the shortcuts masking was meant to remove.
Relation to Prior Work
Three papers show that restricting the classifier to anatomy helps, and they motivate the proposal. Masking subcutaneous fat in lung ultrasound raised AUC from 0.6553 to 0.7742 by removing a body-habitus shortcut (Pleura and Adipose in Lung Ultrasound AI). Weighting global average pooling with a Gaussian-smoothed lung mask improved CT triage specificity from 85.8% to 87.8% (mask-weighted GAP for COVID-19 CT). Lesion-mask pooling reached 0.982 malignancy AUC on an external breast ultrasound set (Lesion-Centric Latent Phenotypes). All three use fixed masks, and none reports fairness across subgroups. The ultrasound paper explicitly fails when consolidation disrupts the pleural line. We keep their pooling mechanism but make the mask learnable within limits.
Two papers constrain the design. CheXmask documents higher mask failure on AP views and notes that masks defined by aerated lung clip out severe opacities (CheXmask). This is the excision problem we target. The cytology work pairs total variation with a low-rank background model to remove slide-discoloration shortcuts, but it reports blurring of fine structure and is validated at a single center (Background removal for debiasing cytology). The melanocytic confocal microscopy segmenter uses a Dice-plus-total-variation loss across six US and Italian clinics and plateaus at 70–74% sensitivity on distorted patterns (MED-Net). Both warn that smoothness can erase fine pathology. We therefore apply total variation to the pooling mask, not to the image or the segmentation, and we measure its effect on subtle findings directly.
The landmark-graph work supplies the prior. It showed that pixel U-Nets learn center-specific shortcuts, with latents clustering by hospital, while graph-structured anatomy mixes domains (Multi-center segmentation with heterogeneous labels). We reuse those masks as the anchor.
On fairness, shortcut testing modulates how much a model encodes a demographic attribute using gradient reversal. It confirmed race-linked cardiomegaly unfairness on CheXpert (Spearman ρ = 0.469). It also showed that scrubbing an attribute fails when disease genuinely looks different across groups (ShorT). ShorT works on unmasked images and needs demographic labels at training time. Our method needs neither, and we use ShorT as both a competitor and a diagnostic. FairerOPTH restricts attention to annotated lesion features to reduce age and sex gaps (FairerOPTH), but it requires 67 manually boxed feature types from a single center. Our anchor comes free from CheXmask.
Research Program
The model takes the CheXmask anatomical mask $A$ (lungs plus heart, values in [0,1]) and predicts a residual map $r$ from a light head on the backbone's early features. The pooling mask is $m = \sigma(\alpha \cdot \mathrm{logit}(A) + r)$, where $\sigma$ is the sigmoid and $\alpha$ is a fixed sharpness constant. The final feature map is averaged with weights $m$. Training minimizes
$$\mathcal{L} = \mathcal{L}_{\mathrm{cls}} + \lambda_{\mathrm{TV}} \sum_{i,j} \big(|m_{i+1,j}-m_{i,j}| + |m_{i,j+1}-m_{i,j}|\big) + \lambda_{\mathrm{out}} \sum_{i,j} m_{i,j}(1-A_{i,j}),$$
where $\mathcal{L}_{\mathrm{cls}}$ is the multi-label classification loss, the middle term is total variation on the mask, and the last term charges for mask area outside anatomy.
The two penalties divide the work. The area term allows extension beyond anatomy but makes it costly. The total variation term makes a scattered extension far more costly than a compact one of the same area. This is because total variation scales with boundary length, and many small islands have much more boundary than one blob.
The program has four parts. First, an audit using existing CheXmask quality scores: does estimated mask quality differ by sex, age bracket, race (where available) and AP versus PA view? That tells us whether hard masking itself carries population-specific bias. Second, the method above, with $\lambda_{\mathrm{TV}}$ and $\lambda_{\mathrm{out}}$ swept jointly. Third, a controlled shortcut-injection study to test the asymmetry hypothesis. Fourth, a multi-center, small-sample evaluation with subgroup breakdowns.
Baselines
The naive reference is an unmasked DenseNet-121 with standard global average pooling, following the CheXpert setup. That paper reports in-distribution AUC of 0.97 for pleural effusion and 0.94 for edema on the Stanford consensus set (CheXpert). We use those figures only to check that our training reproduces in-distribution quality. No reported result was found on Lacuna for small-sample, single-center-trained CheXpert models evaluated on MIMIC, NIH, Padchest and VinDr with subgroup breakdowns.
The strongest masking competitor is fixed mask-weighted pooling with a Gaussian-smoothed CheXmask mask, following the CT triage paper (mask-weighted GAP). That paper reports +0.9% sensitivity and +2.0% specificity over plain pooling on its CT task. It is the most direct form of the idea without learning. We add a hard-crop variant, and a fixed-dilation variant that enlarges the mask by a tuned margin. The dilation variant is the cheapest answer to the excision problem.
The strongest fairness competitor is ShorT-style gradient reversal on age, sex and race over the unmasked model (ShorT). It is run with demographic labels that our method never sees. This favors the competitor on purpose. ShorT reports that this halves unfairness on chest X-rays but gives no absolute AUC figure in the file.
The ablations isolate the key ingredient. They are the learned mask with $\lambda_{\mathrm{TV}}=0$, the learned mask with no anatomical anchor (total variation and area penalty only), and our method combined with ShorT. Every model uses the same backbone, input resolution, augmentation, training subsets and a 24-configuration hyperparameter search. Model selection uses a held-out split from the training center only, never the external centers.
Beating the baselines means two things at once, both on external centers and averaged over five seeds. On AP-view positives for consolidation, effusion and edema, our method must match or exceed fixed-dilation masking in sensitivity. On the headline robustness metrics, it must beat fixed mask-weighted pooling in worst-subgroup AUC. If fixed dilation matches us on both, the learned mask is not worth its complexity, and we will report that.
Why not just…?
Why not just hard-mask with CheXmask and be done? Hard masking is our expected robustness winner on normal-looking images, and we fully include it. CheXmask itself documents two problems: masks defined by aerated lung clip severe opacities, and failure rates rise on AP views, which come disproportionately from acutely ill patients. The quality audit and the AP-positive sensitivity comparison will show whether hard masking loses sensitivity in exactly the cases that matter. If it does not, the excision half of our motivation collapses.
Why not just dilate the mask by a fixed margin? This is the real threat. Dilation is simple, needs no learning, and cannot absorb shortcuts outside its margin. Its weakness is that one margin cannot fit both a small apical nodule and a large basal effusion. A margin wide enough for effusions also admits cables, lines and soft tissue at the lung edges. We tune the margin separately per center-free validation split and compare directly. If dilation ties us, we will conclude that the learned extension adds nothing.
Why not just use gradient reversal on demographics? ShorT shows that this works when the attribute is a non-causal shortcut. It also shows that it fails when disease presentation truly differs across groups, as with acne across ages. It needs labels that many cohorts lack, and it does nothing about center-specific acquisition marks that are unrelated to demographics. We run it with full labels, alone and combined with our mask, to see whether the two are complementary or redundant.
Why not just train on more centers? Pooling data does not remove shortcuts when collection is skewed. RETFound's 1.6 million images still tied its representation to UK demographics and devices (RETFound). Our question is also specifically about small single-center cohorts. As a reference ceiling, we still train one pooled multi-center model.
Proposed Experiments
Training uses CheXpert subsampled to 500, 2,000 and 10,000 patients, with labels restricted to findings shared with the external sets: cardiomegaly, effusion, consolidation, edema and atelectasis, with VinDr mapped where its labels overlap. External testing uses MIMIC-CXR, ChestX-ray8, Padchest and VinDr-CXR, all with CheXmask masks. Metrics are per-finding AUC per center, worst-subgroup AUC, and the maximum minus minimum AUC across sex, age brackets and race where available. We also report AP versus PA sensitivity at a fixed specificity, and the fraction of Grad-CAM attribution falling inside the anatomical mask.
The first decisive check is shortcut injection. We stamp synthetic center-correlated marks into training images in two forms: scattered small glyphs, and a single contiguous gray patch. Each form is placed either inside or outside the lungs. At test time we flip the correlation and measure how much the model relied on each mark. The hypothesis predicts a specific pattern. The total-variation mask should suppress scattered marks much more than the $\lambda_{\mathrm{TV}}=0$ ablation does. It should give little protection against a contiguous patch inside the anatomy. That pattern states the method's limits honestly.
The second check is excision recovery. On AP-view positives with large opacities, where pathology extends past the CheXmask boundary, learned-mask sensitivity should exceed hard masking. We will inspect whether the extension covers the opacity or drifts into soft tissue.
The third check addresses the oversmoothing concern raised by the cytology and MED-Net papers. We sweep $\lambda_{\mathrm{TV}}$ and track sensitivity on small findings (nodule and mass labels, where available) separately. If small-finding sensitivity falls faster than robustness improves, smoothness is costing the patients it was meant to protect.
The fourth check tests the small-sample hypothesis. The advantage over the unmasked model should be largest at 500 patients and shrink by 10,000. If instead it is flat or grows with data, the mechanism is not what we claim.
References
- CheXmask: a large-scale dataset of anatomical segmentation masks for multi-center chest x-ray images
- The Role of Pleura and Adipose in Lung Ultrasound AI
- Computed tomography–based COVID–19 triage through a deep neural network using mask–weighted global average pooling
- Lesion-Centric Latent Phenotypes from Segmentation Encoders for Breast Ultrasound Interpretability
- Background removal for debiasing computer-aided cytological diagnosis
- Segmentation of cellular patterns in confocal images of melanocytic lesions in vivo via a multiscale encoder-decoder network (MED-Net)
- Multi-Center Anatomical Segmentation with Heterogeneous Labels Via Landmark-Based Models
- Detecting shortcut learning for fair medical AI using shortcut testing
- Fairer AI in ophthalmology via implicit fairness learning for mitigating sexism and ageism
- CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
- A foundation model for generalizable disease detection from retinal images