Context Budgets That Shrink With Class Size: Anatomical Masking Scaled to Minority-Class Evidence

Medical image classifiers often learn the wrong cue. A pneumothorax model learns to spot the chest tube placed after treatment. A lung ultrasound model learns how thick the patient's subcutaneous fat is. Two remedies have been tried. One masks out everything outside the organ. The other keeps the whole image for context. The evidence points both ways. Zeroing out non-lung tissue improves CT triage and lung ultrasound scoring, but chest X-ray models that see only the segmented lungs do worse than models that also see the full image.

We propose that both findings hold, and that what separates them is how many examples a class has. A rare class, such as squamous cell carcinoma in 18 female patients, cannot afford to draw evidence from the millions of out-of-organ pixels. With so few examples, any correlation found there is more likely noise or a shortcut than biology. A common class can afford that context. Our method gives each class its own allowance for out-of-organ evidence, and it shrinks the allowance as the class gets rarer, at a rate set by a standard sample-complexity argument rather than by tuning.

The idea joins three problems that are usually treated separately. Anatomical noise is the source of the spurious features. The high-dimensional, low-sample-size problem (many more input dimensions than patients) is the reason those features are easy to fit. Class imbalance determines how small that sample size is for the classes that matter most. The core claim is simple. The right amount of background suppression is not one global setting. It is a per-class quantity tied to how much data each class has.

Core Claim

Prior work picks one masking strength for the whole model. Three papers support hard masking. Mask-weighted global average pooling for CT COVID-19 triage multiplies ResNet-50 features by a smoothed 3D U-Net lung mask. This improved specificity by 2 points over standard pooling. The Role of Pleura and Adipose in Lung Ultrasound AI masks subcutaneous fat and raises severity-scoring AUC from 0.6553 to 0.7742. Reducing Reliance on Spurious Features with Spatial Specificity shows that binary-label training on SIIM-ACR reaches a robust AUROC of only 0.46. Segmentation-mask supervision raises it to 0.82.

SDFN points the other way for context. It fuses segmented-lung crops with the full chest X-ray and reaches a mean AUC of 0.815, against 0.807 for lungs alone and 0.804 for the full image alone. None of these papers asks whether the answer differs across classes of very different size.

Audits of hidden stratification explain why class size should matter. Hidden stratification causes clinically meaningful failures shows that shortcuts concentrate on rare, high-severity subgroups. We turn that observation into a training rule.

Concretely, the classifier has two pooled feature vectors per image. The first, $f_{in}$, is pooled inside a soft, dilated organ mask. The second, $f_{out}$, is pooled outside it. Each class logit is

$$z_c = w_c^\top f_{in} + v_c^\top f_{out},$$

where $w_c$ and $v_c$ are the class-$c$ weights on in-organ and out-of-organ features. Training adds a penalty

$$\lambda \sum_c \sqrt{d_{out}/n_c}\;\lVert v_c \rVert_2,$$

where $d_{out}$ is the dimension of $f_{out}$, $n_c$ is the number of training examples of class $c$, and $\lambda$ is a single global constant. The $\sqrt{d/n}$ form comes from the usual complexity bound for a linear predictor estimated from $n$ examples in $d$ dimensions. A class with one-twenty-fifth the data therefore pays five times more per unit of out-of-organ weight.

The group norm can drive $v_c$ exactly to zero. The result is hard masking for rare classes, fused context for common ones, and a continuum in between. Only one hyperparameter is tuned. This matters because a free per-class strength cannot be tuned reliably by cross-validation on a class with 18 examples.

Research Program

The first part of the program is building the model. Organ masks come from off-the-shelf segmenters: a lung U-Net of the kind used in the CT triage paper, which reports Dice 96.5% on LUNA16, and a JSRT-trained lung segmenter for chest X-ray, where SDFN reports Dice 0.98. Masks are Gaussian-smoothed and dilated so that small errors at the organ boundary do not clip lesion margins. Only organ masks are needed, never lesion masks. Organ masks are cheap, which avoids the annotation-cost objection raised against lesion-level supervision.

For 3D CT we place the two-pathway pooling inside the slice-level attention aggregation of Fair Lung Disease Diagnosis via Gender-Adversarial Attention MIL. That paper's multiple-instance learning (scoring each slice and letting a learned attention weight decide which slices count) handles the through-plane dimension. It leaves in-plane chest wall, fat, and scanner bed untouched, and our pathways cover that gap.

The second part is analysis. We test whether the learned out-of-organ allowance $\lVert v_c \rVert$ actually tracks $n_c$. We also test whether shortcut reliance concentrates in classes whose $v_c$ stays large.

The third part is a controlled test of the scaling law itself. On SIIM-ACR we subsample pneumothorax-positive images to create minority sizes spanning two orders of magnitude, keeping the test set fixed. The proposal predicts that the gap between hard masking and full context grows as the minority shrinks. It also predicts that the frequency-scaled penalty tracks the better of the two at every size.

Baselines

The naive reference point is a full-image classifier trained on the same backbone with class-balanced weighting and the same oversampling as every other arm. It shows what imbalance handling achieves with no anatomical constraint.

The strongest existing method on the multi-class CT benchmark is the full pipeline of Fair Lung Disease Diagnosis via Gender-Adversarial Attention MIL. It combines a ConvNeXt slice encoder, attention-based multiple-instance pooling, gradient reversal against gender, focal loss with label smoothing, weighted oversampling, and out-of-fold threshold tuning. It reports a mean validation competition score of 0.685 (std 0.030), female macro-F1 of 0.691, male macro-F1 of 0.679, and squamous cell carcinoma F1 of 0.366. We will rerun it on the same joint class-by-gender stratified folds, with the same encoder, the same epochs, and the same hyperparameter search budget. Our method will then be added on top of the same pipeline rather than replacing it, so the comparison isolates the pooling change. Beating it means a higher mean score outside one fold-standard-deviation and a clearly higher carcinoma F1, especially for female patients.

The hard-masking baseline is mask-weighted global average pooling as in the CT triage paper, given identical imbalance handling. This corresponds to setting $v_c = 0$ for all classes. The full-context baseline is SDFN-style fusion with a single uniform penalty on $v_c$, which corresponds to dropping the $\sqrt{1/n_c}$ factor. Its single $\lambda$ gets the same search budget as ours.

Three ablations isolate the key ingredient. The first is the uniform penalty just described. The second inverts the scaling so that rare classes get the most context. The third tunes per-class penalties freely by nested cross-validation. If free tuning matches the scaled rule, the rule offers convenience, not insight.

On SIIM-ACR the reference numbers are those reported by the spatial-specificity paper: robust AUROC of 0.46 for binary labels, 0.72 for bounding boxes, and 0.82 for segmentation masks. Our method uses only lung masks, not lesion masks. Reaching the bounding-box level without lesion annotation would count as a meaningful result. On the lung ultrasound cohort, if it can be obtained, the masked result of 0.7742 AUC is the target.

Why not just…?

Why not just hard-mask everything for all classes? This is the main threat on benchmarks where context carries no signal. On those benchmarks we expect the scaled penalty simply to converge to hard masking. SDFN's 0.807 versus 0.815 gap shows that context helps for at least some chest X-ray classes. The SIIM-ACR subsampling sweep and the per-class breakdown on Chest X-ray 14 will show whether hard masking costs accuracy on common classes. If it never does, we will report that context budgets are unnecessary.

Why not just use focal or class-balanced loss on the full image? Reweighting changes how much minority errors count, not where the evidence comes from. SIIM-ACR's 0.46 robust AUROC under plain binary training shows the network reaches for the chest tube. In the fair CT paper, focal loss plus oversampling still left carcinoma F1 at 0.366. Every arm in our study already includes this reweighting, so the naive baseline measures exactly what it buys on its own.

Why not just tune one global context weight, or tune it per class by cross-validation? One global weight forces common and rare classes to share a compromise. Per-class tuning on 18 examples has validation folds of three or four cases, which is too noisy to select anything. The uniform-penalty and free-tuning ablations test both claims directly. If either matches the scaled rule within fold variance, the scaling claim fails.

Why not just augment the minority class with a generative model? Geometry-based VAE augmentation raised ADNI balanced accuracy from 77.7% to 86.3% on an imbalanced split. It is complementary rather than competing. A generator trained on 18 unmasked scans can also reproduce their background shortcuts, so synthetic cases do not remove the need to constrain where evidence comes from. We treat this as an add-on, not a replacement.

Proposed Experiments

The decisive experiment is the SIIM-ACR imbalance sweep. Positives are subsampled to roughly 1,000, 300, 100, and 30 images, and we measure robust AUROC on tube-stratified test images. The proposal is supported only if two things hold. First, the advantage of masking over context must grow as the minority shrinks. Second, the scaled penalty must stay within noise of the best fixed choice at every size, while the uniform penalty does not. The penalty can only act on out-of-organ features, and chest tubes partly overlap the lung field on projection images. In-mask shortcuts are therefore outside what this method can fix, and we will report that portion separately.

The real-world test is the fair-diagnosis CT challenge. It has 889 volumes in four classes, and only 18 of them are female squamous cell carcinoma. We report the competition score, per-class F1, and gender-specific macro-F1 against the rerun pipeline. As a secondary check, Chest X-ray 14 tests whether the rule keeps SDFN's context gains for common pathologies while not losing ground on its rarest ones.

Three diagnostics complete the study. We plot learned $\lVert v_c \rVert$ against $n_c$. We stratify accuracy by segmentation Dice to see whether segmenter failures on severe disease hurt minority classes more. And we compare out-of-organ attribution mass per class against its in-organ counterpart. A null result, where the learned allowances bear no relation to class size or the ablations tie, would itself settle the masking-versus-context dispute in favor of one global choice.