Medical image classifiers usually miss the cases that matter most: the small pneumothorax, the single suspicious slice in a CT volume, the rare tumor subtype. Two separate problems cause this. The first is global feature insufficiency. When a network averages a whole scan into one vector, a lesion covering under one percent of the image is drowned out by normal anatomy, and the model learns to lean on context such as chest drainage tubes instead (Searching for Pneumothorax in Half a Million Chest X-Ray Images). The second is class imbalance. Rare diseases get few training examples, so average-loss training lets them fail quietly.
Our proposal treats these as one problem. Inside each disease class there is a second, hidden imbalance: easy positives with large or treated lesions outnumber subtle ones with small lesions. A spatial model that looks at patches or slices can estimate how much of each scan its evidence comes from. We use that estimate to define subgroups by lesion extent within each class. We then train with group distributionally robust optimization, an objective that minimizes the loss of the worst-performing subgroup instead of the average. Crucially, the "worst-group" search runs separately inside each class, so the rare classes themselves are never the target of the adversary.
Core Claim
We claim that robust training for sparse-lesion medical imaging should reweight lesion-extent strata within each class, and should not reweight individual samples or pool all classes together. The extent strata are inferred from the attention map of a multiple instance learning model. Multiple instance learning (MIL) treats a scan as a bag of patches or slices that carries only a scan-level label.
Concretely, a frozen first-pass MIL model assigns each training bag an effective lesion extent $e(x) = \exp(H(a(x)))/n(x)$. Here $a(x)$ is the model's attention distribution over the $n(x)$ instances of bag $x$, and $H$ is its entropy, so $e(x)$ is the fraction of instances the model effectively relies on. Bags of class $c$ are binned by $e$ into strata $b \in \{1,\dots,B\}$, and the second-stage model is trained with
$$\min_\theta \sum_{c} \pi_c \max_{q_c \in \Delta_B} \sum_{b=1}^{B} q_{c,b}\, L_{c,b}(\theta).$$
In this objective, $L_{c,b}$ is the mean loss on stratum $b$ of class $c$. The term $q_c$ is a distribution over that class's strata chosen by the adversary, and $\pi_c$ are fixed class weights (uniform over classes). Because each $q_c$ lives in its own simplex, the adversary cannot move probability mass from majority classes onto the minority class. Moving mass across classes that way is the failure mode that DRAUC identified for pooled Wasserstein robust training on long-tailed data. The inner maximization is solved with the standard exponentiated-gradient group update, so it costs almost nothing beyond ordinary training. There is no per-batch gradient-ascent loop.
The prediction is specific: small-extent positives, and above all small-extent positives of rare classes, should gain the most. Overall AUC should stay flat or improve. Pooled robust training should do worse than plain training on the rare classes.
Relation to Prior Work
Three papers each connect two of the three directions, and our design takes something different from each.
Distributionally Robust Optimization for Deep Kernel MIL is the method-supplying neighbor on the spatial side. It puts an uncertainty set over instance weights inside a bag, which yields a robust soft pooling. That model held 93.49% AUC on the ShanghaiTech outlier benchmark, where standard MIL fell to about 62%. It was built for video with balanced bag classes. We keep robustness at the bag level and use the instance attention only to define groups, for a specific reason. In a positive bag with a sub-1% lesion, a worst case over instance weights spreads weight onto background, which pushes the model to call normal tissue positive. That rewards exactly the global context shortcuts we want to remove. We test this concern directly rather than assume it.
Class Unbiasing for Generalization in Medical Diagnosis motivates the proposal on the imbalance side. It combines group robust training over classes with a penalty on the gap between positive-class and negative-class loss, reaching macro-F1 0.627 against 0.483 for the best baseline on DAIC-WOZ. However, its groups are whole classes and its features are global vectors. That leaves it blind to the within-class subtle-versus-obvious split, which is where sparse lesions fail.
DRAUC constrains the design. Its analysis of how training representations settle into a fixed geometry late in training shows that pooled robust objectives concentrate their attack on tail classes. Its class-decoupled variant fixes this, but only for global features on CIFAR-scale images. We borrow the decoupling principle and move it to strata defined by spatial evidence.
On the spatial side, attention MIL for CT (Fair Lung Disease Diagnosis from Chest CT) and region mining with smooth-max pooling for pelvic fractures (Weakly Supervised Universal Fracture Detection in Pelvic X-rays) show that local aggregation beats global pooling. The fracture model reached AUC 0.975 against 0.962 for a single-stage global model. Neither uses robust objectives. The CT paper's rare squamous cell carcinoma class still sits at F1 0.366 despite focal loss and oversampling. That gap is the one we target.
Research Program
The work has three parts.
The first is a diagnostic study. On imbalanced medical data, we measure how three existing robust objectives redistribute error across class and extent strata. The three objectives are pooled sample-level robust training, class-level group robust training, and instance-weight robust pooling in the style of DRO-DKMIL. This tests two hypotheses: that pooled robustness attacks rare classes in medical imaging as DRAUC found on CIFAR, and that instance-level robust pooling harms small-lesion positives.
The second is the method: a two-stage attention MIL model with the class-decoupled extent-stratified objective above. We study how many strata $B$ to use and whether extent should be estimated by attention entropy or by a thresholded instance count.
The third is a validity check on the inferred strata. On data with lesion masks or boxes, we compare inferred extent with true lesion area. We also run an oracle version that forms strata from true area. The oracle bounds how much is lost by inferring groups from the model. There is a known circularity risk: a first-pass model that misses a subtle lesion may give it diffuse attention and a large inferred extent. The oracle comparison and a direct correlation analysis quantify this. If the risk is serious, we fall back to strata based on first-pass confidence crossed with extent.
Baselines
The naive reference is attention MIL trained with plain cross-entropy on the same backbone and patch or slice tiling. It shows what local aggregation alone buys.
The strongest existing method on our primary imbalanced benchmark is the recipe from Fair Lung Disease Diagnosis from Chest CT. It combines attention MIL on ConvNeXt, focal loss with label smoothing, weighted oversampling, stratified cross-validation over class and gender, threshold tuning on held-out folds, and a gradient-reversal gender adversary. It reports a mean competition score of 0.685 (average per-gender macro-F1), with squamous cell carcinoma F1 of 0.366. We rerun it ourselves on the same folds, with the same backbone, epochs, and hyperparameter search budget as our method, keeping its gender adversary in both arms so the comparison isolates the objective. Beating it means a higher squamous cell carcinoma F1 with the competition score at or above the reproduced baseline, both averaged over folds, with fold-level variation reported.
For chest X-ray pneumothorax, the closest reported figure uses a different setup: retrieval over global DenseNet features reached AUC 0.891 on pneumothorax versus normal (Searching for Pneumothorax). There is no reported result on Lacuna for pneumothorax classification stratified by lesion area on SIIM-ACR. We therefore reproduce all baselines there under a shared budget.
The robust competitors are three. The first is class-level group robust training in the manner of Class Unbiasing, both with and without its loss-gap penalty. The second is pooled sample-level robust training. The third is instance-weight robust pooling following DRO-DKMIL, implemented with a KL ball over attention weights. We implement it without the Gaussian-process head so compute is comparable, and note this deviation.
Two ablations isolate the key ingredient. One keeps extent strata but drops class decoupling, using a single adversary over all (class, stratum) groups. The other keeps class decoupling but uses a single stratum per class, which reduces to class-level group robust training. The oracle-strata model serves as an upper reference.
Why not just…?
Why not just use focal loss, which already upweights hard examples? This is the most serious threat. Hard examples and small lesions overlap, and focal loss needs no grouping. Two points argue against it. First, the CT evidence shows focal loss plus oversampling still leaves the rare class at F1 0.366 (Fair Lung Disease Diagnosis from Chest CT). Second, per-sample hard-example weighting also upweights mislabeled scans. That matters because chest X-ray labels come from automated report mining with explicit uncertainty handling (CheXpert). Group-level weighting averages label noise within a stratum. The strongest-baseline comparison settles the question. If focal loss matches our method on small-extent sensitivity, the stratification is not worth its complexity, and we will report that.
Why not just apply group robust training over classes, as Class Unbiasing does? Class-level groups cannot see that a class's loss is dominated by its easy, large-lesion members. The subtle members stay underweighted even when the class as a whole is protected. The single-stratum ablation is exactly this alternative. The comparison is decided on the smallest-extent tercile of each class.
Why not just use robust instance pooling from DRO-DKMIL, which already combines MIL with worst-case weights? That set was designed for bags where the positive evidence is not vanishingly sparse. In sparse-lesion scans, we expect its worst case to spread weight onto background and so reward context shortcuts. The diagnostic study measures this directly, by checking whether small-extent positive sensitivity drops relative to plain attention MIL. If instance-level robustness does not hurt, we will fold it into our model rather than argue against it.
Why not just use a higher-resolution global backbone? The global-versus-local evidence already favors local aggregation: 0.975 against 0.962 AUC for pelvic fractures (Weakly Supervised Universal Fracture Detection), plus consistent gains from local pretraining (Unsupervised Local Discrimination for Medical Images). More importantly, a global model provides no extent signal from which to build strata. We include one high-resolution global model as a reference point only.
Proposed Experiments
The primary imbalanced benchmark is the Fair Disease Diagnosis Challenge chest CT dataset (889 volumes; healthy, COVID-19, adenocarcinoma, squamous cell carcinoma), with slices as instances. The primary sparse-lesion benchmark trains on pneumothorax labels from ChestX-ray14 and CheXpert with patch instances. It then evaluates on SIIM-ACR, whose masks give true lesion area for stratified evaluation and for the oracle comparison. A secondary check uses Chest ImaGenome anatomical boxes on MIMIC-CXR findings to test whether inferred extent tracks the size of the anatomical region involved.
The metrics are per-class F1 and macro-F1, the challenge's per-gender competition score, overall AUC, and worst-stratum AUC across true-area terciles. The clinically central metric is sensitivity at 95% specificity on the smallest-area tercile. All results are averaged over the challenge folds or over five seeds, with paired bootstrap intervals.
Three results would decide the question. First, our method must raise smallest-tercile sensitivity and squamous cell carcinoma F1 over the reproduced focal-loss recipe without lowering overall AUC. Second, removing class decoupling must reproduce, in medical data, the rare-class degradation DRAUC reported on long-tailed CIFAR. Third, the inferred-strata model must recover most of the oracle-strata gain; if it does not, extent inference and not the objective is the bottleneck.
A falsifying outcome would be focal loss or class-level group robust training matching us on small lesions. That would mean within-class lesion-extent imbalance is not a separate failure mode worth modeling.
References
- Searching for Pneumothorax in Half a Million Chest X-Ray Images
- DRAUC: An Instance-wise Distributionally Robust AUC Optimization Framework
- Distributionally Robust Optimization for Deep Kernel Multiple Instance Learning
- Class Unbiasing for Generalization in Medical Diagnosis
- Fair Lung Disease Diagnosis from Chest CT via Gender-Adversarial Attention Multiple Instance Learning
- Weakly Supervised Universal Fracture Detection in Pelvic X-Rays
- CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
- Unsupervised Local Discrimination for Medical Images