Background Covariance: Teaching Small Medical Classifiers What to Ignore with a Few Masks

Medical image classifiers trained on a few hundred cases often learn to detect the treatment rather than the disease. A pneumothorax model learns to find the chest drain. A melanoma model learns to find the surgeon's skin marker. The best known fix is spatial supervision, which means drawing a lesion mask on every training image. Masks help a great deal, but they cost expert time on every case. We propose a much cheaper use of the same information. Draw masks on a small fraction of images, measure how the network's features vary across the background regions outside the lesion, and train the classifier to be insensitive to exactly those directions of variation. Training does not sample augmented images. It uses a closed-form loss that averages over infinitely many simulated feature shifts, and it runs in one forward pass.

The mechanism comes from implicit semantic data augmentation (ISDA), a technique that treats augmentation as random shifts of a network's deep features drawn from a Gaussian. It replaces the expected loss over those shifts with an analytic upper bound. ISDA estimates the Gaussian from each class's own feature covariance. We argue that this choice fails in small clinical cohorts, for two reasons. With tens of samples per class, the covariance only spans the few training points it was computed from. It also mixes lesion variation, which the classifier must stay sensitive to, with background variation, which it should ignore. Swapping in a covariance estimated from mask-defined background features gives the classifier the right thing to become invariant to, at the cost of a handful of masks.

Core Claim

In small-sample medical classification with spatial shortcuts, the implicit-augmentation loss should use a background covariance: the shrunk covariance of network features pooled only over non-lesion pixels, pooled across all classes. Class-conditional covariance is the wrong choice. We predict three things on subsampled SIIM-ACR pneumothorax and ISIC melanoma data. With masks on roughly 10–20% of training images, this method should recover most of the robust-AUROC gain that full-mask supervision achieves in Reducing Reliance on Spurious Features in Medical Image Classification with Spatial Specificity. It should beat vanilla ISDA and shrinkage-regularized ISDA at every training size. Its advantage should grow as the number of training images shrinks.

The loss is the standard ISDA surrogate with a different covariance. For a sample with penultimate feature $a$ and label $y$, and classifier weights $w_j$ and biases $b_j$ for each class $j$, it is

$$ \bar{L}(a,y) = -\log \frac{e^{w_y^\top a + b_y}}{\sum_j e^{w_j^\top a + b_j + \frac{\lambda}{2}(w_j - w_y)^\top \Sigma_{bg} (w_j - w_y)}} $$

where $\lambda$ is the augmentation strength. The background covariance is

$$ \Sigma_{bg} = (1-\alpha)\hat{S}_{bg} + \alpha\,\tau I, $$

where $\hat{S}_{bg}$ is the sample covariance of background-pooled features and $\tau I$ is a scaled identity target with $\tau$ its average variance. The shrinkage intensity $\alpha$ is set by the analytic linear-shrinkage formula, so it needs no tuning.

The quadratic term penalizes any classifier weight difference that lies along a direction in which backgrounds vary. The chest-tube direction is such a direction whenever tubes appear in some images and not others. Lesion-appearance directions are left unpenalized.

How This Differs from Prior Work

Three papers supply the method we build on. Implicit Semantic Data Augmentation for Deep Networks introduced the closed-form bound with a running class-conditional covariance. It reported a reduction in ImageNet ResNet-50 top-1 error from 23.0% to 21.9% at 5–7% extra training time. It also flagged the difficulty of tuning the augmentation strength. Better Embedding and More Shots for Few-shot Learning applied ISDA in 1- and 5-shot heads on natural images. It left open how to form covariances when classes have almost no samples. Semantic Similarity Covariance Matrix Shrinkage showed, for financial returns, that shrinking a starved covariance toward an informative target beats uninformative shrinkage. We borrow its analytic shrinkage idea. Our informative signal enters through which pixels define the covariance, not through the shrinkage target.

Two lines motivate the proposal by showing the gap. Imagine by Reasoning (RISDA) lets data-starved classes borrow covariance from related data-rich classes, chosen via confusion matrices. That works on long-tailed benchmarks. In a small binary clinical cohort, however, there are no head classes to borrow from. Borrowed covariances would also carry whatever shortcuts the donor classes contain.

Robustness to Spurious Correlations via Dynamic Knowledge Transfer (DKT) and Boosting Model Resilience via Implicit Adversarial Data Augmentation (Meta-IADA) do target spurious correlations in feature space. Both reach about 92.5% worst-group accuracy on Waterbirds. DKT depends on group structure and a reinforcement-learning policy network. Meta-IADA depends on a clean, balanced meta-set. Neither exists for a 100-case private cohort. Our method has no auxiliary learner and no meta-split, and it needs no group labels. The only side information is a few masks, which clinicians already know how to produce.

The constraint on the design comes from Saab et al. They show that masks break the shortcut: robust AUROC on SIIM-ACR rises from 0.46 with image labels to 0.72 with boxes and 0.82 with masks. They use masks as dense per-image supervision. We test whether masks are needed only as a statistical description of nuisance variation, which would require far fewer of them.

Research Program

The pipeline has four steps.

  1. Train a backbone. We use a compact backbone, ResNet-18. Deeper networks overfit at this scale, according to Unveiling COVID-19 from Chest X-ray with deep learning and MedMNIST v2. It starts from ImageNet weights and is trained with standard augmentation.
  2. Pool background features. For each masked image, we average-pool the final convolutional feature map over positions outside the downsampled lesion mask. This gives a background vector in the same space as the ordinary globally pooled feature. For negative images, the whole image is background.
  3. Estimate and shrink the covariance. We compute the covariance across all masked images and both classes, then shrink it analytically. The estimate is refreshed with a running update as the backbone trains.
  4. Train with the surrogate loss. The loss above replaces cross-entropy.

A cheaper variant that needs no masks at all builds the covariance from negatives only. Its risk is that it misses shortcuts that appear only in positives, and we test it as a separate arm.

The analysis has two parts. First, we check whether the method removes the shortcut for the right reason. We fit a linear probe for tube presence (on ISIC, marker presence) using the annotations that define the robust test split. We then measure how much of that probe direction lies in the top eigenvectors of the background covariance. We compare against the class-conditional covariance ISDA would use.

Second, we check whether ISDA's known sensitivity to augmentation strength is reduced. Our covariance excludes lesion variation, so large $\lambda$ should hurt less. We sweep $\lambda$ across two orders of magnitude and report the full curve.

Baselines

Empirical risk minimization (ERM). This is the do-nothing reference: plain cross-entropy with the same backbone and flips/crops. Saab et al. report its robust AUROC at 0.46 on full SIIM-ACR. On ISIC they report it as worse than random. We rerun ERM on each subsampled training size rather than quoting these numbers, because our N differs.

AugMix. AugMix is the strongest simple augmentation in scarce clinical data. Limitations of Synthetic Data Generation in Specialized Data-Scarce Domains found that no generative approach consistently beat it across five trauma tasks. We run it with the same backbone, schedule, and tuning budget as our method.

Spatial supervision (Saab et al.). This is the strongest existing competitor, since it uses the same side information. We run it in two configurations: with the same subset of masks our method gets, and with masks on all images, which serves as a ceiling. The reported reference points on full SIIM-ACR are robust AUROC 0.82 with segmentation masks and 0.72 with boxes; on ISIC it is 0.73 with masks. Beating this baseline means exceeding its robust AUROC at equal mask budget, averaged over five seeds, with non-overlapping bootstrap intervals. Reaching at least 80% of its full-mask gain over ERM while using ≤20% of the masks would also count as a practical win even without beating the ceiling.

SilverLining. SilverLining is an input-level shortcut filter. It reports a cross-hospital chest X-ray AUC improvement from 0.72 to 0.77. It is a data-first competitor with no mask requirement, and we run it through the same classifier.

ISDA ablations. The ablations that isolate our key ingredient are all ISDA with identical $\lambda$ search and budget. We compare ISDA with its original class-conditional covariance, ISDA with that covariance analytically shrunk, and ISDA with a pooled within-class covariance. If the pooled or shrunk variants match ours, the gain comes from regularization, not from the background-only estimate.

DKT and Meta-IADA. These are reported for context on Waterbirds only. Their published worst-group accuracies are 92.56% (DKT) and 92.5% (Meta-IADA); ERM is about 63% as reported in the DKT paper. Those numbers use full training sets and are not directly comparable to our subsampled regime.

Why not just…?

Why not just use vanilla ISDA? ISDA is the real threat. Its penalty already suppresses classifier weight along directions where positives vary, and positives do vary in tube presence. We predict it falls short for two reasons. With tens of positives, its covariance spans only those few samples. It also penalizes lesion-appearance directions along with the shortcut. The class-covariance ablation settles this directly. If vanilla ISDA matches the background covariance at N ≤ 200, the mask ingredient is unnecessary and we will report that.

Why not just put the few masks into Saab-style spatial supervision? Per-image mask losses give gradient signal only on the masked images. Our covariance turns the same masks into a constraint that acts on every training sample through the shared classifier weights. The equal-mask-budget comparison is the decisive experiment. If Saab et al. with 10% of masks already recovers its full-mask performance, our method adds nothing.

Why not just filter shortcuts at the input, as SilverLining does? Input filtering has to find and remove the shortcut without damaging diagnostic tissue. Chest tubes overlap the pleural region where pneumothorax appears, so removing one risks removing the other. SilverLining is run as a baseline. Its published cross-hospital result is a bar for our method, not proof that our method is needed.

Why not just use generative augmentation to create tube-free positives? Lim et al. found that generators in scarce clinical domains produce canonical, easy images. Classifiers scored 11–45% higher recall on synthetic images than on real ones and gained no real-world robustness. We therefore do not rerun diffusion baselines, and we cite this negative result instead.

Proposed Experiments

Datasets and training sizes. We use SIIM-ACR pneumothorax and ISIC melanoma, subsampled to N ∈ {50, 100, 200, 500} training images with the shortcut correlation preserved. Mask budgets are m ∈ {5%, 10%, 20%, 100%} of training images. Evaluation uses the robust, counter-shortcut test splits of Saab et al., with robust AUROC as the primary metric and overall AUROC as secondary.

Sanity benchmark. Waterbirds is a natural-image check at the same small N, with bird foreground masks treated as the "lesion." There we report worst-group accuracy.

Decisive checks. Three results would decide the question:

  1. At N = 100 and m = 10%, background-covariance ISDA beats vanilla ISDA, shrunk ISDA, and equal-budget Saab supervision on robust AUROC on both medical datasets.
  2. The top eigenvectors of the background covariance capture substantially more of the tube-probe direction than the class covariance does.
  3. The gap over vanilla ISDA widens as N decreases.

Failing the first check at equal mask budget would falsify the core claim. Failing the second while passing the first would mean the method works for a reason we do not understand, and we would say so. Five seeds and bootstrap intervals are reported throughout. No reported result was found on Lacuna for any implicit-augmentation method on SIIM-ACR or ISIC, so all ISDA-family numbers there are new.