Medical image classifiers often learn the wrong thing. A lung ultrasound model can learn to grade COVID-19 severity by measuring the patient's fat layer instead of the lung surface. A breast ultrasound model can learn which scanner produced the image instead of what the lesion looks like. The usual fix is to segment the region of interest, meaning the anatomically relevant part of the image, and then crop or zero out everything else. This works, but it is blunt. It assumes we already know which surrounding tissue is noise, and it throws away tissue around a lesion that may carry real diagnostic signal. A separate line of work learns data augmentation online: a small "augmenter" network chooses image transformations while the classifier trains, and is tuned so that the classifier does well on held-out data. Those augmenters apply one transformation to the whole image.
We propose to join the two by learning a separate augmentation policy for each anatomical zone of a segmented image. The policy decides, per zone, how strongly to distort color and geometry and how often to softly erase that zone. Our central point is that this only helps if the held-out data used to tune the augmenter is built so that the shortcut does not hold there. If validation data comes from the same distribution as training data, the augmenter is rewarded for keeping the shortcut. If validation data is balanced with respect to a nuisance variable we can measure, the augmenter is pushed to destroy whichever zones carry the shortcut and to preserve the zones that carry pathology. The learned per-zone erase rates then give a data-driven answer to a question that hard masking answers by fiat: how much should this model trust each part of the image?
Core Claim
Our claim is that zone-conditioned online augmentation, tuned against a nuisance-balanced validation split, can do two things. It can match hard anatomical masking where the background is purely a shortcut, as with subcutaneous fat in lung ultrasound. It can beat hard masking where the surrounding tissue is informative, as with perilesional breast parenchyma. It should also degrade more gracefully than hard masking when the segmentation itself is wrong. We expect the advantage over hand-designed "scramble the background" augmentation to come from two sources. The policy is learned per domain and per zone, and it can choose to keep context rather than always destroy it.
We separate this claim into two ingredients and ablate each. The first is anatomical zoning rather than a uniform grid. The second is the nuisance-balanced outer objective rather than an ordinary validation split. If the second ingredient carries all the gain and the first carries none, we will report that. A cheaper method would then follow.
Relation to Prior Work
Online policy learning supplies our machinery. Learning Data Augmentation with Online Bilevel Optimization for Image Classification and its histopathology extension Automatic Data Augmentation Learning using Bilevel Optimization for Histopathological Images train a small network that outputs a global affine matrix and color shift. They update it each mini-batch through truncated backpropagation, which approximates the gradient of validation loss with respect to the augmenter by unrolling only a few classifier steps. We reuse this optimizer, but output one parameter vector per zone instead of one per image. Automatic Data Augmentation for 3D Medical Image Segmentation (ASNG) shows that online search beats nnUNet's hand-designed recipe on the Medical Segmentation Decathlon. It also introduces search over magnitude ranges rather than fixed values. Like the Mounsaveng work, it transforms the whole volume and optimizes segmentation accuracy, not a downstream diagnosis.
The closest prior work is Local Patch AutoAugment With Multi-Agent Collaboration (PAA). It assigns a reinforcement-learning agent to each cell of a uniform grid and rewards agents for raising training loss. It reports that agents learn to treat foreground gently and background harshly. PAA motivates our proposal, because that emergent behavior is exactly the differential treatment we want. It also constrains our design in two ways. First, grid cells cut across curved anatomical boundaries such as the pleural line. Second, a reward for raising training loss carries no notion of a shortcut. It will happily distort subtle lesion margins, the "Goldilocks" risk the histopathology work warns about. We differ on both counts. Our zones come from anatomical masks, and our signal is validation loss on nuisance-balanced data, not training difficulty. Methods built around curricula, such as AdaAugment: A Tuning-Free and Adaptive Approach to Enhance Data Augmentation, and adversarial methods such as Adversarial AutoAugment share PAA's whole-image or difficulty-driven framing.
On the region-of-interest side, three papers motivate the proposal by showing that zone-level treatment matters, and also set the bars we must clear. The Role of Pleura and Adipose in Lung Ultrasound AI finds that models exploit subcutaneous fat thickness, and that zeroing that zone raises severity AUC. Lesion-Centric Latent Phenotypes from Segmentation Encoders for Breast Ultrasound Interpretability pools encoder features only inside the predicted lesion mask and calibrates on the target scanner. Region of Interest Detection in Dermoscopic Images for Natural Data-augmentation crops detected lesions at fixed scales and angles. All three make a fixed, hand-chosen decision about the background, and all three inherit any errors in the mask. Our method replaces the fixed decision with a learned one, and it uses soft, feathered zone boundaries so that a slightly wrong mask causes a slightly wrong augmentation rather than a lost lesion. Deep echocardiography is relevant mainly as evidence that masking helps in low-data regimes. This matters because the histopathology bilevel work reports that its augmenter becomes unstable below half of the BACH training set.
Method
A frozen or jointly trained segmenter partitions each image $x$ into $K$ soft zone maps $m_1,\dots,m_K$ that sum to one at every pixel. In breast ultrasound the zones are lesion, a band of a few millimetres around the lesion margin, and the remaining parenchyma. In lung ultrasound they are subcutaneous tissue, the pleural line, and the region below it. In dermoscopy they are lesion, a ring around it, and the remaining skin. For each zone the augmenter outputs a small parameter vector $\theta_k$ containing four quantities: a photometric strength, a local affine magnitude range, a relaxed erase probability $p_k$, and a noise-texture strength. The augmented image is a mask-weighted blend of per-zone transformations:
$$\tilde{x} = \sum_{k=1}^{K} m_k \odot \big[(1-e_k)\,T_{\theta_k}(x) + e_k\, \varepsilon\big],$$
Here $T_{\theta_k}$ is the differentiable transform for zone $k$, $\varepsilon$ is a texture-matched noise fill, and $e_k$ is a relaxed Bernoulli sample with mean $p_k$. Erasing therefore remains differentiable, which addresses the limitation that truncated backpropagation cannot handle Cutout-style operations. With roughly ten parameters per zone, the augmenter is small enough that we expect it to stay stable at clinical dataset sizes. We test this directly.
The outer objective is classifier loss on a validation split $\mathcal{V}_{\text{bal}}$ that is reweighted so the label is independent of a measurable nuisance variable $z$:
$$\min_{\phi}\; \mathbb{E}_{(x,y,z)\sim \mathcal{V}_{\text{bal}}}\, \ell\big(f_{w^*(\phi)}(x), y\big), \quad w^*(\phi) \approx \text{a few SGD steps on } \ell(f_w(\tilde{x}_\phi), y).$$
In this objective, $\phi$ denotes the augmenter weights, $f_w$ the classifier, and $\ell$ cross-entropy. The nuisance variable is chosen per domain, and it can often be read off the segmentation itself. In lung ultrasound it is subcutaneous thickness, measured from the zone mask. In breast ultrasound it is the source scanner, taken from the source dataset. In dermoscopy it is the presence of ruler, ink, or hair artifacts.
Research Program
The first stage builds the zone-wise augmenter on top of the published truncated-backpropagation optimizer and verifies one property: with all zone parameters tied, it reproduces the global augmenter. The second stage applies it in three domains whose context properties differ. In lung ultrasound we believe the background is a shortcut. In breast ultrasound the surrounding tissue is plausibly informative. Dermoscopy is mixed. The third stage analyzes what the policy learned, chiefly the per-zone erase rates and magnitudes over the course of training. We ask whether the policy rediscovers subcutaneous masking, and whether it keeps the perilesional band in breast ultrasound. The fourth stage stress-tests robustness to mask errors and to data scarcity.
Baselines
The naive reference is the same classifier trained on full images with the standard hand-designed augmentation for each domain. It shows how much shortcut learning occurs in the first place. In lung ultrasound, Gare et al. report an unmasked AUC of 0.6553.
The strongest masking competitor is static hard masking of the presumed shortcut zone. For lung ultrasound we follow The Role of Pleura and Adipose in Lung Ultrasound AI, which reports AUC 0.7742 with subcutaneous tissue masked and 0.7642 with the pleura also straightened. For breast ultrasound we use mask-weighted pooling from Lesion-Centric Latent Phenotypes, which reports external AUC 0.982 against 0.852 for a standard CNN. That figure includes calibration on the target domain. We will therefore run mask-weighted pooling both with and without calibration, and give our method the same calibration step when comparing against the calibrated version. For dermoscopy we use the fixed multi-scale crops of Region of Interest Detection in Dermoscopic Images, which report a Jaccard index of 82.3% against 77.1% for DeepLabV3+ alone. That is a segmentation metric, so our dermoscopy comparisons for classification will be our own reruns. All masking baselines use the same segmenter and backbone as our method. To beat masking, our method must match it in lung ultrasound and exceed it, on worst-nuisance-group AUC, in breast ultrasound.
The strongest learned-augmentation competitors are three: the global online bilevel augmenter of Mounsaveng et al., which reports 88.90% against 83.30% on BACH; PAA on a grid with its loss-increase reward; and RandAugment. All are given the same number of training epochs and the same hyperparameter-search budget, measured in GPU hours, as our method. For fairness, the global augmenter is also run with the nuisance-balanced validation split.
Three ablations isolate the key ingredients. The first keeps zone-wise policies but uses an ordinary validation split, which isolates the balanced objective. The second keeps the balanced objective but replaces anatomical zones with a uniform grid of matched cell count, which isolates anatomy. The third fixes the erase probabilities at hand-chosen values while still learning the remaining parameters, which isolates learned context retention.
Why not just…?
Why not just hard-mask the background, as Gare et al. do? Where the background is purely a shortcut, this may be enough, and we expect our method only to tie it in lung ultrasound. Hard masking fails in two situations: when context is informative, and when the mask is wrong. The breast ultrasound experiments test the first, through the perilesional band, which hard masking discards. Controlled mask corruption tests the second, through dilation, erosion, and boundary jitter. If hard masking matches us under both conditions, our proposal reduces to a costlier way of masking, and we will say so.
Why not just hand-design differential augmentation, with strong scrambling outside the lesion and mild changes inside? This is the real threat. PAA's agents converge to roughly this behavior. We include it as a baseline with per-zone magnitudes tuned by grid search under the same compute budget. Our method can win only where the right per-zone treatment is not obvious beforehand: when part of the context should be kept, or when different domains need different treatment. The learned erase rates for the perilesional band make this checkable. If they converge to "erase everything outside the lesion" in every domain, the tuned heuristic is the better recommendation.
Why not just reweight the training loss by nuisance group, without any augmentation? Group-balanced training uses the same nuisance labels that our validation split uses. If those labels alone remove the shortcut, augmentation adds nothing. We run group-balanced training, with and without static masking, as an additional baseline. The zone-swap test described below shows whether reweighting actually removes reliance on the shortcut zone, or merely balances accuracy while the model still reads the fat layer.
Why not just use a uniform grid policy such as PAA? A grid cell straddling the pleural line mixes fat, pleura, and artifact, so it cannot treat them differently. The grid ablation, matched in cell count and objective, measures how much this matters in practice.
Why not just generate synthetic images with a generative model? The file gives no comparative evidence on these datasets. The semi-supervised generative model of Deep echocardiography helps through unlabeled data rather than through shortcut removal, and generative models trained on a few hundred images risk altering lesion margins. We treat this as out of scope rather than claim we can rule it out.
Proposed Experiments
For breast ultrasound, we train on BUSI and BUS-UCLM with source scanner as the nuisance and evaluate externally on BUS-BRA, reporting overall AUC and worst-source AUC. For lung ultrasound, we use the high-frequency linear-probe cohort of Gare et al., if access can be obtained, with patient-level splits and subcutaneous-thickness strata. For dermoscopy, we use ISIC 2017 with artifact-presence strata. Finally, we use BACH in reduced-data subsets of 25%, 50%, and 100% to test whether zoning stabilizes the bilevel optimizer where the global version is reported to degrade.
Three checks decide the outcome. The first is a zone-swap counterfactual: we transplant the subcutaneous zone, or the far-background zone, between images with different labels and measure how much the prediction changes. A shortcut-free model should barely move, while the unmasked baseline should move substantially. The second is the learned policy itself: whether its erase rate is high for subcutaneous tissue and moderate or low for the breast perilesional band. This is a prediction that could fail, and failure would be informative. The third is mask robustness: the drop in AUC as mask error grows, compared with hard masking and mask-weighted pooling. The proposal succeeds if it beats the tuned differential heuristic and group reweighting on worst-group AUC in at least the breast domain, ties hard masking in lung ultrasound, and degrades more slowly than masking under corrupted masks. If the ordinary-validation ablation performs as well as the full method, our central claim about the balanced objective is wrong.
References
- Learning Data Augmentation with Online Bilevel Optimization for Image Classification
- Automatic Data Augmentation Learning using Bilevel Optimization for Histopathological Images
- Automatic Data Augmentation for 3D Medical Image Segmentation
- Local Patch AutoAugment with Multi-Agent Collaboration
- AdaAugment: A Tuning-Free and Adaptive Approach to Enhance Data Augmentation
- Adversarial AutoAugment
- The Role of Pleura and Adipose in Lung Ultrasound AI
- Lesion-Centric Latent Phenotypes from Segmentation Encoders for Breast Ultrasound Interpretability
- Region of Interest Detection in Dermoscopic Images for Natural Data-augmentation
- Deep echocardiography: data-efficient supervised and semi-supervised deep learning towards automated diagnosis of cardiac disease