How Many Expert Masks Is ROI Masking Worth?

Medical image classifiers trained on whole images often learn the wrong thing. They pick up skin thickness, scanner housing or text overlays instead of the disease. A well-supported fix is region-of-interest (ROI) masking: segment the organ or lesion first, then make the classifier look only there. In lung ultrasound, masking out subcutaneous tissue raised diagnostic AUC from 0.655 to 0.774 (The Role of Pleura and Adipose in Lung Ultrasound AI).

The fix has a hidden cost. Someone has to draw the masks that train the segmenter, and expert drawing time is exactly the resource that medical imaging lacks. Published pipelines get their segmenters cheaply. Some borrow a fully annotated external dataset, as the CT triage work did with LUNA16 (Computed tomography–based COVID–19 triage through a deep neural network using mask–weighted global average pooling). Others assume good masks are simply available (Lesion-Centric Latent Phenotypes from Segmentation Encoders for Breast Ultrasound Interpretability; RGMIM). Nobody has asked the practical question. Suppose a clinic has a fixed number of expert hours. Should some of them go to drawing a few dozen masks, or should all of them go to more diagnosis labels? And when the segmenter is trained on few masks and makes errors, which way of using its mask fails least?

Core Claim

Under a fixed annotation budget, ROI masking is worth paying for only if the mask is used in a way that tolerates segmentation errors.

When a segmenter is trained from tens of masks, hard pixel zeroing passes its errors straight to the classifier. A missed lesion edge simply becomes invisible, and at low mask counts this can erase the benefit of masking or reverse it. Soft pooling in feature space degrades gracefully. This means averaging the classifier's features with weights equal to the segmenter's predicted foreground probability, while keeping a small floor weight on the background.

We predict that soft pooling with a background floor reaches most of the full-mask benefit with a small mask budget. The budget should be smaller still when the segmenter also learns from unlabeled images. If so, there is a measurable crossover cost ratio: the ratio of mask cost to label cost above which masks stop being worth buying.

The deliverable is a set of these budget curves on public benchmarks, together with a simple pooling rule that practitioners can adopt. It is not a new architecture.

Mechanism

A shared pretrained convolutional encoder produces feature map $f(x)$ at location $x$. A light segmentation head predicts foreground probability $p(x)$. We use a convolutional encoder rather than a vision transformer because the mammography benchmark found CNNs beat ViTs and MedSAM2 at roughly 600 labeled images (Benchmarking Transfer Learning for Dense Breast Tissue Segmentation on Small Mammogram Datasets).

The classifier sees the pooled feature

$$z = \frac{\sum_x w(x)\, f(x)}{\sum_x w(x)}, \qquad w(x) = \epsilon + (1-\epsilon)\, p(x),$$

where $\epsilon \in [0,1]$ is the background floor. Setting $\epsilon=1$ recovers ordinary global average pooling. Setting $\epsilon=0$ with $p$ rounded to 0 or 1 recovers hard masking.

This generalizes two earlier pooling rules:

  • the Gaussian-smoothed but deterministic weights of the CT triage paper;
  • the mask-weighted pooling of the breast ultrasound work.

Both of those assume the mask is right.

The segmentation head is trained on $n_m$ expert masks. It also trains on the remaining unlabeled images using multi-scale consistency with an uncertainty correction, borrowed from Uncertainty Rectified Pyramid Consistency. That paper estimates per-voxel uncertainty from disagreement between predictions at different scales, at the cost of one forward pass.

As an optional variant, we shrink $p(x)$ toward 0.5 wherever that uncertainty is high. This variant must earn its place against plain $p(x)$; it is not assumed to help.

The total annotation cost is $C = n_\ell + r\, n_m$. Here $n_\ell$ is the number of diagnosis labels and $r$ is how many diagnosis labels one mask costs. We have no measured value of $r$, so every result is reported as a function of $r$ rather than at one assumed value.

Research Program

Benchmarks. We use benchmarks from the file where both masks and diagnosis labels exist:

  • Breast ultrasound. Train and tune on BUSI and BUS-UCLM; evaluate on BUS-BRA as an external cohort.
  • Dermoscopy. ISIC 2017, segmentation plus melanoma classification.
  • Lung nodules. LIDC-IDRI, nodule contours plus malignancy.

The budget grid. For each benchmark we vary the number of masks over 0, 5, 10, 20, 50, 100 and all. We vary the number of diagnosis labels over a matching grid of fractions. Every cell is trained with five seeds.

Mask-use rules. Each cell is run under these rules:

  • no mask;
  • hard zeroing in pixels, as in Deep echocardiography;
  • binary feature pooling;
  • soft pooling with a range of $\epsilon$;
  • soft pooling with the uncertainty-shrunk $p$;
  • a raw-plus-masked two-branch ensemble.

Outputs. The program produces three things:

  • iso-cost curves, showing the best AUC reachable at each total cost $C$ for each value of $r$;
  • the crossover $r^*$ above which zero masks is optimal;
  • a sensitivity analysis showing how $r^*$ moves with $\epsilon$.

Shortcut stress test. In-distribution AUC can hide shortcut learning. So we add a controlled test: in training only, a synthetic marker correlated with the label is pasted outside the ROI. We measure the AUC drop on clean test images. This isolates the reason masking exists, separately from its effect on ordinary accuracy.

Baselines

Naive reference. An ImageNet-initialized EfficientNet trained end to end on whole images, with standard augmentation (flips, crops, rotations, CutMix). It gets the same number of diagnosis labels and the same tuning budget as every masked variant, and it spends zero on masks. Masking must beat it at equal total cost $C$, not just at equal label count. Otherwise masks are not worth buying.

Strongest existing ROI method, at full mask supervision. This is the mask-weighted latent pooling of Lesion-Centric Latent Phenotypes, which reports a malignancy-probing AUC of 0.982 on the breast ultrasound cohorts. We reproduce it with segmenters trained on all masks. It serves as the ceiling that low-budget variants are measured against. Our claim is supported if floor pooling reaches 90% of the gap between the naive reference and this ceiling using 20 or fewer masks.

On LIDC-IDRI, the matching reference is Probabilistic Radiomics, at 0.9566 AUC and 91.52% accuracy on its high-ambiguity split. Our reproduction will use its split.

Strongest zero-annotation competitor. This is SAMAug-C from Boosting Medical Image Classification with Segmentation Foundation Model. It uses zero-shot Segment Anything masks, adds a brightness offset inside the masked region, and ensembles with a raw-image branch. It reports 86.67% accuracy on ISIC 2017, against 84.45% for the raw baseline. It is the most dangerous competitor because its masks cost nothing.

To keep the comparison fair, SAMAug-C uses the same backbone, labels and number of tuning runs. We report it both as published and with our floor pooling applied to Segment Anything probabilities. Expert masks are worth their price only if some $n_m > 0$ beats this point at matched $C$ for a plausible $r$.

Ablations. Three ablations isolate the key ingredient:

  • $\epsilon=0$ against tuned $\epsilon$, which tests the background floor;
  • binary $p$ against soft $p$, which tests whether keeping the probabilities matters;
  • with and without unlabeled consistency training of the segmentation head, which tests how much semi-supervision lowers the required mask count.

Why not just…?

Why not just augment heavily and skip masks? Augmentation does not remove a background feature that reliably predicts the label; it only jitters it. The lung ultrasound study found Grad-CAM attention stayed on subcutaneous tissue until that tissue was masked, and its unmasked AUC was 0.655. MisMatch (MisMatch) reports that image-level augmentations regularize poorly in low-label regimes, though that finding concerns segmentation, not classification. Our naive baseline and the synthetic-marker test settle the question directly on our benchmarks.

Why not just use zero-shot Segment Anything masks, which cost nothing? This is a real threat. On high-contrast lesions such as dermoscopy, Segment Anything masks may already be good enough, and the honest result could be $n_m^*=0$. Two pieces of evidence point the other way on harder targets. The mammography benchmark found MedSAM2 weak on diffuse, low-contrast tissue. Plain mask overlay with Segment Anything lowered ISIC 2017 accuracy to 82.53%. The iso-cost curves report a separate answer for each benchmark. We expect Segment Anything to win on ISIC but not on LIDC-IDRI or breast ultrasound, and we will report whichever outcome we find.

Why not just spend the whole budget on more diagnosis labels? That is the $n_m=0$ column of our grid, so it is measured rather than argued against. The proposal is falsified if that column dominates for every $r$ at or above 1.

Why not just use the segmenter's probability map without the uncertainty correction? Plain $p(x)$ with a floor is likely to be as good. Because of that, plain soft pooling is the main method, and the uncertainty shrinkage is an ablation that is kept only if it improves external-cohort AUC across seeds.

Proposed Experiments and Decisive Checks

The first decisive check is the crossing pattern of the curves. Hard pixel masking should fall below the no-mask baseline at 5 to 10 masks on at least two of the three benchmarks, while floor pooling stays above it. If both rules behave the same, error tolerance is not the issue and the core claim fails.

The second check is external transfer. On BUS-BRA, a low-budget floor-pooled model should lose less AUC relative to the internal test set than the whole-image baseline does. That is the outcome we would expect if masking removes site-specific background cues.

The third check is the shortcut stress test. If masking matters for robustness, floor pooling with a small mask budget should cut the clean-test AUC drop substantially relative to the whole-image model.

All runs use one GPU-scale budget per cell, reported alongside the results. The mammography benchmark's under-8-hour single-H100 pipeline gives a reference point for this budget.