A common way to stop diagnostic models from cheating is anatomical masking. One network first outlines the organ or lesion, and a second network makes the diagnosis using only what lies inside that outline. This works. In lung ultrasound, masking out subcutaneous fat raised severity-classification AUC from 0.6553 to 0.7742, because the unmasked model had learned body habitus rather than lung disease (The Role of Pleura and Adipose in Lung Ultrasound AI). The catch is that the outline comes from a segmentation network, and segmentation networks fail on scanners they have not seen. On MRI from unseen scanners, AMOS reports a drop of over 20% (AMOS). A hard mask drawn by a failing segmenter cuts off pathology or passes in junk, so the diagnostic model inherits every upstream error.
We propose to make the mask's margin depend on how much the segmenter can be trusted on each scan. Every case gets a soft mask whose border is a band of width $r$ around the predicted region. The width is set by a cheap per-case estimate of boundary instability: how much the predicted outline moves when the input is jittered. The diagnostic model is trained on masks whose margins are randomly widened and shrunk, so it can handle whatever margin it receives at test time. In two sentences: masking protects against shortcuts only while the outline is right. So we widen the window exactly when the segmenter's outline becomes unstable under shift, and we test whether that per-case adaptivity beats simply training with a sloppy, fixed-width mask.
Core Claim
Under scanner or sequence shift, the main way masked diagnostic pipelines fail is that the mask truncates the target region, not that the classifier misreads the image. A margin that scales with perturbation-measured boundary instability should recover most of the diagnostic accuracy lost to that truncation. It should do this while keeping most of the shortcut protection that full masking gives, and a single fixed margin cannot hold both properties across in-domain and shifted cases.
The claim can fail in an informative way. If a fixed margin, tuned once and paired with margin-randomized training, matches the adaptive version, then the lesson is that masked pipelines need training on imperfect masks, not uncertainty estimation. We would report that result as the finding.
Relation to Prior Work
Three lines of work motivate the problem. They all establish that masking or isolating the region of interest helps diagnosis, and all assume that a correct mask is available. Gare et al. mask lung ultrasound with hand-built pleural-line detection on 93 patients from a single center (The Role of Pleura and Adipose in Lung Ultrasound AI). Deep echocardiography zeroes non-cardiac pixels to improve label efficiency (Deep echocardiography). Capsule networks on segmented tumors beat whole-brain inputs (Brain Tumor Type Classification via Capsule Networks).
Two architecture-level variants push the mask inside the network rather than applying it to the input, but they still use hard masks and are evaluated within one domain:
- Mask-weighted pooling. A mask-weighted global average pooling layer aggregates features only over segmented lung for COVID triage (CT-based COVID-19 triage with mask-weighted GAP).
- Lesion-restricted features. DenseSharp+ crops latent feature clouds to segmented nodules and reaches 0.9566 AUC on LIDC-IDRI (Probabilistic Radiomics).
Our mechanism is designed to drop into either variant. We differ from all of them by evaluating under shift and by making the mask's extent a function of segmenter reliability.
Work on domain shift supplies the uncertainty signal and constrains the design. Ouyang et al. (Improved post-hoc probability calibration for out-of-domain MRI segmentation) show that a segmentation's instability under intensity and contrast perturbation is informative about out-of-domain errors. They use it, together with a shape prior from a denoising autoencoder, to set per-pixel temperatures. They also report that the shape prior fails for irregular structures such as tumors. We therefore reuse only the perturbation-susceptibility signal and drop the shape prior, so the method applies to lesions as well as organs.
Two training-time augmentation methods attack shift by making the segmenter itself more robust, and both treat the whole image globally with no notion of a region of interest:
- StyCona decomposes images with singular value decomposition and perturbs the style and content parts separately (Style Content Decomposition-based Data Augmentation).
- FreeSDG mixes frequency bands and adds a self-supervised reconstruction task (Frequency-mixed Single-source Domain Generalization).
OptTTA adapts input augmentations at test time to a frozen segmenter. It costs many gradient steps per scan and also reports improvements only in segmentation Dice (OptTTA). These methods are complementary competitors: a more robust segmenter shrinks the problem we address. We will test whether adaptive margins still help on top of the best of them.
Research Program
The pipeline has three parts.
Segmenter. Any segmenter works. We use a well-tuned U-Net, because AMOS found a classic 3D U-Net at 89.04% mean Dice beating Swin-UNETR at 86.32% on CT.
Boundary-instability score. For each test scan we run $K$ (around 8) forward passes under random intensity, contrast, and noise perturbations of the kind Ouyang et al. use. We then compute $u$, the mean distance between each perturbed predicted boundary and the unperturbed one, normalized by the size of the predicted region. This costs $K$ forward passes and no backward passes, which is far cheaper than OptTTA.
Soft mask. The soft mask is
$$m_i = \sigma\!\left(\frac{r(u) - d_i}{\tau}\right), \qquad r(u) = r_0 + \alpha u,$$
where:
- $d_i$ is the signed distance of pixel $i$ from the predicted boundary, positive outside the region;
- $\sigma$ is the logistic function;
- $\tau$ sets how soft the edge is;
- $r_0$ is the in-domain margin;
- $\alpha$ converts instability into extra margin.
The mask multiplies either the input image or the pooling weights, depending on the downstream architecture.
Fitting and training the classifier. The two constants $r_0$ and $\alpha$ are fit on held-out source-domain cases that receive synthetic artifacts. We use TorchIO-style simulations as in Ouyang et al., so no target-domain data is touched. During classifier training, each case's margin is drawn at random over a range of $r$ values, and its masks are deliberately corrupted by erosion, dilation, and the segmenter's own errors on augmented images. This exposes the classifier to the kind of imperfect masks it will see under shift.
Mechanism analysis. Alongside accuracy, we test the mechanism directly. We measure what fraction of the ground-truth target region each mask truncates, and how strongly truncation predicts diagnostic error under shift. The core claim requires this relationship to be strong, and requires $u$ to rank cases by truncation reasonably well.
Baselines
Unmasked classifier (naive reference). The same diagnostic network is trained on whole images with standard augmentation and the same compute and tuning budget. It tells us whether masking helps at all under shift. Masking could hurt if truncation dominates.
Hard masking (prior practice). We use the predicted binary mask applied to the input, and separately mask-weighted pooling as in the COVID triage work. The DenseSharp+-style lesion-restricted head serves as the LIDC variant. This is the closest existing practice. It reports 0.9566 AUC in-domain on LIDC-IDRI (Probabilistic Radiomics) and a 0.7742 AUC masking gain in lung ultrasound (The Role of Pleura and Adipose). We found no reported result on Lacuna for any masked diagnostic pipeline under cross-scanner shift. We therefore re-run these baselines ourselves on identical splits.
Tuned fixed margin with margin-randomized training (key ablation). This baseline is the proposed system with $\alpha = 0$ and $r_0$ chosen on the same held-out synthetic-artifact cases. It is the strongest and most dangerous competitor, and it isolates the single ingredient we claim matters: per-case adaptivity. A second ablation keeps adaptive margins but trains without margin randomization.
Soft probability mask. The segmenter's softmax output is used directly as the mask. This tests whether ordinary probabilities already widen when they should.
Strongest segmenter-side method stacked under every variant. The segmenter is trained with StyCona augmentation, the best single-source result in the file at 73.39% Dice from bSSFP to LGE and 81.59% from LGE to bSSFP (StyCona). As a frozen-model alternative, we also run OptTTA, which reports a 22.5% average Dice improvement over the unadapted model on M&Ms (OptTTA).
Fairness and the bar for success. All variants share the segmenter, the classifier architecture, the hyperparameter search budget, and the data. Beating the baselines means two things. First, adaptive margins must exceed the tuned fixed margin in shifted-domain macro-AUC by more than the seed-to-seed spread, using five seeds and paired bootstrap tests. Second, they must stay within that spread of hard masking on the shortcut-probe test described below.
Why not just…?
Why not just make the segmenter robust and keep hard masks? If StyCona or OptTTA restores segmentation quality, truncation largely disappears and adaptive margins are moot. This is a real threat. StyCona still leaves bSSFP-to-LGE Dice at 73.39%, which suggests substantial residual boundary error, but Dice alone does not tell us how much of the diagnostically relevant region gets cut off. The stacked experiment settles the question. If adaptive margins add nothing once StyCona is used, we report that segmenter robustness is sufficient.
Why not just pad every mask by a fixed amount? This is the simplest fix and plausibly works. A margin wide enough for shifted cases readmits background in in-domain cases, and background is where the body-fat shortcut in lung ultrasound lived. The fixed-margin ablation, tuned on the same data, combined with the shortcut probe, measures whether that tradeoff is real. If no tradeoff appears, the fixed margin wins and we will say so.
Why not just drop masking and augment harder? Gare et al. show that unmasked training learns habitus shortcuts that ordinary training does not break. However, their augmentation was generic and their data came from one center. The unmasked baseline trained with StyCona augmentation tests directly whether strong style augmentation removes the need for masking.
Why not just use calibrated softmax probabilities as the mask? Under shift, segmenters are confidently wrong. Ouyang et al. report an expected calibration error of 16.78% for local temperature scaling from bSSFP to LGE, improving only to 15.37% with their method. Probability masks may therefore stay sharp exactly when they should widen. The soft-probability baseline checks this.
Proposed Experiments
Primary testbed: cardiac MRI. We train the segmenter and the diagnostic classifier on ACDC, which carries per-subject diagnosis groups and on which CASCADE reports 91.63% Dice in-domain (Cascaded Attention Decoding). We evaluate under three shifts:
- simulated acquisition artifacts in the style of Ouyang et al.;
- the bSSFP-to-LGE sequence shift of MS-CMR;
- multi-vendor M&Ms scans.
The M&Ms evaluation depends on per-subject pathology labels whose coverage of the ACDC diagnosis classes we must verify. If coverage is partial, we restrict evaluation to the shared classes.
Second testbed: lesions. We use LIDC-IDRI nodule malignancy, with shift produced by dose and noise simulation and by StyCona-style style perturbations. This tests irregular lesions, the setting where shape priors break.
Metrics. For segmentation we report Dice. For the mask we report the truncated fraction of the target region. For diagnosis we report macro-AUC, accuracy, and expected calibration error. We also report a risk–coverage curve that uses $u$ to defer cases.
Decisive check 1: mechanism. In the shifted domains, truncation must explain a large share of the diagnostic error, and $u$ must rank-correlate with truncation. If either fails, the premise fails.
Decisive check 2: shortcut probe. We plant a synthetic background marker outside the anatomy that correlates with the label during training and is decorrelated at test time. Unmasked models should collapse on this test, and hard masks should be immune. The proposal must sit near hard masking in-domain while gaining over it under shift, whereas a single wide fixed margin cannot do both.
Decisive check 3: added value over a robust segmenter. All comparisons are repeated with the StyCona-trained segmenter underneath. This determines whether adaptive margins add value beyond simply making the segmenter more robust.
References
- The Role of Pleura and Adipose in Lung Ultrasound AI
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Deep echocardiography: data-efficient supervised and semi-supervised deep learning towards automated diagnosis of cardiac disease
- Brain Tumor Type Classification via Capsule Networks
- Computed tomography–based COVID–19 triage through a deep neural network using mask–weighted global average pooling
- Probabilistic Radiomics: Ambiguous Diagnosis with Controllable Shape Analysis
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- Style Content Decomposition-based Data Augmentation for Domain Generalizable Medical Image Segmentation
- Frequency-Mixed Single-Source Domain Generalization for Medical Image Segmentation
- OptTTA: Learnable Test-Time Augmentation for Source-Free Medical Image Segmentation Under Domain Shift
- Medical Image Segmentation via Cascaded Attention Decoding