When a deep network is trained on a few hundred medical images, it often learns the wrong thing. It recognizes which hospital or scanner produced an image, or reads a text tag in the corner, instead of finding the disease. Adding attention, a learned weighting that tells the network which image regions to look at, should help in principle. In practice it does not. On skin-lesion data, attention layers placed on top of convolutional networks gave no significant gain, and they put their focus spikes on background corners rather than lesion structures (When Medical Imaging Met Self-Attention). The common fix is to segment the organ with a separate model and crop everything else away before classification (Unveiling COVID-19 from Chest X-ray with deep learning). Cropping is brittle: any region the segmenter misses is gone for good, along with border context that clinicians actually use.
We propose a middle path taken from residual attention. A segmentation mask is not used to cut the image. It is added as a fixed bias to the attention scores before the softmax, the step that turns raw scores into weights. Attention therefore starts out anchored on the anatomy, and the network only has to learn deviations from that anchor. Background tokens become expensive to attend to but are never erased, so the model can recover from a wrong mask. The idea is testable in a sharp way. If this design matches hard cropping at suppressing shortcuts, and degrades much more gracefully when the mask is wrong, it is worth having. If it does not, cropping wins and the result is still informative.
Core Claim
The prior-as-bias design comes from ResMatch, which adds classical descriptor similarity and relative geometry directly into the cross- and self-attention scores of a feature matcher, so the network learns only the leftover correction. ResMatch was built for large-scale matching (YFCC100M, ScanNet). We move it to small-cohort medical classification, where the prior is an anatomical mask and its job is to block shortcuts rather than speed up matching. For a token $j$ in a convolutional feature map, let $p_j \in [0,1]$ be the fraction of that token's area covered by the downsampled organ or lesion mask. The attention logit between query $i$ and key $j$ becomes
$$a_{ij} = \frac{q_i^\top k_j}{\sqrt{d}} + \beta_h \log(p_j + \epsilon) + \text{Prev}_{ij},$$
where $q_i, k_j$ are query and key vectors of dimension $d$, $\beta_h \ge 0$ is a learned strength for head $h$, $\epsilon$ sets the maximum penalty on fully off-mask tokens, and $\text{Prev}_{ij}$ is the previous layer's pre-softmax logit. That last term is the zero-parameter score carry-over of RealFormer, included so the prior persists across layers instead of being washed out. As $\beta_h \to \infty$ this reduces to hard cropping. At $\beta_h = 0$ it is unconstrained attention, the setting that failed in the skin-lesion study.
We deliberately do not use the classic residual attention mask $H(x) = (1 + M(x)) \odot F(x)$ from the Residual Attention Network with $M$ set to the anatomical prior. Because of its identity term, that formula can at most double on-mask features and leaves background features fully intact. It preserves gradients well, which is why it trains very deep networks. It cannot suppress a corner tag. We expect it to fail at shortcut suppression and include it to show that failure. The claim is narrower: a log-prior logit bias, which competes with background tokens inside the softmax, gives a tunable trade-off between shortcut suppression and tolerance to mask errors. Neither cropping nor multiplicative residual masks provide that trade-off.
Research Program
The backbone is ImageNet-pretrained ResNet-18, because convolutional networks beat attention models in this regime (When Medical Imaging Met Self-Attention) and ResNet-18 beat ResNet-50 on small chest X-ray (CXR) data (Unveiling COVID-19). Self-attention blocks are added after stages 3 and 4 through residual connections, in the embedded-local-attention style of the skin-lesion study, with the bias above applied in every block. Priors come from segmentation models trained independently of the classification labels. For chest X-rays this is the U-Net lung segmenter in the EIDOSLAB codebase. For dermoscopy it is a lesion segmenter; the file lists no ready-made lesion masks, so obtaining these is a dependency we would have to resolve.
The program has three parts. First, we measure whether the bias suppresses shortcuts as well as cropping does. Second, we deliberately corrupt the masks to test whether the soft prior recovers where cropping cannot. Third, we sweep $\beta$ to map the trade-off and check whether learned per-head $\beta_h$ settles near its useful region. We also track a diagnostic that ties both directions together: the share of attention mass falling off-mask, and whether corner "register" spikes persist.
Baselines
The naive reference point is plain ResNet-18 with pretraining on the RSNA Pneumonia Detection Challenge data, run on uncropped images. This is the model known to cluster CXRs by source repository. It tells us how much shortcut there is to remove.
The strongest existing method is the cropping pipeline of Unveiling COVID-19: histogram equalization, U-Net lung masking, then a ResNet-18 pretrained on RSNA. On CORDA (447 CXRs) it reports sensitivity 0.54 and specificity 0.80. We will rerun it from the released code with the same splits, the same segmenter weights, and the same pretraining. Every method gets an identical hyperparameter search budget, selected on a validation split from the source domain only. On clean masks, beating this baseline means matching it within confidence intervals on CORDA and on the synthetic-shortcut test described below. Under corrupted masks, it means a significantly higher area under the ROC curve (AUROC).
For dermoscopy, the reference is ResNet-18 and EfficientNet-B0 trained on ISIC, with reported balanced accuracies of about 73.9% and 75.4%. Their attention variants reached 72.1% and 76.6%, the latter gain not significant (When Medical Imaging Met Self-Attention). The file gives no PAD-UFES-20 out-of-distribution numbers for these models (no reported result found on Lacuna for PAD-UFES-20 attention baselines), so we will rerun them ourselves.
Ablations isolate the key ingredient. These are the same attention blocks with $\beta = 0$; the multiplicative $(1 + M_{\text{prior}})$ mask; the prior fed in as an extra input channel; the bias without the RealFormer carry-over; and fixed versus learned $\beta_h$.
Why not just…?
Why not just crop with the segmenter? This is the real threat. On clean masks, cropping may match or beat us, since COVID evidence lies mostly inside the lungs. Our case rests on two points. A cropped image cannot recover lesion tissue the segmenter missed. Border context also matters in dermoscopy, where the "B" (border) criterion lives at the lesion edge. The mask-corruption experiment settles this. If cropping stays equal at every corruption level, we drop the claim.
Why not just use a plain CNN? Plain CNNs do beat naive attention here (When Medical Imaging Met Self-Attention), but uncropped they still learn scanner and tag shortcuts (Unveiling COVID-19). The synthetic-shortcut test measures how much of that our blocks remove relative to the CNN they sit on.
Why not just concatenate the mask as an extra input channel, or penalize attention outside the mask with a loss? The channel version lets the network ignore the mask, and the loss version needs a weight to tune. Both are run as ablations under the same budget. If the off-mask attention penalty matches the logit bias, the contribution reduces to "constrain attention to anatomy," and we will report it that way.
Proposed Experiments
The first decisive check is a controlled shortcut. In training, we stamp a small marker in an image corner that correlates with the label at 90%, then remove it or flip the correlation at test time. We measure the AUROC drop and the attention mass on the marker. The bias should match cropping and clearly beat the $(1 + M)$ mask and $\beta = 0$.
The second check is a natural shortcut. We merge public COVID-ChestXRay with CORDA, train, and evaluate on held-out CORDA. We report sensitivity, specificity, and AUROC against the 0.54/0.80 reference, plus how well a linear probe on the learned features can predict the image source.
The third check is mask corruption. We erode, dilate, and shift the prior masks by increasing amounts, drop masks entirely for 20% of test images, and plot AUROC against corruption level for cropping and for the bias. The proposal succeeds only if the bias curve degrades significantly more slowly.
The fourth check is a sample-size sweep. We subsample training sets to 50, 100, 200, and 447 images, run ISIC to PAD-UFES-20 on the same schedule, and use five seeds with paired bootstrap tests, since the reported attention gains in this regime were not significant. The $\beta$ sweep and the learned-$\beta_h$ trajectories, run on the same splits, show whether the useful operating point is consistent across tasks or must be tuned for each one.
References
- When Medical Imaging Met Self-Attention: A Love Story That Didn't Quite Work out
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- ResMatch: Residual Attention Learning for Feature Matching
- RealFormer: Transformer Likes Residual Attention
- Residual Attention Network for Image Classification