Edge-Gated Axial Attention for Small, Faint Lesions

Axial attention is a cheap way to give a segmentation network a global view. Instead of letting every pixel compare itself with every other pixel, each pixel attends only to its own row and then its own column, which cuts cost from quadratic in the image size to roughly the image size times its side length. This has a side effect that matters a great deal in medicine. When a lesion fills a tiny part of the image, as small polyps, kidney tumors and enhancing brain-tumor rims do, nearly every row and column passing through it is background. A lesion pixel then pools its features mostly from healthy tissue on the same line. When the lesion edge has low contrast, nothing in the attention weights stops this mixing. We think this "bleeding along the strip" is a form of class imbalance that lives inside the attention operator, not in the loss. If so, loss-level rebalancing cannot fully fix it.

We propose a simple change to axial attention. Along each one-dimensional strip, the network predicts a non-negative "boundary evidence" value at every position. Attention between two positions on the strip is then down-weighted by the total boundary evidence lying between them. Along a line this total is exactly the shortest-path crossing cost, and it can be computed with a single running sum. That makes the operation an exact one-dimensional geodesic barrier at almost no extra cost. The equivalent quantity in full 2D attention would require an expensive shortest-path computation. In this one respect, the axial factorization is what makes edge-aware long-range attention cheap, rather than being only a compromise for efficiency.

Core Claim

Our central claim is that edge-gated axial attention keeps the attention of foreground pixels concentrated on foreground, and that it does so independently of how small the lesion is. Plain axial attention, by contrast, loses this concentration as the lesion fraction shrinks. As a consequence, segmentation gains should be concentrated in the smallest and lowest-contrast lesions, and they should persist when a strong imbalance-correcting loss is already in use.

For a query at position $i$ on a strip and a key at position $j$, let $b_s \ge 0$ be the predicted boundary evidence at position $s$, and let $B_t = \sum_{s \le t} b_s$ be its running sum. The edge-gated attention weight is

$$ a_{ij} = \frac{\exp\left(q_i^\top k_j/\sqrt{d} - \lambda\,|B_j - B_i|\right)}{\sum_{j'} \exp\left(q_i^\top k_{j'}/\sqrt{d} - \lambda\,|B_{j'} - B_i|\right)}, $$

where $q_i$ and $k_j$ are the query and key vectors, $d$ is their dimension, and $\lambda$ is a learned per-head strength. The term $|B_j - B_i|$ is exactly the boundary evidence crossed between $i$ and $j$. A head with $\lambda \to 0$ falls back to ordinary axial attention, so the model can choose to keep cross-boundary context where that context helps.

This differs clearly from the closest work. DFormerv2 multiplies axis-decomposed attention by a decay based on the depth and pixel distance between two endpoints. Its decay is built from an external depth sensor for indoor scenes, and it depends only on the two endpoints, not on what lies between them. Our barrier is learned from image features and accumulated along the path. That is the property needed when two tissues have similar intensity at both endpoints but are separated by a faint edge. CaraNet pairs axial attention with reverse attention, which erases confident foreground so the decoder looks at margins. That changes which features are fed into attention, but within a row or column the affinities are still unconstrained. The file lists exactly this as CaraNet's open gap.

Research Program

We will build the operator as a drop-in replacement for the axial blocks in two host networks. The first is a 2D polyp network with a CaraNet-style decoder. The second is a 3D encoder-decoder with depth, height and width axial blocks in the style of AABTS-Net. The boundary evidence $b$ comes from a small per-strip head on the same features. It receives weak supervision from a boundary band derived from the ground-truth mask.

Boundary pixels are themselves rare, so this auxiliary loss is rebalanced in the way A Unified Loss proposes for hard pixels. We will also test a version with no barrier supervision, where the barrier is learned only from the segmentation loss. That version tells us whether the mechanism, rather than the extra labels, does the work. We keep the gating of positional encodings from Medical Transformer in all axial variants, because that paper showed ungated axial positional terms are unstable on small medical datasets. This choice constrains our design and is not a contribution.

The analysis centers on a diagnostic we call attention purity. Purity is the fraction of a foreground query's attention mass that lands on foreground keys, which we compute from ground-truth masks at every axial layer. We measure purity as a function of lesion fraction, in bins spanning roughly 0.01% to 10% of the image, and as a function of local edge contrast. If strip bleeding is real, vanilla axial purity should fall as lesions shrink, while edge-gated purity should stay roughly flat. A second analysis checks the known diagonal blind spot of axis decomposition, also noted by DFormerv2. In a cascade, row information reaches a target pixel only by passing through a column. A barrier on either hop can therefore block legitimate diagonal context. We will measure whether thin oblique structures lose recall.

Baselines

The naive reference point is the same host network with plain axial attention and a standard Dice plus cross-entropy loss. In the 3D setting this is essentially the AABTS-Net recipe, which reports BraTS 2021 validation Dice of 0.830, 0.861 and 0.922 for enhancing tumor, tumor core and whole tumor. It also reports a tumor-core Hausdorff distance of 11.178, which that paper acknowledges trails tuned convolutional pipelines. We will retrain it ourselves under our own splits and budget rather than rely on that figure.

The strongest existing methods differ by benchmark. For polyps, these are MEGANet, which guides attention with Laplacian-pyramid edge maps and reports 78.9% mean Dice on unseen ETIS, and CaraNet, which reports 74.7% on ETIS. PraNet reports 0.628 on ETIS and serves as a reverse-attention reference. For 3D lesions, the strongest competitor is the Balance Loss of A Unified Loss, which reports a kidney-tumor Dice of 66.8 on KiTS19 against 59.8 for cross-entropy. This is the most direct competitor on the imbalance question, because it already targets hard boundary pixels and background false positives. For MEGANet and CaraNet, we will train from their published recipes on the standard polyp protocol: train on Kvasir-SEG plus CVC-ClinicDB, then test on ETIS and CVC-ColonDB. We will also apply the same augmentation and epoch budget as our model. All methods receive the same number of hyperparameter trials, at least three seeds, and matched parameter counts within about 10%. Beating them means a mean Dice gain beyond seed variance on ETIS and ColonDB, together with a gain on the smallest-lesion bin. A gain only on large polyps would not count.

The key ablation swaps our path-accumulated barrier for an endpoint decay in the DFormerv2 style, built from the feature or intensity difference between query and key. The two versions have the same parameters and differ only in whether evidence is summed along the path. A second ablation feeds MEGANet's Laplacian edge map in place of the learned $b$. This tests whether learning the barrier avoids the noise amplification that the file flags for fixed second-order filters.

Why not just…?

Why not just fix the loss instead? The soft-versus-hard analysis in A Unified Loss shows that much of the damage from imbalance comes through the loss, and Balance Loss is a serious threat to our claim. However, a loss reweights how errors are penalized. It does not change the fact that a lesion pixel's feature has already been averaged with background before the prediction head ever sees it. We settle this with a two-by-two design: plain versus edge-gated attention, crossed with Dice-plus-cross-entropy versus Balance Loss, run on KiTS19 and BraTS. If Balance Loss with plain axial attention matches our model's small-lesion Dice and purity, our mechanism is redundant, and we would report that result.

Why not just use local window attention or convolutions, since small lesions do not need long range? This is the most plausible alternative. If bleeding is the problem, restricting attention to a local window also removes it. Window attention, however, gives up the long-range context that AABTS-Net's ablation found useful for fuzzy tumor margins, and the edge-gated heads can keep that context where no boundary intervenes. We include a cross-shaped window baseline in the style of CSWin with strip widths matched for compute, plus a strong convolutional baseline. If either equals our model on the small-lesion bins and on the cross-center polyp tests, we will narrow the claim to efficiency only.

Why not just add a boundary branch or reverse attention, as CaraNet and PraNet do? Those methods steer decoder focus toward the margin, but they leave the affinity computation inside each row unconstrained. Our CaraNet comparison isolates this difference: we keep CaraNet's reverse attention and replace only its axial blocks with edge-gated ones. Any gain in that comparison is attributable to the barrier itself.

Proposed Experiments

The 2D study uses the polyp protocol above, reporting mean Dice and mean IoU on ETIS and CVC-ColonDB, stratified by polyp area. The 3D studies use KiTS19 kidney tumor and NIH Pancreas, for which the file has no per-method number beyond Balance Loss's 66.8 tumor Dice and no reported number for Pancreas. They also use BraTS 2021, reporting Dice and 95th-percentile Hausdorff distance for enhancing tumor and tumor core, and SegTHOR esophagus as a thin, faint structure. For SegTHOR we note only the convolutional EMSN result of 0.8597 Dice from EMSN. We report FLOPs and memory to confirm that the running-sum barrier adds negligible cost.

Three checks decide the outcome. First, purity curves must show vanilla axial purity falling with lesion fraction while edge-gated purity stays flat. If vanilla purity is already flat, the motivating hypothesis is false and the project stops there. Second, in the factorial design, the edge-gated gain must survive under Balance Loss. Third, the path-accumulated barrier must beat the endpoint decay, specifically on low-contrast edges where the intensities on either side are similar. If only the endpoint version is needed, the idea reduces to a medical port of DFormerv2.