When a segmentation network has to trace a faint border, such as a polyp that looks like the colon wall or a lesion lost in ultrasound speckle, it usually decides pixel by pixel whether each location is boundary. That local decision fails in two ways. Under clinical noise, a random bright streak looks as much like an edge as a real but faint tissue margin does. Boundary-sharpening modules make this worse, because they are built to amplify exactly the high-frequency signal that noise produces. This proposal treats the boundary as a small set of candidate fragments and lets the network compare those fragments with one another. A fragment is a short segment joining two nearby candidate boundary points. The bet is that a real faint margin is supported by other fragments that continue it with consistent orientation, curvature and contrast polarity, while a noise-induced fragment has no such partners. That difference is invisible locally but visible when fragments exchange information.
The mechanism comes from scene-graph work, where "edge-to-edge attention" means relationship representations attend directly to other relationship representations, after a learned selection step discards most candidate pairs. Here that mechanism runs over a few thousand contour fragments sampled from a boundary heatmap, not over every pixel pair. It is inserted into a standard polyp and lesion segmentation network and judged on one question: does it keep boundaries accurate under corruptions the model never saw in training, beyond what noise augmentation plus a boundary loss already achieves?
Core Claim
Noise-induced pseudo-edges and true low-contrast edges are hard to tell apart from local evidence, but easy to tell apart from how they relate to other boundary fragments. A learned fragment-selection step followed by edge-to-edge attention should therefore reduce spurious boundaries and boundary-distance error under held-out corruptions, and do better than an equal-budget attention over boundary points. If point-level attention does as well, the "edge" part of the idea is unnecessary. The proposal is built so that this outcome would be visible.
Where This Sits
The relational ingredient comes from Devil's on the Edges: Selective Quad Attention for Scene Graph Generation (SQUAT). SQUAT scores candidate object pairs, keeps the top fraction, and passes messages node-to-node, node-to-edge, edge-to-node and edge-to-edge. It reports mR@100 of 16.5 on Visual Genome SGDet and a drop of nearly 9% when the selection module is removed. That paper supplies the method and a warning: unpruned candidate sets drown the useful context. Its edges connect detected bounding boxes, which dense segmentation lacks, so the main design problem is what counts as a node and an edge in a continuous image. Earlier relational modules, Relation Networks for Object Detection and LinkNet, attend among object proposals only. They show that context between entities helps, but they have no notion of a relation attending to a relation.
On the medical side, several boundary-focused segmenters motivate the gap, and none of them lets boundary pieces communicate. MISNet runs parallel axial reverse attention and boundary attention. CaraNet erases confident foreground to force attention onto margins. DB-MSMUNet adds a convolutional edge path. HF-UNet regresses a Gaussian contour map. Combating Uncertainty with Novel Losses sharpens boundary decisions purely through loss terms. All five treat boundary evidence as a dense map processed by convolutions or factorized attention along image axes, and none was evaluated under corruption.
The robustness literature constrains the design. Reducing Textural Bias shows that clean-trained CNNs collapse to 0–1% Dice under heavy noise, and that salt-and-pepper plus blur augmentation largely rescues them, so any architectural claim must be made on top of such augmentation, not against an unaugmented model. Hyperbolic U-Net gets robustness from latent geometry at about 11x inference cost and has no explicit boundary structure. MediMeta-C shows that clean leaderboard rank does not predict corruption robustness, which is why clean Dice alone cannot decide this work. FIVES and TransNetR contribute natural, non-simulated degradation: poor-readability fundus images and multi-center colonoscopy, respectively.
Research Program
The work has three parts: a diagnostic study, a module, and a controlled comparison.
The diagnostic comes first because it tests the premise. Pretrained MISNet- and CaraNet-style reverse-attention models, together with a plain decoder of the same backbone, are evaluated under a graded corruption suite. The suite uses the seven MediMeta-C corruption types (Gaussian noise, impulse noise, motion blur, zoom blur, brightness, contrast, pixelation) plus speckle. The study measures whether reverse-attention models lose boundary accuracy faster than the plain decoder as severity rises, and counts false boundary pixels far from any true contour. If reverse attention turns out to be no more fragile than the plain decoder, the "boundary modules amplify noise" story is wrong. The proposal would then narrow to the low-contrast ambiguity problem alone.
The module sits on a shared backbone and decoder that produce a mask and a boundary heatmap. The steps are as follows.
- Candidate points. Take roughly 512 local maxima of the heatmap.
- Candidate fragments. Connect each point to its six nearest neighbors.
- Fragment tokens. Build each fragment's token from three sources: decoder features averaged along the segment, its geometry (length, angle, and turning angle relative to adjacent fragments), and the signed intensity contrast across it.
- Selection. A small scorer keeps the top $\rho$ fraction of fragments.
- Message passing. SQUAT-style four-channel attention runs over the kept fragments and their endpoint points.
- Write-back. Updated point features are splatted back onto the feature grid with small Gaussian kernels and fused into a final refinement convolution.
With this design the attention runs over at most a few thousand tokens, which avoids the quadratic cost over pixels that confines relational methods to discrete graphs.
Fragments also receive direct supervision. A fragment $(i,j)$ between points $p_i$ and $p_j$ gets label
$$y_{ij} = \mathbb{1}\left[\max_{t \in [0,1]} d\big((1-t)p_i + t p_j,\ \partial M\big) < \delta\right],$$
where $\partial M$ is the ground-truth contour, $d$ is Euclidean distance, $t$ indexes positions along the segment, and $\delta$ is a tolerance of a few pixels. The selection scorer and a post-attention fragment classifier both train against $y_{ij}$ with a focal loss, alongside the usual Dice and cross-entropy mask losses.
Every model in the study, including every baseline, trains with the same augmentation recipe drawn from Reducing Textural Bias: impulse noise at probabilities 0.01–0.10 combined with Gaussian blur. Evaluation corruptions are then split into two groups. One group overlaps the training noise (Gaussian, impulse). The other is held out entirely (motion blur, speckle, contrast, pixelation). The held-out group is where any architectural claim must be made.
Proposed Experiments
Training and test data. The main polyp setting trains on Kvasir-SEG. It is tested in-distribution, on CVC-ClinicDB and ETIS (the small, concealed polyps where reverse attention was designed to help), and on multi-center PolypGen and BKAI-IGH for natural shift. Each test set is also run through the corruption suite at five severities.
Speckle setting. BUSI breast ultrasound tests speckle as a native, not simulated, corruption.
Thin-structure setting. FIVES tests thin, low-contrast curvilinear structures. Results are reported separately for its roughly 5% poor-readability images, which carry natural blur and illumination distortion.
Metrics. Dice and IoU track region overlap. Boundary quality is measured with 95th-percentile Hausdorff distance and average surface distance. A spurious-boundary rate counts predicted boundary pixels more than $\delta$ from any ground-truth contour. Robustness is summarized as mean corruption error normalized to the plain decoder, in the style of MediMeta-C. Latency and memory per image are reported for every model.
All numbers come from three seeds, with confidence intervals from a bootstrap over test images.
Baselines
Naive reference. This is the shared backbone and decoder with no boundary module, trained with the same augmentation, the same schedule and the same hyperparameter search budget. It anchors every relative number. It is also the direct answer to whether augmentation alone is enough.
Strongest accuracy baseline. On the polyp benchmarks this is MISNet, which reports mDice of 0.903 on Kvasir-SEG and 0.918 on CVC-ClinicDB. It is the best boundary-specific architecture in the file and the most direct embodiment of "axial boundary attention is enough." On ETIS, CaraNet reports 74.7% mDice and is the reference for small, low-contrast targets. Both are retrained on our splits with our augmentation and the same backbone family, and their published numbers serve as a sanity check that our reimplementation is faithful. Beating them means two things. First, clean Dice must be within 0.5 points of theirs. Second, HD95 and the spurious-boundary rate must be significantly better on the held-out corruptions and on PolypGen.
Multi-center baseline. For natural shift, TransNetR reports Dice 0.8706 on Kvasir-SEG and a 2.81% mIoU gain over the next best model on PolypGen. The file gives no absolute PolypGen figure, so we measure one ourselves under identical training.
Robustness baseline. Hyperbolic U-Net reports average gains of 48% under Gaussian and 22% under speckle noise relative to Euclidean U-Nets. The file gives no reported absolute BUSI or Kvasir number for it, so it is rerun on our splits. Latency is reported alongside its scores, because its roughly 11x overhead is part of the comparison.
Ablations of the key ingredient. These run at matched token count and parameter count:
- Points only. Attention among boundary points alone, with no fragment tokens and no edge-to-edge channel. This isolates the edge representation.
- No edge-to-edge. Fragment tokens are kept, but the edge-to-edge channel is dropped. This isolates relation-to-relation messages.
- No selection. All candidate fragments are kept. This tests SQUAT's pruning result in this domain.
- Random fragments. Neighbors are chosen at random instead of by nearest distance. This rules out a pure capacity effect.
The proposal counts as supported only if the full module beats the points-only variant on held-out-corruption boundary metrics.
Why not just…?
Why not just train with heavier noise augmentation? This is the most serious threat, because Reducing Textural Bias shows augmentation recovering Dice from near zero to 72–98% under heavy noise. We answer it by giving every arm that augmentation and measuring only on corruptions excluded from it. The same paper reports that overly severe augmentation hurts clean performance, which limits how far this route can be pushed. If the augmented plain decoder matches the full module on held-out corruptions, the architectural claim fails, and we will report that outcome.
Why not just add a boundary loss or contour head? Combating Uncertainty lifts left-atrium Dice from 82.01% to 91.54% with loss terms alone, and HF-UNet cuts ASD from 3.892 to 1.366 mm with contour regression. These are strong results, so the shared decoder already carries a boundary heatmap loss. Any gain from our module is therefore measured over a boundary-supervised model, not a bare one. The argument for relational messages is that losses score each pixel independently and cannot use the absence of a continuing fragment as evidence. The spurious-boundary rate under held-out noise tests exactly this.
Why not just use plain self-attention over pixels in a boundary band? If fragments add nothing, attention among boundary points with the same token budget should match them. The points-only ablation is this alternative, run at identical cost. A tie would mean the SQUAT-style edge formulation is ornamental, and we would then recommend the simpler model.
Why not just use hyperbolic features or axial reverse attention? Both are baselines, so the question is settled empirically. Hyperbolic U-Net offers global class separation but no contour structure, at about 11x cost. Reverse attention in CaraNet and MISNet is the design the diagnostic study directly examines for noise amplification.
Decisive Checks
The idea fails if any of three things happens:
- The diagnostic shows reverse attention is no more corruption-sensitive than a plain decoder.
- The points-only ablation matches the full module on held-out-corruption HD95 and spurious-boundary rate.
- Clean Dice on Kvasir-SEG or CVC-ClinicDB drops more than half a point below retrained MISNet.
The idea succeeds if three things hold together:
- The module lowers the spurious-boundary rate and HD95 on held-out corruptions and on PolypGen, BUSI and the FIVES poor-readability subset.
- The gain is larger than the gain from the no-edge-to-edge ablation.
- Added latency stays under 1.5x the plain decoder, far below the hyperbolic alternative.
A final qualitative check links the mechanism to the metric: kept-fragment maps on corrupted images should show noise fragments being dropped while faint true margins survive.
References
- Devil's on the Edges: Selective Quad Attention for Scene Graph Generation
- Relation Networks for Object Detection
- LinkNet: Relational Embedding for Scene Graph
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- CaraNet: context axial reverse attention network for segmentation of small medical objects
- DB-MSMUNet: Dual Branch Multi-Scale Mamba UNet for Pancreatic CT Scans Segmentation
- HF-UNet: Learning Hierarchically Inter-Task Relevance in Multi-Task U-Net for Accurate Prostate Segmentation in CT Images
- Combating Uncertainty with Novel Losses for Automatic Left Atrium Segmentation
- Reducing Textural Bias Improves Robustness of Deep Segmentation Models
- Hyperbolic U-Net for Robust Medical Image Segmentation
- On the Robustness of Medical Vision-Language Models: Are They Truly Generalizable?
- FIVES: A Fundus Image Dataset for Artificial Intelligence based Vessel Segmentation
- TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing