In colonoscopy and ultrasound video, the hardest pixels to segment sit on the border of a flat polyp or a thyroid nodule. There the lesion fades into healthy tissue with almost no change in brightness. A single frame often lacks the information needed to place that border. The video usually does have it: as the camera or probe moves, lighting, viewing angle and speckle change, and a border segment that is invisible in one frame may be clearly visible a few frames earlier or later. Current video segmentation models do not exploit this directly. They either spread temporal reasoning evenly over every pixel, or they sharpen boundaries afterward with single-frame tools.
We propose a video segmentation model that splits attention into two parts. Spatial attention operates within each frame and handles the easy interior of the object. Temporal attention, meaning attention across frames, is applied only to a thin, adaptively sized band of uncertain tokens around the predicted border. Inside that band, each token looks at nearby locations in recent frames and gives more weight to frames where that piece of border is locally clear. Put in two sentences: the model asks other frames for help only where the current frame is ambiguous, and it trusts whichever frame saw that edge best. We test whether this focused use of time improves segmentation on the "Indefinable Boundaries" cases that existing video models handle worst. The comparison is against spending the same compute on dense temporal attention, or simply smoothing per-frame predictions over time.
Core Claim
Our claim is that, at matched compute, temporal attention concentrated on an uncertainty band and weighted by per-frame edge clarity beats dense temporal attention specifically on low-contrast boundaries. It also beats per-frame boundary networks followed by temporal smoothing. Gains should be largest on the Indefinable Boundaries and Small Objects attributes of SUN-SEG, and should appear in boundary-distance metrics, not only in region overlap. If the gain disappears when the band is replaced by a random token subset of equal size, or when the clarity weighting is removed, the claim fails.
Two findings motivate the design. Sparse Video-Gen showed that even in dense 3D attention, heads separate on their own into spatial heads (block patterns within a frame) and temporal heads (diagonal patterns that follow pixels across time). That suggests the factorization costs little expressive power. PNS+ benchmarked 13 models and identified Indefinable Boundaries and Small Objects as the main failure modes of temporal models. Its own model has no boundary-specific mechanism. Together, these locate the open problem: decoupled temporal attention exists, and boundary attention exists, but temporal attention has not been pointed at the boundary.
Position Relative to Existing Work
The closest existing system is MAST. It splits attention into intra-frame self-attention and inter-frame mutual attention between an anchor frame and one reference frame at a fixed stride of two. Boundaries are sharpened later, in the decoder, with reversal attention. We differ in three ways. Our temporal attention covers a multi-frame window rather than a pair. It applies only to boundary-band tokens rather than all tokens. And it acts on the boundary inside the encoder rather than after it.
HFSTI-Net pairs Fourier-domain spatial processing with a memory bank queried by cross-attention. Its memory query is applied to the whole frame, with no notion of where the frame is uncertain. Its authors also leave transfer to ultrasound unproven, and we test that transfer.
Vivim splits video scanning into forward-temporal, backward-temporal and spatial state-space passes for ultrasound, and adds a global affine boundary loss. It shows that decoupling helps on blurry lesions. Its boundary signal is global, however, and its temporal pathway treats all positions the same way.
DVIS shows the opposite extreme. It keeps a frozen image segmenter and does temporal reasoning only over compact object queries. That makes it cheap, but it cannot reach pixel-level boundary gradients. We treat this as evidence that temporal reasoning must touch low-level features to help with ambiguous edges.
On the boundary side, Polyper supplies our starting mechanism. It erodes and dilates a coarse mask to separate an interior region from a "potential boundary region." It then uses cross-attention from confident interior features to resolve border pixels, and removing this region separation costs 4.40 mIoU points on ETIS. Its morphological kernel is fixed and it uses one frame. We replace the fixed kernel with a band defined by prediction uncertainty and temporal disagreement, and we let the band query other frames.
PraNet and MEGANet play supporting roles. PraNet's reverse attention erases the confident foreground so later layers focus on the perimeter, which is the same intuition applied within a single frame. MEGANet injects Laplacian edge priors at every scale. Its own analysis warns that such differential operators amplify noise and speckle, which is why we estimate edge clarity from learned features rather than raw Laplacians.
Finally, NA-UNETR argues that in low-contrast regimes, global attention lets background noise swamp faint boundaries, and that local neighborhood attention works better. HFSTI-Net and PolypFlow argue the reverse, in favor of global frequency shortcuts. Our design takes a position in this disagreement. Temporal attention for a boundary token is local in space, a small window around its location in each past frame. Long-range context reaches the band only indirectly, through the spatial branch. An ablation with an unrestricted spatial window tests which side holds for the temporal pathway.
Research Program
The backbone is PVTv2, the same backbone used by HFSTI-Net and MAST, so that differences come from the attention design.
Spatial branch. Each frame passes through standard within-frame attention. A light head produces a coarse mask and two per-token quantities. The first is uncertainty, measured as the entropy of the coarse prediction. The second is a learned boundary-clarity score $c$ between 0 and 1. It is trained to predict how well that token's local boundary is resolved, with the target derived from the local agreement between prediction and ground truth during training.
Band selection. The band $B_t$ for frame $t$ is the set of tokens whose uncertainty exceeds a threshold, together with tokens where the current coarse mask disagrees with the previous frame's mask after warping by a cheap feature-matching step. Because the band follows uncertainty rather than a fixed kernel, it widens on flat, diffuse polyps and narrows on crisp ones. This addresses the fixed-geometry limitation of Polyper. The band is always seeded from the current frame's own prediction, not only from memory. This guards against the drift and initialization sensitivity that MemSAM reports, where a bad first frame corrupts the entire video.
Temporal branch. For each token $i$ in $B_t$, the temporal branch attends to keys $j$ drawn from a spatial window $W(i)$ in each of the previous $K$ frames, which are held in a small feature memory. Attention weights are
$$ a_{ij} = \mathrm{softmax}_j\!\left(\frac{q_i^\top k_j}{\sqrt{d}} + \log c_j\right), $$
where $q_i$ and $k_j$ are query and key vectors of dimension $d$, and $c_j$ is the clarity score of the key token in its own frame. The additive $\log c_j$ term multiplies each frame's contribution by how clearly that frame saw its edge. This is the mechanism for "borrowing from clearer frames." The attended features are added back to the band tokens only. Interior and background tokens skip the temporal branch entirely.
Cost. The cost scales with the band size $|B_t|$ times $K|W|$ instead of $N \cdot T$. This should keep inference above the roughly 25 to 30 frames per second that clinical endoscopy needs. HFSTI-Net reports 31.27 FPS and Vivim 35.33 FPS as reference points. The intended outcome is to spend temporal compute where the error is, not merely to save compute.
Training loss. We combine weighted binary cross-entropy and weighted IoU, the HFSTI-Net loss, with a boundary term applied inside the band.
Analysis. We will visualize, per video, which past frames each band token draws from. We will also test whether those frames have measurably higher local contrast at the corresponding location. This tests the mechanism directly rather than inferring it from Dice.
Baselines
Naive reference: per-frame model. The first reference point is the same PVTv2 encoder and decoder trained per frame, with no temporal branch and no band. Every other gain is measured from it. A second naive reference adds temporal smoothing: the per-frame model's masks are averaged over a short window after flow-based warping. Together they establish how much of any gain is simply temporal averaging.
Strongest per-frame boundary method. We run Polyper's region separation and boundary-sensitive attention on the shared backbone, frame by frame. It is the strongest image-only competitor in the file, with ETIS mDice 86.51% as reported in Polyper. It isolates whether our boundary handling needs time at all.
Strongest video competitor. HFSTI-Net reports SUN-SEG Hard Dice 86.27% in HFSTI-Net. MAST reports 0.803 in MAST, and PNS+ is listed at 0.737 in that comparison. The gap between the HFSTI-Net and MAST figures may partly reflect differing protocols. We will therefore retrain both HFSTI-Net and MAST on the same SUN-SEG training split, input resolution, augmentation, epoch count and hyperparameter search budget as our model. We will report our reproduced numbers next to the published ones.
Ultrasound competitor. On VTUS the strongest reported competitor is Vivim, with Dice 0.8324 in Vivim. We will retrain it under our protocol.
Matched-compute ablations. These isolate the key ingredient. In the dense-temporal ablation, every token receives the same multi-frame windowed temporal attention, and frames or window size are adjusted to match our FLOPs. The random-band ablation uses a band of identical size chosen at random. The fixed-band ablation uses a band from Polyper-style morphology with a fixed kernel. The no-clarity ablation removes the $\log c_j$ term.
What counts as beating the competitors. Beating HFSTI-Net means exceeding our reproduced HFSTI-Net on SUN-SEG Hard Dice while staying above 25 FPS. The key result, however, is a clearly larger margin on the Indefinable Boundaries attribute and on boundary distance (HD95) than on overall Dice. No reported result was found on Lacuna for per-attribute Indefinable Boundaries Dice for any of these models, so all attribute numbers will come from our own reproductions.
Why not just…?
Why not just run a strong per-frame boundary model and smooth its masks over time with optical flow? This is the most serious threat, because averaging alone can reduce flicker. The file's evidence suggests a ceiling. In PNS+, the image model ACSNet reaches 0.713 Dice on SUN-SEG-Easy (Unseen) against 0.756 for the spatio-temporal model. HFSTI-Net describes per-frame "shape collapse" under low contrast. Post-hoc smoothing averages masks that may all be wrong in the same way. It cannot recover a border that no single prediction found, whereas feature-level borrowing can combine partial evidence before the decision. That reasoning is not proof. Our Polyper-plus-flow-smoothing baseline settles it directly: if it matches our gain on the Indefinable Boundaries subset, we drop the claim.
Why not just apply dense temporal attention to every token over more frames, as a memory-bank model already does? If the benefit came only from more temporal context, dense attention at equal compute would match us. The matched-FLOPs dense ablation and the random-band ablation test exactly this. The hypothesis predicts that dense attention wastes its budget on interior and background tokens that are already resolved. It also predicts that random bands lose most of the gain. Sparse Video-Gen found that sparsifying too early in its computation damages global structure, so concentrating attention carries a real risk of hurting overall shape. That is why the spatial branch stays dense.
Why not just adapt a foundation model like SAM with a memory module? MemSAM shows this works well on echocardiography, with CAMUS mDice of 93.31%. It also reports strong sensitivity to the first frame's quality, and it inherits the cost of a SAM ViT, which strains real-time colonoscopy. We include MemSAM as a reference on CAMUS. Its memory is whole-frame, so a version of it with our band and clarity weighting is a natural follow-up if the mechanism holds.
Why not just use a state-space model such as Vivim? Vivim's decoupled scans are linear in sequence length. Its scans move along fixed one-dimensional orders, though, so they cannot select which past frame to trust for a particular edge token. The clarity-weighted, token-to-token selection is the part we claim matters. Retraining Vivim on SUN-SEG and comparing on the Indefinable Boundaries subset will show whether that selectivity adds anything beyond linear-time temporal mixing.
Proposed Experiments
Primary benchmark. We train on the SUN-SEG training split and report Dice and mIoU on the Easy and Hard subsets, under Seen and Unseen settings. We break results down by the Indefinable Boundaries, Small Objects, Fast-motion and Ghosting attributes, and add HD95 and a boundary F-measure. Frames per second are reported on one GPU type for all models.
Cross-domain transfer. We test on CVC-612 and CVC-300 video splits; PNS-Net reports CVC-300-TV maxDice 0.840 in PNS-Net. We then retrain on VTUS and CAMUS without architectural changes, to test the ultrasound transfer that HFSTI-Net left open.
Decisive checks. There are three. First, the full model must beat the matched-FLOPs dense ablation and the random-band ablation on the Indefinable Boundaries subset by more than run-to-run variation, measured over three seeds. Second, removing the clarity term must reduce the Indefinable Boundaries gain. In addition, the frames receiving high attention must show higher local contrast at the matched location, measured on held-out videos. Third, the Polyper-plus-flow-smoothing baseline must fall short of the full model on the Indefinable Boundaries subset and on HD95.
If the first check fails, the work becomes a negative result on concentrated temporal attention. If only the clarity check fails, the contribution narrows to adaptive band selection. If the smoothing baseline matches us, the honest conclusion is that feature-level temporal reasoning is unnecessary for ambiguous medical boundaries. That would also be worth reporting.
References
- Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity
- Video Polyp Segmentation: A Deep Learning Perspective
- MAST: Video Polyp Segmentation with a Mixture-Attention Siamese Transformer
- HFSTI-Net: Hierarchical Frequency-spatial-temporal Interactions for Video Polyp Segmentation
- Vivim: a Video Vision Mamba for Medical Video Object Segmentation
- DVIS: Decoupled Video Instance Segmentation Framework
- Polyper: Boundary Sensitive Polyp Segmentation
- PraNet: Parallel Reverse Attention Network for Polyp Segmentation
- MEGANet: Multi-Scale Edge-Guided Attention Network for Weak Boundary Polyp Segmentation
- A Neighborhood Attention Transformer Network for Enhanced 3D Segmentation of the Left Anterior Descending Artery
- PolypFlow: Reinforcing Polyp Segmentation with Flow-Driven Dynamics
- MemSAM: Taming Segment Anything Model for Echocardiography Video Segmentation
- Progressively Normalized Self-Attention Network for Video Polyp Segmentation