When a network segments a thin structure such as a 2-pixel guidewire in a cardiac X-ray video, one frame of faint contrast is enough to cut the predicted wire in two. Overlap scores barely register the break, yet the output is useless for navigation. Two families of fixes exist, and neither was built for live video. Topology-aware training losses keep structures connected but work only on single images or static volumes. Models that use time look back at most one frame. We propose a streaming video segmenter that looks back over many past frames at low cost, using dilated temporal convolutions: filters that skip frames at exponentially growing intervals, so a few layers can see a long history. The segmenter is trained with a connectivity loss that is thrown away at test time, so it adds nothing to inference cost.
The key observation is specific to thin structures. A large organ that moves a few pixels between frames still overlaps its earlier position, so combining features at the same pixel across time helps. A 2-pixel wire that moves 5 pixels does not overlap itself at all. Temporal filtering at a fixed pixel therefore averages the wire with empty background and makes breaks worse, not better. We therefore tie the spatial reach of each temporal layer to how far the structure can move over that layer's time gap. We call this motion-matched dilation. We claim it is the ingredient that lets long temporal context repair topological breaks in real time.
Core Claim
Consider a structure of width $w$ pixels moving at up to $v$ pixels per frame. A temporal layer that combines frame $t$ with frame $t-d$ at the same spatial location can only help if $w > v\,d$. For guidewires and small vessels under cardiac motion this condition fails for almost every useful $d$. We therefore pair each temporal dilation $d_l$ at layer $l$ with a spatial dilation $r_l$ of a $k\times k$ kernel, chosen so that the spatial reach covers the displacement:
$$ r_l\,\frac{k-1}{2} \;\ge\; v\,d_l . $$
When this requires $r_l$ larger than the gridding limit $r \le k/s$ described in Rethinking Dilated Convolution for Real-time Semantic Segmentation, where $s$ is the stride, we split the reach across two stacked kernels instead of skipping pixels. The network is causal, meaning each output uses only current and past frames. It caches its past per-layer features, so each new frame costs one light 2D encoder pass plus a handful of small temporal convolutions. It is trained with a per-frame soft-skeleton connectivity loss. The claim has two parts. First, this design lowers the rate of broken structures per frame and the frame-to-frame flicker in connectivity, compared with single-frame, two-frame and recurrent models at matched compute. Second, the gain disappears when the motion-matching constraint is removed. The second part is what makes the idea falsifiable.
Where This Sits
Three topology-loss papers supply the training signal and also show why it has stayed static. clDice supervises the centerline instead of the full mask, using a differentiable skeleton built from repeated min- and max-pooling. It adds under 10% training time on a 1024×1024 image. TopoMatch matches topological features only when they overlap in space, which fixes the spatial blindness of earlier persistent-homology losses, but it costs $O(n^3)$ in pixel count. Efficient Betti Matching extends this matching to 3D, but its solver runs only on the CPU. We reuse clDice as the main loss and TopoMatch as a per-frame 2D variant. These losses are used only during training, so their cost never reaches inference; this is the point raised by the lightweight-versus-heavy split in the file, where Lv et al. pair clDice with a slow Swin-UNetR backbone.
We deliberately do not treat time as a third spatial axis for these losses. The same non-overlap argument means a moving wire is not voxel-connected in $(x,y,t)$ at clinical frame rates, so a 3D skeleton or 3D Betti loss would measure breaks that are really just motion.
The architecture takes its temporal mechanism from audio. There, stacked dilated convolutions grow the receptive field exponentially with depth while keeping full resolution, as surveyed in Deep Learning for Audio Signal Processing and shown on raw waveforms in stacked dilated CNNs for bandwidth extension. RegSeg supplies the efficient spatial backbone: 78.3 mIoU at 37 FPS on a T4 GPU with 3.34M parameters on Cityscapes, plus better handling of thin vertical objects than models that downsample aggressively. RegSeg has no temporal component and no connectivity loss. Audio models have no motion.
The closest competitor on the task is Deep Segmentation and Registration in X-Ray Angiography Video. It is a Siamese U-Net on frame pairs with an optical-flow consistency loss, reporting Dice 0.98 at 90 FPS. Its context is two frames, and it depends on flow estimation in images where contrast agent flows in and washes out. SF-VD motivates the domain: it shows 2–3-pixel guidewires are hard enough to justify diffusion-based augmentation. It also constrains the design, because iterative sampling is offline only. We use its F-video and P-image data but none of its generator at inference.
Research Program
We will build a per-frame encoder in the style of RegSeg, followed by a causal temporal stack with dilations 1, 2, 4 and 8. That gives a 16-frame view, which matches the F-video clip length. Each temporal layer uses a depthwise spatial kernel whose dilation follows the motion-matching rule. $v$ is estimated from the 95th percentile of annotated wire displacement in the training set. At inference, per-layer ring buffers hold cached features, so per-frame cost does not depend on the history length.
Training uses the combined Dice and soft-clDice loss applied to every output frame, with TopoMatch per frame as a variant. The analysis has three parts. The first is a sweep that turns the motion-matching rule on and off across synthetic speeds. The second characterizes temporal gridding, meaning thin or brief events that fall between sampled frames, as dilation grows. The third profiles latency on the same T4 hardware RegSeg reports.
Baselines
The naive reference is a single-frame RegSeg-style model trained with Dice and cross-entropy, run independently on each frame. It uses the same encoder, data and tuning budget as our model. It anchors both speed and the breakage rate we aim to reduce.
The strongest existing method on the question is the two-frame Siamese U-Net with flow consistency of Vlontzos and Mikolajczyk. Its reported numbers are Dice 0.98 at 90 FPS on a Titan-X, on their 36,000-frame angiography set. We found no reported result on Lacuna for guidewire segmentation Dice, clDice or Betti error on F-video or P-image; the only reported number there is SF-VD's FVD of 24.38. We will therefore reimplement the Siamese model on F-video and P-image with the same encoder width, clDice added for fairness, and the same hyperparameter search budget. Beating it means a lower per-frame wire-break rate and lower connectivity flicker at equal or better latency, with Dice no worse than 0.01 below it.
A third baseline is a convolutional GRU, a recurrent cell with convolutional gates that carries a hidden state from frame to frame. It is matched in FLOPs and trained with the same loss. It is the strongest cheap streaming competitor, because it also has constant per-frame cost.
The ablation that isolates our key ingredient is the identical model with temporal layers at spatial dilation 1. It shares architecture, loss and budget and removes only motion matching. A second ablation removes clDice to separate the architecture's effect from the loss's.
Why not just…?
Why not just train a per-frame model with clDice and smooth its outputs over time? This is a real threat, because it is nearly free. A running maximum or exponential moving average over the last few probability maps could bridge a one-frame fade. It shares the flaw our core claim identifies: averaging at a fixed pixel across frames smears a moving 2-pixel wire into a ghost trail, or averages it into background. We will include max- and average-pooling over time windows of 2 to 8 frames as extra baselines. If they match our break rate on real fluoroscopy, the motion-matching claim is wrong for this domain, and we will report that.
Why not just warp the previous prediction with optical flow? That is the Vlontzos design, and it is our main baseline. Flow assumes brightness stays constant between frames, which contrast inflow and washout violate. Flow also cannot recover a wire that has been faint for several consecutive frames, because it sees only one frame back. The contrast-dropout experiment below tests this directly.
Why not just use a recurrent ConvGRU, which is causal and cheap? This is also a real competitor. A recurrent cell with small kernels faces the same non-overlap problem, but it could learn to propagate information spatially over many steps. The FLOP-matched comparison settles which approach gives better breakage for the cost. We will also run a ConvGRU with motion-matched kernel dilation, so that a win cannot be credited to the dilated temporal convolution when it actually comes from motion matching.
Why not just use a video transformer? Attention cost grows quadratically with the number of tokens, and patching the input blurs 1–2-pixel structures. Lv et al. show Swin-UNetR-class models are heavy even on static volumes. We will report one small video transformer for completeness, but real-time streaming, not accuracy alone, is the criterion that rules it out.
Proposed Experiments
The decisive synthetic check renders thin curves 1 to 3 pixels wide on fluoroscopy-like noise. Speed $v$ varies from 0 to 8 pixels per frame, and contrast is randomly dropped for 1 to 6 consecutive frames. The prediction is sharp: the ablation without motion matching should help only while $w > v\,d$, and its benefit should collapse beyond that point. Our model's benefit should persist. If both curves track each other, the central claim fails.
On F-video (400 fully annotated 16-frame videos) and P-image (4,000 labeled and 10,000 unlabeled frames), we report per-frame Dice, clDice and Betti-0 error. We also report two measures of our own. The wire-break rate is the fraction of frames where the predicted wire has more than one connected component while the ground truth has one. Connectivity flicker is the fraction of consecutive frame pairs where the predicted component count changes but the ground truth's does not. Results will be stratified by the lowest-contrast decile of frames, where temporal context should matter most.
If the Vlontzos angiography data is accessible, we repeat the evaluation there to check that the method carries over to contrast-filled vessel trees. Latency is reported as mean and 99th-percentile milliseconds per frame, plus peak memory, on a T4 at batch size 1 in streaming mode, with and without the feature cache. We also report training-time overhead for clDice and TopoMatch, to confirm that the connectivity loss costs nothing at inference.
References
- Rethinking Dilated Convolution for Real-time Semantic Segmentation
- clDice - A Novel Topology-Preserving Loss Function for Tubular Structure Segmentation
- Topologically Faithful Image Segmentation via Induced Matching of Persistence Barcodes
- Efficient Betti Matching Enables Topology-Aware 3D Segmentation via Persistent Homology
- Automatic segmentation of mandibular canal using transformer based neural networks
- Deep Learning for Audio Signal Processing
- Waveform Modeling Using Stacked Dilated Convolutional Neural Networks for Speech Bandwidth Extension
- Deep Segmentation and Registration in X-Ray Angiography Video
- Label-Efficient Data Augmentation with Video Diffusion Models for Guidewire Segmentation in Cardiac Fluoroscopy