Models that trace thin structures in video, such as retinal vessels in handheld fundus recordings, coronary arteries in X-ray angiography, or vessel landmarks in laparoscopic surgery, fail in a way that standard metrics barely register. A vessel can be correctly connected in one frame, break at a single pixel in the next, and reconnect in the frame after. Pixel-overlap scores hardly change. But anything that consumes the result as a network, such as path tracing, branch counting, or flow simulation, sees the vessel tree split and rejoin many times per second. We call this topological flicker.
We propose to suppress it with an adversarial critic. A critic, or discriminator, is a second network trained to tell real examples from model outputs, and its feedback trains the model. Ours sees only thin-line skeletons of the predicted masks across a window of frames. It learns what plausible changes in vessel connectivity over time look like. Its "real" examples come from a cheap source: expert labels on sparse keyframes, carried to the frames in between by smooth, invertible image registration. A smooth invertible warp cannot cut or glue a structure, so these carried labels have the right connectivity even when their exact geometry is off. The critic transfers that connectivity information, and not the faulty geometry, to the thousands of unlabeled frames where today's topology losses never apply.
Core Claim
Existing topology-preserving losses are computed on single frames or static volumes, and only where labels exist. Examples are centerline Dice, which compares the thinned skeleton of one mask against the other mask (clDice), and persistent-homology losses, which count connected pieces and loops across thresholds (TopoMatch, Byrne et al.). In the video datasets that matter, labels are sparse. RVD annotates 1,270 keyframes out of about 130,000 frames, and L3D annotates 1,152 frames. Most frames therefore receive no topological supervision at all.
Recurrent adversarial video models do supervise every frame for temporal plausibility. Examples are RT-GAN, with its 3-frame spatiotemporal critic, and RAVE, with its recurrent discriminator. But their critics look at dense pixel fields through downsampling layers that wash out one-pixel lines. They were also tested only on broad structures such as colonoscopy folds and global tone.
Our claim is that a critic restricted to skeleton sequences, with real examples built from registration-propagated keyframe labels, reduces frame-to-frame changes in vessel connectivity on unlabeled frames. It should do so without lowering per-keyframe overlap. It should also beat the obvious non-adversarial alternative of training directly on the propagated labels.
What Is Built
The segmenter is a lightweight recurrent model in the style of RT-GAN. It predicts the mask $y'_t$ from the current frame $x_t$, the previous frame $x_{t-1}$, and its own previous output $y'_{t-1}$. The backbone is the same one used by the per-frame baselines.
On every frame we compute a soft skeleton $s_t$ of the predicted mask, using the min/max-pooling skeletonization from clDice. This is the only part of the mask the critic sees. The critic receives a window of $K$ skeletons stacked with the matching image frames, with $K=8$ as the default. It runs at full resolution with dilated rather than strided convolutions, so one-pixel paths are not averaged away. A small convolutional recurrent state carries information across windows and is detached at window boundaries. This truncation trades long-range gradients for stability, which addresses the long-horizon instability noted for recurrent critics.
The "real" examples are built as follows. For each pair of adjacent labeled keyframes, we estimate a smooth invertible deformation between the frames using an off-the-shelf registration code base (PC-Reg-RT). We warp both keyframe labels into every intermediate frame and blend them by temporal distance. Then we skeletonize the result. Because the warps are invertible and smooth, each carried label has the same number of connected pieces and loops as its source keyframe. Where the two keyframes disagree in topology, for example when a branch fills with contrast, the blend window is where the change happens.
The training objective is
$$ \mathcal{L} = \sum_{t \in \mathcal{K}} \Big[(1-\alpha)\,\mathcal{L}_{\text{Dice}}(y'_t, y_t) + \alpha\,\mathcal{L}_{\text{clDice}}(y'_t, y_t)\Big] + \lambda\,\mathcal{L}_{\text{adv}}\big(D(s_{t:t+K}, x_{t:t+K})\big). $$
Here $\mathcal{K}$ is the set of labeled keyframes, $y_t$ is the expert label, $\alpha$ is the clDice compound weight from the clDice paper, $D$ is the skeleton-sequence critic, and $\lambda$ weights the adversarial term. The adversarial term is applied to all frames, labeled or not.
The design separates two kinds of information. Geometry comes only from true labels. Plausible connectivity dynamics come from propagated labels, but only as a distribution the critic learns, never as targets to match pixel by pixel.
Relation to Prior Work
Two papers motivate the gap. TopoNet combines clDice with a persistence loss on isolated laparoscopic frames. It has no recurrence and no critic, so it says nothing about connectivity between frames. RT-GAN supplies our recurrent design but uses dense critics on bulky targets. Our change is the critic's input, skeleton sequences, and its source of real data, propagated labels.
Two papers constrain the design. The coronary angiography work (coronary video segmentation) smooths outputs by weighted frame averaging and reports failure when contrast is thin or uneven. The angiography Siamese network (Deep Segmentation and Registration in X-Ray Angiography Video) enforces consistency between frames by warping masks with optical flow. It reaches 0.98 Dice but reports boundary inaccuracies, and it carries no topological term. Together these warn that deterministic temporal tying blurs moving thin vessels. That is why we use warps only to make critic examples, never to force predictions to match pixel by pixel.
TopoMatch shows that plain persistence matching pairs features from opposite sides of the image up to 99.4% of the time. We reuse its spatially grounded induced matching for evaluation, not training. Full spatiotemporal persistent homology is too costly per training step: Byrne et al. report about 15 minutes per 3D volume.
Research Program
Training uses RVD as the main testbed. It is true handheld video with severe motion and pulsation, and its reported cross-domain collapse (28.31% mIoU when trained on bench-top images) shows how fragile static models are there. The 365-sequence X-ray angiography dataset is the second testbed. L3D is a third testbed, but only if the surrounding unlabeled video frames can be obtained alongside its keyframes. We will verify this with the TopoNet release (cuiruize/TopoNet). If they cannot, L3D is used only to check per-frame parity with TopoNet.
On the angiography data, the weak top-hat labels that paper extracts on all frames give a second, denser source of real examples. Comparing the two sources tells us how much the critic depends on how its real data is made.
For evaluation we introduce a topological flicker rate that needs no labels. For each pair of consecutive frames, we warp the frame-$t$ prediction to frame $t+1$. We then compute the TopoMatch induced matching between the warped prediction and the frame-$(t+1)$ prediction, counting only features above a fixed persistence threshold, and count the unmatched features. The flicker rate is the average of this count over all frame pairs.
On held-out labeled keyframes we report Dice, IoU, clDice, Betti-number errors, and Betti-matching error. The Betti-matching error is computed with the fast implementation bundled with TopoNet. We also check that the flicker metric tracks human judgment on a small sample of rated clips.
Baselines
The naive reference is a per-frame U-Net trained with Dice on keyframes only and applied to every frame independently. It shows how much flicker a model with no temporal or topological knowledge produces. The second reference is the same network plus the clDice compound loss, which is the static topology remedy from the clDice paper. On L3D the per-frame topological reference is TopoNet itself, run from its released code. Its reported 65.19% DSC, 50.56% IoU and 28.07-pixel ASSD (average symmetric surface distance) are the numbers our segmenter must at least match on keyframes.
The strongest per-frame competitor on RVD is Mask2Former (Swin-L), which RVD reports at 74.09% mIoU for binary vessel segmentation. On X-ray angiography, the Siamese flow-consistency network reports 0.98 binary Dice. We found no reported result on Lacuna for any temporal topology measure, such as flicker or connectivity change between frames, on any of these datasets. Those numbers will come from our own runs.
The two strongest temporal competitors are both real threats. The first is a post-hoc temporal filter in the style of the coronary paper, applied to the clDice model. The second is an RT-GAN-style recurrent segmenter with its 2D frame critic and 3-frame dense 3D critic, plus clDice on keyframes. This second baseline isolates whether looking at skeletons rather than dense masks matters.
All methods share the backbone, the training frames, the keyframe split, and an equal search over learning rate and loss weights. We also give them an equal GPU-hour budget, with the adversarial methods charged for critic compute.
The ablation that isolates the key ingredient keeps everything fixed and swaps the critic's real examples. In one variant the real examples are propagated labels and in the other they are per-frame keyframe skeletons with no temporal ordering. This tests whether the critic is learning dynamics or only shape. A second ablation removes the recurrent input $y'_{t-1}$ to test whether feedback drift matters.
We count our method as having won if it cuts flicker rate by a clear margin over both RT-GAN+clDice and the filtered clDice model, while keyframe Dice stays within one point of the best per-frame baseline and Betti error does not get worse. The required margin is set before testing, by bootstrap over sequences.
Why not just…?
Why not just train a recurrent segmenter with per-frame clDice and skip the adversary? This model receives topology supervision only on keyframes, about 1% of frames in RVD. Its recurrent feedback can carry a break forward unchecked, which is exactly the drift RT-GAN-style generators are prone to. It is our second ablation and the closest threat. If it matches our flicker rate, the critic adds nothing, and we will report that.
Why not just smooth the output over time or warp masks with optical flow? The coronary paper reports that frame averaging fails under thin or uneven contrast. The angiography paper's flow-tied model reports boundary errors despite 0.98 Dice. RAVE notes that flow is costly and unreliable under non-rigid motion. Both appear as baselines, and Betti error on keyframes will show whether they trade connectivity for smoothness.
Why not just train directly on the registration-propagated labels with clDice? This is the most serious alternative, because it uses the same cheap signal without an adversary. Its weakness is that propagated labels are geometrically wrong exactly where vessels deform or fill with contrast. Matching them pixel by pixel should pull predictions toward those errors, whereas the critic only asks that connectivity changes look plausible. We run this exact variant. If it equals our method on flicker and keyframe accuracy, we will drop the adversary and report the simpler method.
Why not just use a video transformer? On RVD, the video transformer baseline reaches only 51.25% mIoU on localizing vessel pulsations in time. Swin-UNetR on mandibular canals still needed clDice to stop broken outputs, so global attention alone does not keep thin structures connected.
Decisive Checks
The central experiment is the propagated-label ablation on RVD and X-ray angiography, measured by flicker rate. The adversarial variant must beat direct pseudo-label supervision on flicker without losing keyframe Dice or clDice. Otherwise the proposal's core idea fails.
A second check perturbs the registration on purpose, by adding warp noise to the propagated labels. We expect the adversarial method to degrade more gracefully than direct supervision, since it uses the labels' connectivity and not their geometry.
A third check plots flicker against keyframe Dice as $\lambda$ varies. This maps the trade-off between temporal stability and topological accuracy that the literature has left uncharacterized.
References
- clDice - A Novel Topology-Preserving Loss Function for Tubular Structure Segmentation
- Topologically Faithful Image Segmentation via Induced Matching of Persistence Barcodes
- A Persistent Homology-Based Topological Loss for CNN-Based Multiclass Segmentation of CMR
- RVD: A Handheld Device-Based Fundus Video Dataset for Retinal Vessel Segmentation
- Topology-Constrained Learning for Efficient Laparoscopic Liver Landmark Detection
- RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approaches
- An Efficient Recurrent Adversarial Framework for Unsupervised Real-Time Video Enhancement
- Coronary angiography video segmentation method for assisting cardiovascular disease interventional treatment
- Deep Segmentation and Registration in X-Ray Angiography Video
- Automatic segmentation of mandibular canal using transformer based neural networks