Letting Overlapping Fetoscopic Frames Supervise Each Other's Blurry Vessel Edges

A fetoscope, the thin camera surgeons use inside the womb to laser-seal abnormal placental blood vessels, sees only a few centimetres of tissue at a time. The images are murky: amniotic fluid scatters light, particles float past, and vessel edges fade into the surrounding tissue. Two tasks follow from this. Segmentation networks must outline the vessels, and those outlines are then stitched into a wide map of the placenta, a process called mosaicking. Both tasks fail at the same place, the faint vessel boundary. The narrow field of view also brings one useful property. Because the camera sweeps slowly, every patch of placenta appears in many consecutive frames, each seen at a different distance, angle and illumination. A boundary that is hopelessly blurred in one frame is often crisp in a neighbouring one.

We propose to use that redundancy as a training signal. During training, predictions from neighbouring frames are warped onto the current frame and blended pixel by pixel, with weights set by a small network that estimates how trustworthy each view is at that location. The blend then serves as a soft target for the current frame, but only at pixels where the current view is judged less reliable than its neighbours. The result is a composite loss whose balance between ordinary supervision and cross-frame agreement changes across the image instead of being fixed by a single global weight. At test time the network is still a single-frame segmenter, with no added latency, and its vessel maps should be sharper at edges and therefore better material for stitching.

Core Claim

In limited-field-of-view video, overlapping views carry most of the information needed to resolve low-contrast boundaries. They can be exploited through a reliability-weighted consistency loss, which avoids architectural boundary modules and hand-crafted edge priors. The weighting across views is the "adaptive modality weighting" ingredient: each overlapping frame is treated as a separate, unequally reliable sensor of the same tissue. We claim three things. First, the gain over a well-tuned U-Net will concentrate in a band a few pixels wide around vessel edges and in low-contrast or turbid frames. Second, learned per-pixel view weights will beat uniform averaging of the same warped neighbours. Third, the sharper maps will reduce accumulated drift when they are fed to the same registration pipeline that Deep Placental Vessel Segmentation for Fetoscopic Mosaicking uses.

Let $\hat p_s$ be the vessel probability map predicted for frame $s$, and $W_{s\to t}$ the warp from frame $s$ into frame $t$, estimated by registering vessel maps. Let $\mathcal N(t)$ be a window of nearby frames that includes $t$ itself, and let $\alpha_s(x)$ be the reliability weight for view $s$ at pixel $x$, normalised over $\mathcal N(t)$. The blended target is

$$\bar p_t(x) = \sum_{s \in \mathcal N(t)} \alpha_s(x)\, W_{s\to t}\hat p_s(x),$$

treated as a constant (no gradient). The training loss is

$$L = L_{\text{BCE+IoU}} + \lambda_b L_{\text{bnd}} + \lambda_c \sum_x \big(1-\alpha_t(x)\big)\,\ell\big(\hat p_t(x), \bar p_t(x)\big).$$

The first term is the supervised loss of Bano et al. The second is a distance-based boundary loss in the style of Boundary loss for highly unbalanced segmentation. The third is the cross-frame term, where $\ell$ is binary cross-entropy. The factor $1-\alpha_t(x)$ is the key design choice. Where the current frame is clear, the consistency term effectively switches off, so it cannot compete with interior supervision. Where the frame is murky and its neighbours are clear, the neighbours dominate. The third term applies to unlabelled video frames as well as labelled ones.

Relation to Prior Work

Two papers motivate the proposal by showing that the gap is real. Deep Placental Vessel Segmentation for Fetoscopic Mosaicking trains a U-Net with a fixed BCE+IoU loss and then registers its vessel maps with pyramidal Lucas-Kanade. Segmentation never sees the video or the registration it feeds. The FetReg2021 challenge (FetReg2021; findings paper) likewise scores segmentation and registration as separate tasks.

The same challenge also constrains the design. The organisers' plain ResNet50-U-Net reached 0.6763 mIoU, while the top participant, which added regression of hand-crafted gradient histograms (HoG) to sharpen edges, reached only 0.6411. Globally applied edge terms can therefore hurt. This result is why our consistency term is gated per pixel rather than added everywhere with a fixed weight.

Two further papers supply the weighting idea, applied elsewhere. Adaptive Weighting Multi-Field-Of-View CNN learns image-conditioned weights over expert networks trained at different magnifications. It weights views at one location and one time, and it does not use cross-view agreement as supervision. Robust endoscopic image mosaicking via fusion of multimodal estimation fuses three pose estimators using their uncertainty. It weights whole-frame motion estimates, not per-pixel segmentation evidence. Both papers support the choice of reliability weighting over uniform averaging. H2ASeg shows that target-aware weighting of physical modalities (PET and CT) helps tumour boundaries in static volumes. We carry the idea to views of the same tissue across time.

Boundary-focused architectures such as PraNet and CaraNet attack edges inside a single image, and they cannot recover information the frame does not contain. Our method brings in information from other frames. The Direction 3 literature also notes that gating networks become overconfident about degraded inputs. We address this by training the weighting network directly against labels rather than letting it drift freely.

Research Program

The segmenter is a ResNet-backbone U-Net matching Bano et al., so any gain cannot be credited to capacity.

Warps between frames up to $k$ apart, with $k$ between 2 and 8, are estimated by pyramidal Lucas-Kanade on the current vessel maps. They are re-estimated each epoch, so registration and segmentation improve in alternation. Consecutive frames move little, so local homographies are adequate within the window even though they fail over hundreds of frames.

The reliability network is a small convolutional net. For each view it receives the local image patch, the warped prediction, and the photometric registration residual, and it outputs a per-pixel probability that this view's warped prediction is correct. It is trained with cross-entropy on labelled frames $t$, using the ground truth at $t$ to score each neighbour's warped prediction. This requires labels only at the target frame, and it makes the weights calibrated estimates of correctness rather than free attention scores. The weights $\alpha$ are these probabilities normalised over the window. On unlabelled frames from the FetReg training clips, the frozen reliability network sets $\alpha$ and the consistency term does the work.

Beyond accuracy, we will analyse what the weights learn. We will check whether $\alpha_t$ tracks measurable local contrast, specular highlights, and particle occlusion, and whether consistency gradients actually concentrate in the boundary band.

Baselines

The naive reference point is the FetReg2021 organisers' ResNet50-U-Net with aggressive augmentation. The challenge reports 0.6763 overall mIoU for it, above every participant, which makes it the strongest competitor on segmentation. We will retrain it with the same augmentation recipe, backbone, epochs, and hyperparameter search budget as our model, twenty configurations each, tuned on the same validation split.

For the vessel-only setting, the reference is the U-Net of Deep Placental Vessel Segmentation for Fetoscopic Mosaicking, which reports average Dice 0.78 and IoU 0.66 over six folds. We rerun it on the same folds. Beating these baselines means a statistically reliable gain in vessel IoU, paired across folds or test clips, with a larger gain in boundary IoU and 95th-percentile Hausdorff distance. The file contains no reported boundary-metric result for FetReg, so we compute these metrics for every method ourselves.

The strongest single-frame boundary competitor is the same U-Net with the boundary-loss term added and no cross-frame term. The strongest cross-frame competitor is the full method with learned weights replaced by uniform averaging over the window.

Two ablations isolate the key ingredients. The first removes the $1-\alpha_t$ gate, applying consistency everywhere with one fixed weight. This tests the claim, suggested by the HoG result, that ungated auxiliary terms hurt. The second uses only labelled frames, which separates the effect of the weighting from the effect of extra unlabelled video.

For mosaicking, the baseline is the FetReg affine registration, reported at 0.9348 five-frame SSIM. We also run Lucas-Kanade on Bano-style maps, where drift is described only as "negligible" with no scalar value reported on Lacuna. We keep the registration algorithm fixed and swap only the vessel maps it receives.

Why not just…?

Why not just use a well-tuned U-Net with heavy augmentation? This is the most serious threat, because FetReg found such a baseline beat all specialised entries (0.6763 versus 0.6411 mIoU). Our method keeps that exact network and training recipe and changes only the loss, so the comparison is clean. Augmentation creates synthetic variations of a single view. It cannot supply the information that a crisper neighbouring view actually contains. If our gain disappears once augmentation is tuned equally hard, the claim is wrong and we will report that.

Why not just add a boundary loss to the single-frame model? Boundary losses sharpen edges that the frame already supports, but a turbid frame may not support them at all. The file also documents confounder-induced false edges from speculars and shadows, which a boundary loss rewards rather than suppresses. The "boundary loss only" baseline tests this directly. We predict it helps on clear frames and fails on low-contrast frames, where cross-frame evidence should matter most.

Why not just average warped neighbour predictions, at training or test time? Uniform averaging mixes in exactly the degraded views that should be ignored. Adaptive Weighting Multi-Field-Of-View CNN and Robust endoscopic image mosaicking both found that reliability weighting beats unweighted combination of views or estimators. This is still an empirical question in our setting, and the uniform-average ablation settles it. If uniform averaging matches learned weights, we drop the weighting claim and keep the simpler method.

Why not just fix drift with a pose graph with loop closure? That approach, used in Robust endoscopic image mosaicking, is complementary: it corrects the global trajectory but leaves the per-frame registration signal unchanged. We hold registration fixed so that drift changes can be attributed to map quality. A pose graph could be layered on top afterwards.

Proposed Experiments

The main segmentation experiment uses FetReg2021: 2,060 labelled training images, 18 training clips as the unlabelled pool, and a test set of 658 images plus 6 clips from unseen procedures. We report overall mIoU against the 0.6763 reference, vessel-class IoU, boundary IoU, and Hausdorff distance. Test frames are stratified by measured local contrast, so we can check whether gains grow as contrast drops. That pattern is the signature the core claim predicts.

A cross-centre split, training on one hospital and testing on the other (the UCLH clear views versus the IGG red-spotlight views), tests whether calibrated reliability weights transfer across illumination. We expect the most benefit here, but a failure of the weighting network under shift is also a plausible outcome worth reporting.

The mosaicking experiment feeds each method's maps into the same pyramidal Lucas-Kanade pipeline on the 6 FetReg test clips and on the Bano et al. sequences of up to 400 frames. We report five-frame SSIM against 0.9348, plus drift measured as reprojection error of annotated vessel landmarks after revisiting a region.

The decisive checks are three. First, the gain must localise to the boundary band. Second, learned weights must beat uniform averaging specifically on turbid clips. Third, the gated loss must not lower interior accuracy, unlike the ungated ablation. Inference cost equals that of the plain U-Net, which keeps the method within the 33 to 132 ms per frame range reported for challenge entries.