Certified Memory Gates for Real-Time Semi-Supervised Polyp Video Segmentation

Video segmentation models used in live colonoscopy have to run at video frame rate. They are also trained with only a few expert-annotated frames, because labeling every frame is too expensive. The fastest semi-supervised video models resolve this by keeping a memory of past frames: the model writes its own predictions into the memory and reads them back to segment later frames. That loop has a weak point. A decision rule called the gate admits or rejects each frame for the memory, and today it is either a hand-built continuity check or the network's raw confidence. Neural networks are often most confident exactly when they are wrong. So one overconfident bad frame, such as a motion-blurred one, a specular flash, or a tool occlusion, can contaminate the memory and drag down every frame that follows.

This note proposes treating the gate as a calibrated selective-prediction problem. A tiny head attached to the segmenter predicts, in a single pass, how good the current frame's mask is, meaning its expected overlap with the truth. The admission threshold on that prediction is then chosen on held-out labeled clips so that the rate of wrongly admitted frames has a finite-sample bound. The same number tells the clinician when not to trust the overlay. The project first asks whether gate quality matters at all, using an oracle gate. It then asks whether a learned, calibrated estimate beats cheaper signals, namely temperature-scaled entropy and raw cross-frame disagreement, without losing real-time speed.

Core Claim

In memory-based semi-supervised video segmentation, accuracy and reliability both depend on how well the model ranks its own frames by quality. Per-pixel calibration matters much less. A frame-level error estimator can fix this ranking. It is trained on cross-fitted labeled frames, so it sees the segmenter's real errors instead of its near-perfect training-set fits, and it is thresholded on held-out clips with a statistical guarantee. The claim is that this estimator reduces memory contamination, improves accuracy on hard clips, and supplies a usable deferral signal, all at under a millisecond of added cost per frame.

The idea is narrower than "add uncertainty to a video model," and deliberately so. If an oracle gate that knows the true frame quality does not improve on the existing heuristic gate, the premise is false, and the project says so early.

Where This Sits

The closest work is VPSentry. It builds a long-term memory of foreground and background feature summaries ("prototypes") for semi-supervised polyp video segmentation. A "Sentry" gate, which checks visual continuity between frames, keeps corrupt pseudo-labels out of that memory. VPSentry reaches 83.95 Dice on SUN-SEG-Hard with 10% labeled frames at 28–30 FPS on an RTX 4090. Its gate is a heuristic with no calibrated meaning, and its masks carry no reliability estimate. We keep its architecture and replace only the gate.

DMNet caches past frames in a global memory using raw predictive entropy as the selection signal, and runs at 38 FPS on surgical video. It is fully supervised and never asks whether that entropy is trustworthy. Our work tests exactly this question.

Two works motivate the concern that self-training amplifies overconfidence. On the Calibration of Image Semi-Supervised Learning Models shows that pseudo-label training drives models toward confident errors. Its CalibrateMix reduces FixMatch's calibration error by 8.51% on CIFAR-100, but only for static classification. Evidential Prototype Learning masks unreliable voxels using Dirichlet-based uncertainty before building prototypes in 3D semi-supervised segmentation. It shows that uncertainty-filtered prototypes help, but its fusion machinery is heavy and was never timed.

The single-pass uncertainty methods supply our cheapest strong competitor. Region-based evidential deep learning trains a network to output Dirichlet "evidence" under a Dice-based risk and gets uncertainty about 20 times faster than Monte Carlo dropout (3.38 s versus 63.68 s per MRI volume). Uncertainty in Real-Time Semantic Segmentation on Embedded Systems propagates mean and variance analytically through a probabilistic final layer and runs at about 57 FPS on a Jetson AGX Xavier. Both give pixel-level uncertainty. Neither is designed or evaluated as a ranking signal for memory admission.

Conformal methods supply the thresholding technique and constrain it. COMPASS and Kandinsky Conformal Prediction both show that coverage guarantees for segmentation become informative only when the score adapts to the instance. COMPASS roughly halves interval width compared with plain split conformal prediction on Kvasir polyps. Both assume dense calibration labels and independent images.

Progressive Bayesian Inference is a useful foil. It uses Bayesian reasoning to tame pseudo-labels during scribble training, then discards it, and ships a fast but blind deterministic model. We argue the uncertainty signal should survive to deployment, because in streaming video it keeps acting on the memory.

Research Program

The segmenter is VPSentry's backbone, prototype memory and propagation module, unchanged. We attach a quality head: a two-layer network whose inputs are already computed at each frame. These are pooled decoder features, summary statistics of the pixel entropy map, the similarity of the current frame's prototypes to the memory bank, and the disagreement between the current mask and the mask propagated from the previous frame. The head outputs $\hat{q}_t$, a prediction of the Dice score of frame $t$'s mask. Its cost is a few thousand multiply-adds on features that already exist, so the latency budget is essentially untouched. We will measure this rather than assume it.

Training the head honestly is the crux. On the labeled frames the segmenter was trained on, its masks are nearly perfect, so a head trained there learns that everything is fine. We therefore cross-fit. The labeled clips are split into $K$ folds. Segmenter checkpoints trained without a given fold produce masks for that fold, and the head regresses those masks' true Dice. We also expand the error distribution by corrupting held-out labeled frames with blur, specular highlights and synthetic occluders. SUN-SEG's ten visual-attribute tags let us check that these corruptions resemble real hard cases. A cheaper variant holds out a quarter of the labeled clips instead of cross-fitting, and it will be compared against the $K$-fold version.

The gate admits frame $t$ into memory only if $\hat{q}_t \ge \lambda$. A frame counts as a false admission if its true Dice falls below a quality floor $\delta$. On calibration clips that the head never saw, we choose

$$\hat{\lambda} = \min\{\lambda : U_{n}(\lambda) \le \alpha\},$$

where $U_n(\lambda)$ is a one-sided binomial upper confidence bound on the false-admission rate among admitted calibration frames at threshold $\lambda$, $n$ is the number of calibration clips, and $\alpha$ is the tolerated rate. Frames within a clip are strongly correlated, so the guarantee is stated, and the bound computed, at the clip level. This is the main honest weakness: the guarantee holds only if deployment clips are exchangeable with calibration clips. We will report how far it breaks on the SUN-SEG Unseen split.

At inference, the same $\hat{q}_t$ drives a clinician-facing signal. Frames below a second, deferral threshold are marked as unreliable. This is evaluated as selective prediction: the risk–coverage curve of Dice against the fraction of frames shown.

Baselines

The naive reference is VPSentry with its gate disabled, so that every frame is written to memory. It shows how much any gating buys. The strongest existing method is VPSentry with its original continuity-based Sentry gate, at its reported 83.95 Dice on SUN-SEG-Hard under 10% labels (VPSentry). We retrain it ourselves with the authors' recipe on the same RTX 4090 class of hardware, and beating it means a statistically reliable Dice gain on SUN-SEG-Hard across at least three seeds at no worse than its 28–30 FPS. For context, PSDNet sits at 81.11 under the same protocol and PNS+ reaches 0.756 Dice on SUN-SEG-Easy (Unseen) at 170 FPS with full supervision (Video Polyp Segmentation: A Deep Learning Perspective). I found no reported calibration or frame-quality-ranking result on SUN-SEG on Lacuna, so those comparisons are new measurements.

Three cheaper gates follow, all using VPSentry's architecture and memory. The first is DMNet-style gating on mean pixel entropy (DMNet). The second is the same signal after temperature scaling fitted on the held-out labeled clips. The third is mean evidential uncertainty from a head trained with the Dice-risk evidential loss of Region-based evidential deep learning, applied to the labeled frames. A fourth is raw cross-frame disagreement alone, gated directly. Every gate gets its threshold tuned on the same calibration clips with the same search budget.

Fairness on labels matters. Our method spends some labeled clips on the head and on calibration, so every baseline may use those clips for segmenter training instead. Total expert labels are therefore identical. The oracle gate, which admits frames by their true Dice, uses test labels purely as an upper bound.

Two ablations isolate the key ingredient. One trains the quality head without cross-fitting, on the segmenter's own training frames. The other uses the learned head with a hand-tuned rather than certified threshold. The first tests whether honest error exposure matters; the second tests whether the guarantee costs accuracy.

Why not just…?

Why not just temperature-scale the softmax and gate on entropy? This is the most serious threat, because it is free at inference. A single temperature rescales confidence globally but cannot reorder frames, and CalibrateMix shows that self-training corrupts the features themselves, not just the logit scale. That argument is indirect, though. The direct test is ranking quality: the area under the risk–coverage curve for frame Dice. If temperature-scaled entropy matches the learned head there, we drop the head and report that the cheap signal suffices.

Why not just gate on cross-frame disagreement, which needs no training? Disagreement is available in a streaming setting at almost no cost and catches abrupt failures. However, it is blind to errors that persist smoothly. An overconfident false positive on a fold of mucosa is stable across frames and passes a continuity test, and that is precisely the failure that pollutes memory. Disagreement is therefore an input to our head and also a standalone baseline. The attribute-sliced results on SUN-SEG will show whether the learned head adds anything beyond disagreement on slow-motion clips.

Why not just use Monte Carlo dropout or an ensemble as the gate? These are better-founded uncertainty estimates. But the gate runs online at every frame, and 10–30 stochastic passes would push VPSentry's 28–30 FPS toward single digits; the 20-fold slowdown measured by Region-based evidential deep learning shows the scale. We run a five-pass ensemble gate offline as a quality reference only. If it far exceeds every single-pass gate, distilling it into the quality head becomes the natural next step.

Why not just discard uncertainty at test time, as in Progressive Bayesian Inference? In a static image model, uncertainty that is discarded after training costs nothing at deployment. In a memory model, the gate keeps writing to memory during deployment, so removing it reintroduces the contamination loop. The no-gate baseline measures this cost directly.

Proposed Experiments

The primary benchmark is SUN-SEG with 10% and 20% labeled frames, the protocol used by VPSentry. Results are reported on the Easy and Hard subsets and on Seen and Unseen splits, and broken down by SUN-SEG's visual attributes. The secondary benchmark is EndoVis17 instrument segmentation with labels subsampled to 10%. DMNet's fully supervised 61.03 mDice at 38 FPS gives the ceiling there.

The metrics are Dice and FPS on an RTX 4090. We also report FPS on a Jetson-class device, since speeds measured on desktop GPUs often degrade on embedded hardware. For the gate itself, we measure the realized false-admission rate on test clips against the target $\alpha$, the calibration error of $\hat{q}_t$ against true Dice, and the risk–coverage area for deferral.

The decisive check comes first and is cheap. It compares VPSentry with the oracle gate against VPSentry with its own gate. If the oracle gains less than about one Dice point on SUN-SEG-Hard, memory admission is not the bottleneck and the project stops. If the oracle gains substantially, the second check is where the learned head falls in the gap between the Sentry gate and the oracle, compared with temperature-scaled entropy and with disagreement alone. The third check is whether the certified threshold keeps the realized false-admission rate at or below $\alpha$ on Unseen clips. Failure there would be a useful negative result about clip-level exchangeability in colonoscopy.