Adapting Real-Time Polyp Segmenters to a New Hospital with Frame-Level Tags Only

A polyp segmenter trained at one hospital often loses accuracy at another. Camera optics, lighting and bowel preparation differ, and lightweight real-time models are hit hardest. Fixing this today means drawing pixel masks at the new site, or bolting on adaptation machinery that slows the model below video frame rate. This note proposes a cheaper route. At the new hospital, collect only a yes/no tag per frame ("polyp visible" or not). Then fine-tune the segmenter's own output map so that it explains those tags. The core rule is simple: on frames tagged negative, every pixel must be background; on frames tagged positive, a small set of pixels must be confidently polyp. The model's architecture and inference pass are unchanged, so it runs exactly as fast after adaptation as before.

The bet is that the negative frames do most of the work. Lightweight segmenters such as RUPNet, a three-block residual encoder-decoder running at 152.60 frames per second (FPS), are reported to over-segment under specular glare and low-contrast mucosa. Those failures concentrate on site-specific background appearance. A frame tag costs seconds to assign and may be derivable from procedure logs. A polyp-free frame from the new hospital is a fully labeled example of that hospital's background. We expect this signal to sharpen boundaries on positive frames too, not just suppress false alarms on empty ones.

Core Claim

Frame-level tags from a target hospital, applied as a pooled presence loss on the segmentation map itself, recover a large share of the cross-center accuracy gap of a real-time segmenter at zero added inference cost. The improvement is driven mainly by the negative frames. It should therefore show up in pixel-level Dice on polyp-containing frames, not only in frame-level false-positive rate. If it shows up only in false-positive rate, the method reduces to "run a classifier and blank the mask," and we will say so.

Position Relative to Prior Work

Three papers motivate the gap.

Weakly Supervised Polyp Segmentation in Colonoscopy Images Using Deep Neural Networks shows that weak bounding boxes can nearly match pixel masks: 81.36% versus 81.52% Dice on Polyp-Box-Seg, at 45 FPS. It needs a box per polyp, however, and never tests on a different center.

TransNetR evaluates on the multi-center PolypGen benchmark at 54.60 FPS. It is fully supervised, and its authors acknowledge a persistent gap between in-distribution and out-of-distribution performance.

Weakly-Supervised Cross-Domain Adaptation for Endoscopic Lesions Segmentation is the closest prior work. It also adapts using image-level target tags (gastroscopy to enteroscopy, 64.43% mean intersection-over-union, mIoU). It does so with Wasserstein transferability scores, dual attention modules, superpixel priors and curriculum self-training, and reports no latency. We test whether almost all of that machinery can be replaced by a tag-explaining loss on an unmodified fast network. We also target a different shift: same procedure, different hospital.

Two papers constrain the design.

Adaptation of Weakly Supervised Localization in Histopathology by Debiasing Predictions shows what goes wrong without labels. Self-training under shift amplifies the model's initial bias until classification collapses to 50.0% on some CAMELYON-17 centers. That paper repairs the collapse with a selective unlearning loss. We avoid the problem differently: tags supply an external corrective signal, and we keep the source pixel loss as an anchor.

The Role of Pleura and Adipose in Lung Ultrasound AI shows that unmasked models latch onto cohort-specific background tissue. Its ROC area under the curve (AUC) rises from 0.6553 to 0.7742 once that background is masked out. This supports our hypothesis that background is where site shift bites.

Two papers supply the method.

HOW trains dense local features through a pooled global objective and then reads out the local features at test time ("train global, test local").

The Implications of Local Correlation on Learning Some Deep Functions gives the theoretical reason this can work: small local patches correlate with the global label. Here the "local features" are the segmentation logits themselves, so no separate readout is needed.

Research Program

Start with a fast segmenter trained with pixel masks on a source set: Kvasir-SEG, plus optionally CVC-ClinicDB. Let $z_i$ be the network's polyp logit at pixel $i$ and $\sigma$ the sigmoid. For a target frame with tag $y \in \{0,1\}$, define a presence score $s$ as the mean of $\sigma(z_i)$ over the $k$ highest-scoring pixels.

$$ s = \frac{1}{k}\sum_{i \in \text{top-}k(z)} \sigma(z_i), \qquad \mathcal{L} = \mathcal{L}_{\text{pix}}^{\text{src}} + \lambda\, \mathrm{BCE}(s, y) + \mu\, \mathbb{1}[y=0]\,\frac{1}{N}\sum_{i=1}^{N} \sigma(z_i) $$

Here $\mathcal{L}_{\text{pix}}^{\text{src}}$ is the usual Dice-plus-cross-entropy loss on source masks. $N$ is the number of pixels. $\lambda$ and $\mu$ weight the tag terms. The last term pushes the whole map toward zero on negative frames.

Top-$k$ pooling plays the role of the norm-ranked selection in HOW. A positive tag rewards a few confident pixels without forcing the polyp to fill the frame. We set $k$ to a small fraction of the frame, chosen on source data from the typical polyp area.

The study varies three things. The first is the number and ratio of target positive and negative tags, swept from a few dozen to the full center. This directly probes the low-data failure that the pathology foundation-model benchmark reports for weak supervision. The second is backbone speed class: RUPNet-sized versus a ResNet-50 encoder. The third is whether a light temporal smoothing of $s$ across adjacent tagged frames helps. We will keep it only if it costs nothing at inference.

Latency is measured end to end at 512×512 on a desktop GPU and on an embedded-class device. This follows the warning in the file that desktop FPS rarely matches clinical carts.

Baselines

The naive reference point is the source-trained segmenter applied to each target center with no adaptation. We report it for the lightweight network and for TransNetR, under the same preprocessing. For TransNetR, the strongest published number in this setting is Dice 0.8706 on Kvasir-SEG at 54.60 FPS. On PolypGen it reports only a relative 2.81% mIoU gain over the next model; no reported absolute per-center PolypGen result found on Lacuna for TransNetR. We therefore retrain it ourselves under our split.

The strongest fully supervised domain-generalization competitor is the cascaded double encoder-decoder of Galdran et al.. Its reported figure is 76.19% Dice on unseen ETIS-LaribPolypDB. With center-balanced sampling it reaches 81.81 ± 1.19 Dice across EndoCV 2021 centers (multi-center version). It roughly doubles compute per frame. We run it with the same source data and, as an upper-bound variant, with pixel masks from the adaptation split of the target center.

The strongest label-efficient domain-generalization result is LFDG, which reports 62.83% mIoU on PolypGen C1 and 55.87% on unseen CVC-ClinicDB. We compare against its CVC-ClinicDB number when our source excludes CVC. We also report FPS for its Vision Transformer backbone, which that paper does not give.

For weak-label adaptation, we reimplement the transferability-plus-pseudo-label pipeline of Dong and Cong on our backbone with the same tags. We also run an unlabeled self-training baseline in the style of SFDA-DeP's starting point.

All methods get the same source data, the same target adaptation frames, and an equal hyperparameter search budget of a fixed number of trials on a held-out source-like validation split. No target masks are used for tuning.

Beating the field means the following. The adapted lightweight model must exceed every same-speed baseline in mean target Dice on positive frames. It must close at least half the gap between "no adaptation" and "target pixel-mask fine-tuning." It must do this while staying at or above 30 FPS on embedded hardware.

Two ablations isolate the key ingredient. One uses positive tags only; the other uses negative tags only. A third, inference-time ablation is the classifier-gating baseline described below.

Why not just…?

Why not just run a small frame classifier and blank the mask on negative frames? This is the real threat. It cheaply removes false alarms on empty frames and costs only a tiny extra head. It cannot, however, change the mask on frames that do contain a polyp. Our claim is that negative frames reshape the decision boundary on positive frames. We therefore report Dice on positive frames separately from frame-level false-positive rate. If gating matches our method on positive-frame Dice, the contribution reduces to gating, and the note's claim is falsified.

Why not just self-train on unlabeled target frames? SFDA-DeP documents that pseudo-label self-training under shift amplifies the dominant-class bias until it collapses. Over-segmentation is exactly such a bias. Tags give an external check, and the unlabeled baseline in our grid measures how much they add.

Why not just annotate boxes at the new site? Box supervision is proven to work (Weakly Supervised Polyp Segmentation). It costs a drawn box per polyp, whereas a tag costs a single keypress per frame or comes from logs. We run box adaptation as a reference and plot Dice against estimated annotation seconds. If boxes win at equal annotation time, that is a reportable negative result.

Why not just use a bigger, more robust model? The cascade of Galdran et al. and TransNetR do generalize better. The first doubles latency, and its authors leave real-time deployment as future work. Our latency table measures whether a heavier unadapted model at the same speed budget beats an adapted light one.

Proposed Experiments

The main protocol treats each PolypGen center as a target in turn. Each center's frames are split by patient into an adaptation half, which is used with frame tags only, and a test half, which is evaluated with masks. CVC-ClinicDB and ETIS-LaribPolypDB serve as additional unseen targets. Negative frames are taken from PolypGen's released polyp-free frames where available, and per-center counts are reported. If some center lacks enough polyp-free frames, that is a stated limitation for that center, not something we patch with mask-derived crops.

Metrics are Dice and mIoU on positive test frames, frame-level false-positive rate on negative test frames, and the spread across centers. We also report FPS and per-frame latency on both devices.

Three outcomes would settle the question. First, the negative-only ablation should recover most of the full method's positive-frame Dice gain. Second, the full method should beat classifier gating on positive-frame Dice. Third, the gain should persist with only a few dozen tagged frames per center. If the first two hold, frame tags are a practical deployment tool for real-time segmenters. If they fail, we will have shown that tag-level supervision fixes alarms but not boundaries, which is also useful to know before clinics invest in collecting tags.