Correcting Slow Drift in Skip Connections for Real-Time Clinical Video Segmentation

Segmentation networks used during live procedures such as colonoscopy have to keep up with the camera, which means roughly 30 to 50 frames per second. They also have to stay accurate while the imaging conditions change underneath them: a new scope model, a different light source, lens fogging, or a different hospital's processing pipeline. Almost all of these networks are U-Net-like. A U-Net is an encoder-decoder that copies high-resolution early features straight to the decoder through "skip connections", and those copied features are what let it draw sharp boundaries around small lesions. But the same copied features are where acquisition changes do the most damage. Early-layer features mostly encode color, brightness and sensor texture, so a shift in the camera reaches the decoder almost unfiltered. Current test-time adaptation methods for medical images correct global normalization statistics or a single per-image style code. They work one image at a time, they need backpropagation, or they feed the model's own predictions back into the update. In a live video stream, that last property makes errors compound.

We propose a forward-only correction that runs only on the skip pathways and rests on two separations. The first is in time: acquisition drift is slow, while image content (a polyp appearing, the camera turning) changes fast. The second is in frequency: drift lives mostly in the smooth, low-frequency part of early features, while the boundaries the skip connection exists to carry live in the high-frequency part. Each skip feature map is split into a blurred low band and a residual high band. A slow running average of the low band's channel statistics, accumulated over the stream, is re-aligned to stored training-time statistics. The high band passes through untouched. The update depends only on the input, never on the model's output, so it cannot amplify its own mistakes. It stores a few moments per channel and no frames, and it should add on the order of a millisecond per frame.

Core Claim

Under gradual or abrupt acquisition drift in clinical video, re-aligning only the low-frequency statistics of skip-connection features, using a slow, prediction-independent running estimate, should recover most of the accuracy lost to drift. It should do so while preserving small-object and boundary accuracy better than adapting normalization statistics throughout the network. It should also stay stable over hour-long videos where output-driven adaptation degrades, all at essentially the latency of the frozen model.

The claim is falsifiable on three fronts. It fails if streaming re-estimation of all normalization statistics does just as well. It fails if correcting the input image's color and contrast does just as well. It fails if the high/low band split adds nothing over aligning the whole skip feature.

Positioning Against Prior Work

The file documents the gap from both sides. Neural illumination calibration for surgical workflow-optimized spectral imaging predicts illumination maps for surgical hyperspectral images and deliberately removes skip connections, so that only smooth illumination passes and tissue detail does not leak through. That paper motivates our frequency split, but it solves a calibration problem rather than segmentation, and it processes single cubes. On the opposite side, EFCNet and CaraNet show that dense multi-level skip routing is what recovers tiny targets. Neither runs in real time or adapts to shift. IDDF2018-ABS-0261 observes that specular reflections travel through a pruned U-Net unless they are filtered beforehand. We keep skip connections and decide what passes through them according to frequency band and timescale.

Three test-time adaptation methods supply the closest technique, and they are where we differ:

All three act on global statistics recomputed for each image. We act only on skip features, accumulate statistics across the stream, and split by frequency.

Two continual methods constrain the design:

  • MGIPT shows that output-driven continual adaptation such as TENT loses more than 5% Dice across rounds. Its prompt tuning avoids this, but it needs iterative passes that do not fit a 33 ms budget.
  • FreeVPS names the "snowball effect" in long colonoscopy videos and controls it with detection-and-tracking association on top of SAM2. It works at the mask level and adds 0 to 3 frames of lag.

Our correction is complementary to FreeVPS: it improves the per-frame detector that FreeVPS depends on. Finally, Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift found that training invariance in advance did not beat plain training on all historical data. That result motivates reactive correction at deployment instead of invariance learned at training time.

Research Program

The base networks are a frozen, trained video polyp segmenter with U-Net-style skips, in two forms. One is a reimplementation of PNS-Net, the 140 FPS normalized self-attention model. The other is a plain three-level U-Net in the style of IDDF2018-ABS-0261. Using both tests whether the effect depends on architecture.

The correction works as follows. At each skip level $l$ and frame $t$, the skip feature $s_t$ is split into a low band $L_t = B * s_t$, where $B$ is a fixed depthwise blur, and a high band $H_t = s_t - L_t$. Running per-channel moments of the low band, $\mu_t$ and $\sigma_t$, are updated by an exponential moving average with rate $\alpha$. The corrected skip feature is

$$\hat{s}_t = \sigma^{src}\,\frac{L_t - \mu_t}{\sigma_t} + \mu^{src} + H_t,$$

where $\mu^{src}$ and $\sigma^{src}$ are the low-band moments recorded on the training set.

The rate $\alpha$ sets the timescale that separates drift from content. We will relate it to the area of the frame typically covered by lesions: a small polyp moves the frame's mean statistics only slightly, so a time constant of several seconds averages it out while still following drift over minutes. We will also measure how much each skip level's statistics vary with acquisition compared with content. This tests the premise that shallow levels are acquisition-dominated and deep levels content-dominated, and the premise decides which levels to correct.

Two optional components are tested separately, so that they cannot carry the main result:

  • A saturation mask computed from the raw input, which suppresses the high band at near-white pixels to handle specular glare.
  • A reset rule, triggered when the low-band statistics jump faster than drift plausibly could (for example, a scene cut or the scope being withdrawn).

Evaluation uses four protocols:

  • In-domain to out-of-domain: train on SUN-SEG and evaluate out-of-domain on PICCOLO and the multi-center PolypGen, following the FreeVPS protocol.
  • Center-switching streams: concatenate PolypGen centers into continual streams, as in the MGIPT continual polyp setting.
  • Long videos: the 15 untrimmed 20 to 50 minute LU-VPS videos test long-horizon stability.
  • Controlled drift: apply gradual, physically motivated drift schedules to SUN-SEG test videos, namely illumination color temperature, gain and white-balance change, defocus, and added sensor noise. We use these because the file contains no colonoscopy video set with multi-year acquisition dates. We label this protocol as simulated drift, not real longitudinal drift.

Small-object accuracy is reported separately on ETIS and on polyps under 5% of frame area. CVC-300-TV and CVC-612-V check that nothing breaks in-domain.

The metrics are:

  • Dice and IoU.
  • Small-object Dice.
  • A boundary F-score, because the claim concerns boundary detail specifically.
  • Temporal consistency, as used by FreeVPS.
  • Dice as a function of time within long videos, which is the snowball test.
  • Measured per-frame latency and memory on an RTX 2080-class GPU and an embedded GPU, at batch size one.

Baselines

The naive reference is the frozen source model with no adaptation, run on identical streams. Every method is measured by how much of the gap between this reference and an oracle it closes. The oracle is the same model fine-tuned on labeled target-center data; it is an upper bound, not a competitor.

The strongest real-time competitor on this exact question is streaming normalization re-estimation. It keeps the same running average as our method but applies it to every batch-norm layer in the network, with no frequency split. It matches our compute, needs no backpropagation and accumulates across frames, so if it equals our method the skip-specific claim fails. Its momentum will be tuned on the same held-out center that tunes our $\alpha$, with the same number of tuning trials.

The second competitor is the Adaptive UNet from On-the-Fly Test-time Adaptation, retrained on the same SUN-SEG source data. It reports 70.19 Dice on CHASE→HRF vessel segmentation, against 61.95 for source-only and 61.20 for TENT. No reported result on polyp video was found on Lacuna for it.

InTEnt (Medical Image Segmentation with InTEnt) represents single-image normalization adaptation. It reports a 71.6% average Dice over 24 cross-site splits at about 0.06 s per image.

TENT represents output-driven continual adaptation. Its failure mode is documented by MGIPT, which reports more than a 5% drop across rounds, against 72.28% DSC for MGIPT on OD/OC and 80.64% DSC for MGIPT on its continual polyp benchmark. MGIPT itself will be run as an accuracy reference without a latency constraint, using its continual polyp setting.

For long videos, the strongest existing system is FreeVPS, which reports F1 63.0, Dice 68.2 and temporal consistency 0.747 on LU-VPS. We will run it twice: with the frozen detector, and with our corrected detector plugged in.

On the PolypGen and center-switching streams, beating these baselines means a statistically significant Dice gain over streaming whole-network re-estimation, with no loss in small-object or boundary scores, at less than 2 ms of added latency.

Three ablations isolate the ingredients. The first applies the same alignment to the whole skip feature, without the band split. The second applies it to the low band of every layer rather than only the skip levels. The third replaces the stream-level running average with statistics from the current frame alone, which isolates the timescale separation.

Why not just…?

Why not just re-estimate all batch-norm statistics with a running average? This is the most serious threat. It is cheap, needs no labels, and accumulates across the stream just as our method does. Our argument against it is mechanistic. Deep-layer statistics move when content moves: a large polyp filling the view, or the camera pressed against the colon wall. A running average shared across all layers will therefore partly "correct" content changes away. Adapting low-level statistics globally also smears the boundary features, as the file notes for global batch-norm updates in the shape-dictionary method. The ablation that restricts correction to skip levels, together with the per-level variance analysis, settles this directly. If the two methods tie on boundary F-score and small-object Dice, we will report that skip-specific correction is unnecessary.

Why not just normalize the input image (color constancy, contrast equalization, reflection filtering)? Input preprocessing is the cheapest fix, and for pure color casts it may be enough. The evidence against it as a general fix is that Bordner et al. still saw large drops on follow-up scans despite smoothing, contrast equalization and median filtering. Likewise, Neural illumination calibration found that statistical calibration could not handle non-uniform stray light, which degraded segmentation by over 20%. Input normalization also cannot correct changes in noise or blur that only show up as shifted feature statistics. We include stream-level gray-world plus contrast equalization as a baseline. The defocus and noise drift schedules are where we expect the two to separate, and if they do not, we will say so.

Why not just wrap the frozen model in FreeVPS-style tracking? Tracking suppresses transient false positives and drift in the mask memory. It cannot recover a lesion that the shifted detector never fires on, which is a limitation the FreeVPS authors themselves acknowledge, and it adds up to three frames of lag. Our method targets that detector-level miss. The combined run on LU-VPS shows whether the two are complementary or redundant.

Why not just use continual prompt tuning or TENT? These methods learn from the model's own outputs. That ties the update to the model's errors, so on long streams the errors accumulate: MGIPT shows TENT losing more than 5%. MGIPT avoids the collapse, but it needs iterative passes for each image. Our Dice-over-time curves on the 20 to 50 minute LU-VPS videos test whether an input-only update avoids the collapse while staying within the frame budget.

Decisive Checks

The idea is supported only if all four of the following hold.

First, on PolypGen center-switching streams and on simulated gradual drift, skip-only low-band correction beats streaming whole-network re-estimation on small-object Dice and boundary F-score, not just on overall Dice.

Second, the band-split ablation shows that aligning the full skip feature loses boundary accuracy relative to the split version.

Third, Dice over time on LU-VPS stays flat across each full video, while TENT's declines.

Fourth, measured added latency stays under 2 ms per frame on the PNS-Net base, keeping it above 30 FPS on embedded hardware.

A further diagnostic tests the premise behind the frequency split. We measure the fraction of acquisition-driven variance in the low band versus the high band, at each skip level, under real center changes. If drift turns out to sit mostly in the high band (for example, sensor noise), the correction should move to a different band, and we will report that rather than defend the design.