Running Temporal Moments as Skip Connections for Video Segmentation

Segmenting objects in medical video, such as polyps in colonoscopy footage, forces a compromise. The network needs sharp, pixel-accurate boundaries, which come from the high-resolution "skip connections" that carry early, detailed feature maps straight to the decoder. It also needs memory of what the scene looked like over the last several seconds, which is what separates a camouflaged lesion from the wall around it. Current methods get memory either from a short window of raw frames or from a bank of stored full-resolution feature tensors. Both are expensive, and both reach back only a few frames.

Time-series practice offers a cheaper tool. Clinical and sensor pipelines summarize a sliding window with a few running statistics, such as mean, variance and trend, computed at several window lengths at once. We propose to keep those statistics per pixel, inside the skip connections. At each resolution level the network would update, every frame, the running mean and variance of its own features at three timescales (roughly 3, 15 and 60 frames). The decoder would then receive, next to the current frame's detail features, maps saying how this frame deviates from its recent and longer history. This costs a fixed handful of feature maps no matter how long the lookback, which lets a real-time model use a minute-scale history that stored-frame methods cannot afford.

Core Claim

Explicit multi-timescale running moments at every skip level beat three alternatives for fine boundaries on hard video segmentation cases: a learned collapse of a short frame window, a stored-feature memory bank, and a learned recurrent skip state. The hard cases are indefinable boundaries, small objects and fast motion. The advantage should come at constant memory and near-2D speed. The mechanism has two parts. Deviation from a long-timescale mean acts as a per-pixel "novelty" signal that marks a low-contrast lesion against its slowly varying surroundings. Short-timescale variance marks pixels that are unstable because of motion blur or specular highlights, so the decoder can learn to distrust their skip features.

For a feature map $f_t$ at one skip level and frame $t$, with decay rate $\alpha_k$ for timescale $k$, we keep

$$m_k(t) = (1-\alpha_k)\,m_k(t-1) + \alpha_k f_t,\qquad v_k(t) = (1-\alpha_k)\,v_k(t-1) + \alpha_k\,(f_t - m_k(t-1))^2,$$

where $m_k$ and $v_k$ are the per-pixel, per-channel running mean and variance. The effective window is about $1/\alpha_k$ frames, which follows from the geometric decay of the weights. The decoder at that level receives the current skip features concatenated with the deviations $f_t - m_k$, the spreads $\sqrt{v_k}$, and the trend $m_{\text{short}} - m_{\text{long}}$, all on channel-reduced features, followed by a learned gate. Memory per level is fixed at $2K$ reduced maps for $K$ timescales.

Relation to Prior Work

The closest work is SVS-net. It segments coronary vessels from a fixed 4-frame window with a 3D encoder and collapses time inside each skip connection using a $4\times1\times1$ convolution, reporting an F-measure of 0.8428. That design motivates ours: it already shows that the skip connection is the right place to inject time, and that going from 2D to 3D features helped by about 2.5%. But a linear temporal collapse over four frames cannot express variance, and it cannot look back further without growing the 3D encoder. We change what crosses the skip, and at what timescale.

Three video polyp methods mark the current limits of temporal memory. PNS-Net and PNS+ restrict self-attention to a short sliding neighborhood plus a first-frame anchor. HFSTI-Net keeps an explicit memory bank of full feature tensors and masks, read by cross-attention. WavePolyp compares only adjacent frames. The benchmark in PNS+ reports that indefinable boundaries and small objects remain the worst attributes, which is the failure we target. The test-time implicit network of Video Polyp Segmentation using Implicit Networks gains 2–3% on SUN-SEG-Hard. However, it needs per-video optimization and optical flow, which constrains us to stay feed-forward and flow-free.

The time-series side supplies the method. Latent Instructions as Context Surrogates shows that about ten window statistics, turned into prompt tokens, can stand in for thousands of raw context points in a frozen forecaster at lower compute. We borrow the idea that statistics can replace raw history, but apply it per pixel inside a dense decoder rather than as global tokens for a univariate forecaster. Amrollahi et al. motivates going beyond means to variance and volatility. Kurasawa et al. and Damgaard et al. motivate parallel short, medium and long lookbacks for separating acute from baseline change.

The spatial side constrains the design. FC-DenseNet reminds us that caching dense multi-scale features is the memory bottleneck, so the statistics must be channel-reduced. Fast Vision Mamba found that exactly where the skip sits relative to pooling decides whether fine detail survives. We therefore add the statistics beside the current-frame skip features, never in place of them.

The strongest competing design is the recurrent skip of Verma et al., which places bidirectional recurrent networks inside 1D encoder-decoder skips. A 2D convolutional recurrent version is the natural learned counterpart to our fixed moments, and we treat it as a central baseline rather than as prior art we have surpassed.

Research Program

We build a single-frame 2D encoder-decoder on the same pretrained backbone used by PNS+, so that speed comparisons are meaningful. We then add the moment state at each of the three or four skip levels. Frames are processed causally in their natural order, because a live colonoscopy has no future frames.

Camera motion is the obvious risk. Per-pixel statistics over 60 frames are meaningless if the endoscope swings. We address this in two ways. First, long-timescale moments are kept only at the coarser skip levels, where a few pixels of motion matter less. Second, a motion-adaptive variant raises $\alpha_k$, and so forgets faster, when a cheap global frame-difference score spikes. Whether either is enough is an empirical question the experiments answer directly.

Training uses truncated sequences of about 64 frames. Gradients flow through the moment updates, which are differentiable, and the running state starts from the first frame's features. The loss is Dice plus binary cross-entropy. We will also try a boundary-weighted term in the spirit of the variation-weighted loss of Verma et al., as an ablation rather than a core ingredient.

The analysis component asks what the decoder actually uses. We will visualize the learned gates on each statistic and correlate short-scale variance maps with annotated specular highlights and blur. We will also measure whether long-scale deviation maps light up polyp regions before the frame-only prediction does.

Baselines

The naive reference is the same 2D encoder-decoder with plain skips and no temporal state, trained identically. Every temporal gain is measured against it.

The strongest reported competitor on SUN-SEG-Hard is WavePolyp, which reports a Dice of 87.55% and an S-measure of 90.28 at 23.04 FPS on an RTX 3090. It also reports 87.36% Dice when trained on SUN-SEG and tested on CVC-612 without fine-tuning. HFSTI-Net reports 86.27% Dice on SUN-SEG-Hard at 31.27 FPS and represents the memory-bank approach. PNS+ reports 0.756 Dice on SUN-SEG-Easy Unseen at 170 FPS on a V100 and represents fast short-window attention. These reported numbers come from different subset definitions and hardware. We will therefore retrain all three from public code where available, on our exact split, on one GPU type, with the same augmentation and a matched hyperparameter search budget of about 20 trials each. We quote the published numbers only as a sanity check on our reproduction.

Beating the strongest baseline means one of two outcomes. Either the model matches WavePolyp's Dice on SUN-SEG-Hard within 0.5 points while running at least twice as fast with lower memory, or it exceeds WavePolyp on the indefinable-boundary and small-object attributes at comparable speed.

Two ablations isolate the key ingredient. The first is a learned-collapse skip in the style of SVS-net: a temporal convolution over the last $N$ reduced feature maps, with $N$ chosen to use the same memory as our moment state. The second is a convolutional GRU inside each skip, following Verma et al., with a hidden state of the same channel count. We further ablate means-only against means plus variance, single timescale against three, and fixed against motion-adaptive decay.

Why not just…?

Why not just feed a longer window of raw frames into a 3D U-Net? In the SUN-SEG study of PNS+, a 2/3D baseline reached 0.722 Dice against PNS+'s 0.756. PNS-Net also documents that 3D and recurrent models are too slow for live use. Compute for a 3D window grows linearly with window length, while our state is constant. The learned-collapse ablation is a memory-matched version of this idea, so it tests the question directly.

Why not just segment frame by frame and smooth the masks over time? A moving average of masks can only reuse information that the per-frame model already found. HFSTI-Net attributes frame-wise failures to shape collapse and lost tracking, which averaging cannot undo, and averaging thin boundaries blurs them further. We run this as a baseline: the naive 2D model with an exponential moving average on output probabilities at the same three timescales. It tells us whether statistics help because of where they are injected (the skips) or merely because they smooth.

Why not just use a learned recurrent skip, since it also has constant memory? This is the real threat. A convolutional GRU can in principle learn our moments and more. Our bet is that fixed multi-timescale averages hold information over 60 frames reliably, whereas gated recurrences trained on truncated sequences tend to learn short memories. The bet also includes that fixed statistics transfer better across endoscope hardware. If the recurrent skip matches us in-domain and on CVC-612, the claim collapses to "any constant-memory state in the skip helps," and we will report it that way. The attribute breakdown and a probe of effective memory length will show which kind of state each model actually carries: we will perturb a frame and measure how long its effect persists in later predictions.

Why not just use a memory bank or long-context attention? HFSTI-Net stores full tensors and runs at 31 FPS. Latent Instructions as Context Surrogates shows in 1D that raw context length gives diminishing returns at steep cost. We will run HFSTI-Net with its memory length swept, plotting Dice against memory and latency. The comparison is on that curve, not at one operating point.

Proposed Experiments

The primary benchmark is SUN-SEG, reporting Dice, S-measure and weighted F-measure on the Easy and Hard subsets with Seen and Unseen splits, broken down by the benchmark's attributes. The decisive attributes are Indefinable Boundaries, Small Objects and Fast-motion. Generalization is tested by training on SUN-SEG and evaluating without fine-tuning on CVC-612 and CVC-300-TV. We also measure temporal stability as the mean absolute change in predicted foreground probability between consecutive frames inside ground-truth-stable regions, along with FPS and peak GPU memory at 352×352 input on one RTX 3090.

Three outcomes would decide the proposal. First, if long-timescale moments do not beat the single short timescale on Indefinable Boundaries, the multi-scale lookback idea fails for endoscopy. Second, if the Fast-motion attribute degrades relative to the 2D baseline, pixel-wise history is being corrupted by camera motion, and the motion-adaptive decay must rescue it or the approach is limited to steadier footage. Third, the comparison against the convolutional GRU skip at matched memory decides whether explicit statistics carry anything a learned state does not. A positive result on all three, at more than 100 FPS, would show that a cheap time-series summarization idea gives dense video models a long memory without storing frames.