Spending the Parameter Budget on Time in Clinical Video Segmentation

Small segmentation models for medical video usually process each frame independently. When a frame is hard, because the heart has contracted to its smallest size, an acoustic shadow covers the wall, or a polyp sits behind glare, the model has nothing to fall back on. Most lightweight designs try to fix this by building a better per-frame encoder. We propose the opposite allocation. Keep the per-frame encoder deliberately tiny, a few hundred thousand to a couple of million parameters, and spend the rest of the budget on a cheap memory that carries evidence forward from easier frames.

Our central bet is that, at a fixed parameter count, moving capacity from space to time pays off most exactly where clinical data are most imbalanced. That means small targets, end-systolic frames, and boundary pixels. We pair this with a training loss that balances foreground and background across the whole clip rather than frame by frame, so that the rare hard frames are not drowned out. If the bet is right, a sub-3M-parameter model should close most of the gap to 35M-parameter video segmenters on the frames that matter clinically.

Core Claim

The claim has three parts. First, under a matched parameter budget, a tiny convolutional frame encoder plus a linear-time recurrent memory beats a larger frame-by-frame network. Second, the margin grows on minority regimes rather than on average Dice. Third, the margin is further increased by mining hard background pixels across the clip instead of within each frame.

The closest existing system is GDKVM. It segments echocardiography video with a key-value memory updated by a gated delta rule, meaning a memory matrix that is partially decayed and partially overwritten at each frame according to learned, input-dependent gates. Its cost grows linearly with clip length, but it uses 35.2M parameters and optimizes ordinary soft losses. We reuse its memory mechanism. What we test is different: whether that memory lets the spatial encoder shrink by an order of magnitude, and how imbalance-aware training interacts with memory.

Research Program

Architecture. The frame encoder is a shallow two-branch convolutional network. A full-resolution detail path and a narrow low-resolution context path follow the bilateral design of ContextNet and Echo-BiSeNet. Echo-BiSeNet applied that design to single echo frames at 13.7M parameters, and its authors name temporal consistency as the missing piece. We thin the channels further using Ghost modules, which produce half the feature maps by cheap linear transforms of the other half, as in MSGU-Net, together with 1×1 decoder convolutions as in Light-Weight RefineNet.

The low-resolution context tokens feed a gated delta-rule memory taken from GDKVM. This memory supplies global, long-range reasoning over time, while convolutions handle local reasoning within a frame. That is the local-global split of hybrid convolution-transformer designs such as UNETR and TransBTS, but placed along the time axis. We avoid full spatial self-attention because vision transformers overfit small clinical datasets without heavy pre-training (A Guide to Robust Generalization). We also avoid fixed low-frequency spectral mixing of the kind used in S2M-Net. Its 32×32 frequency cutoff throws away the high frequencies that encode the boundaries on which video segmenters fail. PNS+ documents that its weakest subsets are indefinable boundaries and small objects. Spectral mixing appears only as an optional ablation.

Training objective. Xu et al.'s Balance Loss supplies the loss. It shows that millions of tiny soft background probabilities add up to a false-positive penalty that disappears once predictions are thresholded at test time. It corrects this by keeping only as many of the hardest background pixels as there are foreground pixels. It also decays the weight on easy interior pixels so that training focuses on boundaries.

That loss was defined for single images. In video, a per-frame quota forces every frame to contribute equally many negatives, even though hard negatives cluster in a few frames: shadowed frames, glare frames, and end-systole, when the target is smallest. We therefore pool the quota over the clip:

$$ \mathcal{L}_{\text{clip}} = \frac{1}{|F|}\Big(\sum_{p\in F}\ell(p) + \sum_{p\in \mathrm{Top}_{|F|}(B)}\ell(p)\Big) $$

The terms are as follows. $F$ is the set of foreground pixels over all labeled frames of the clip. $B$ is the corresponding set of background pixels. $\mathrm{Top}_{|F|}(B)$ is the $|F|$ background pixels with the highest predicted foreground probability anywhere in the clip. $\ell$ is the per-pixel cross-entropy.

The interior-decay term is kept unchanged, and so is the original two-stage schedule. We do not claim to fix that schedule; we report its sensitivity. On echocardiography, where only end-diastole and end-systole are labeled, the clip pool covers just two frames. Even so, it lets the contracted end-systolic frame claim more negatives when that frame is the harder one. On densely labeled polyp video, the pool spans the whole clip.

Analysis. The main analysis is a budget sweep at roughly 0.5M, 1M, 3M and 10M total parameters. At each budget we compare the same encoder family used frame-by-frame, with all parameters spent on width and depth, against the same family with a fraction moved into memory. Results are broken out by minority slice:

  • end-systolic versus end-diastolic Dice on echocardiography;
  • small-object and indefinable-boundary subsets on SUN-SEG;
  • boundary F-score and 95th-percentile Hausdorff distance on all datasets.

We also measure frame-to-frame mask consistency. No standard temporal metric exists for these benchmarks, so we report the Dice between consecutive predicted masks against the same quantity on ground-truth interpolations where available.

Baselines

The naive reference is a frame-by-frame network at matched parameter count: our encoder with the memory removed and its parameters returned to encoder width. Two published per-frame points sit alongside it. A plain U-Net reached DSC 0.8612 at 28.9M parameters on EchoNet-Dynamic frames. Echo-BiSeNet reached DSC 0.9257 at 13.7M (Echo-BiSeNet). Beating our own matched frame-wise model at every budget, with the largest gap on end-systolic frames, is the minimum result for the core claim.

The strongest existing method is GDKVM. It reports mDice 95.11% on CAMUS (256×256, ten uniformly sampled frames per video) with LVEF Pearson correlation 0.904, and mDice 93.46% on EchoNet-Dynamic (128×128) (GDKVM). We adopt its sampling and resolution protocol exactly, use its released numbers, and rerun it with our loss for a loss-only comparison.

Further echo reference points are:

  • PKEcho-Net, a decoder-free model that reads masks directly from encoder features through region proxies: DSC 94.2 on CAMUS with ResNet-50, at 258 FPS.
  • MemSAM, which adapts Segment Anything with a space-time memory: mDice 92.78% and LVEF correlation 78.92% on EchoNet-Dynamic.

On polyp video, the strongest reference is HFSTI-Net, which mixes Fourier and spatial features on a PVTv2 backbone with a mask-guided memory bank: Dice 86.27% on SUN-SEG Hard at 31.27 FPS. PNS+ reaches 0.756 Dice on SUN-SEG-Easy Unseen at 170 FPS.

"Beating" has different meanings for these baselines. Against GDKVM and HFSTI-Net, a sub-3M model need not win on mean Dice. It must come within about one Dice point on average, and match or exceed them on end-systolic frames and on the small-object and boundary subsets, while running faster. Against PKEcho-Net, at comparable speed, it should win on boundary metrics.

For the loss, we compare five options at an identical tuning budget of a fixed number of trials per loss on a shared validation split:

  • Dice plus cross-entropy;
  • focal loss;
  • Tversky loss;
  • per-frame Balance Loss;
  • clip-level Balance Loss.

Balance Loss's published gain of KiTS19 tumor DSC 66.8 versus 59.8 for cross-entropy shows the size of effect imbalance-aware losses can have. For CVC-ClinicDB and BUSI, the file reports no number on Lacuna.

Why not just…?

Why not just use a bigger frame-by-frame network? The budget sweep tests this directly. If matched-parameter frame-wise models track the memory model on end-systolic and small-object slices, the core claim is false and we will say so. The existing evidence points the other way. PNS+'s spatiotemporal model beat per-frame ACSNet, 0.756 versus 0.713 Dice. GDKVM's ablation without its gated memory lost about 1.7 Dice points. Neither result was obtained at small budgets, though, which is the gap we fill.

Why not just compute Dice per frame, which already normalizes for target size? This is the most serious threat to the loss contribution. Per-frame Dice does equalize end-systolic and end-diastolic frames, and per-frame Balance Loss already removes the soft-versus-hard bias inside each frame. Clip-level mining adds value only if hard negatives are genuinely unevenly spread across frames. We test that directly by measuring where the top-ranked negatives fall over training, and by running a head-to-head comparison against per-frame Dice and per-frame Balance Loss. If clip pooling gives no gain beyond noise on SUN-SEG, where many frames are labeled, we drop it and keep per-frame Balance Loss as a standard component.

Why not just adapt a foundation model with small adapters? MemSAM does exactly this and trails both GDKVM and PKEcho-Net on the reported numbers. It also inherits a failure mode: a shadowed first frame corrupts the whole clip. We include a robustness test that degrades the first frame of each clip with synthetic shadowing. A gated memory that can decay a bad early entry should degrade less than first-frame-anchored propagation.

Why not use 3D convolutions or ConvLSTM for time? The 2/3D convolutional baseline in PNS+ reached 0.722 Dice versus 0.756 for attention-based temporal modeling. 3D kernels also multiply parameters by the temporal kernel size, which defeats the budget. We include a ConvLSTM variant at matched parameters in the sweep so the advantage, if any, is measured rather than assumed.

Proposed Experiments

We use EchoNet-Dynamic and CAMUS under GDKVM's protocol, SUN-SEG Easy and Hard, and CVC-612. CVC-612 has no reported HFSTI-Net number on Lacuna, so we report our own reruns there. We train three seeds per configuration and report means with confidence intervals, along with frames per second on one fixed GPU.

The decisive checks are three:

  1. The parameter-allocation curve: the frame-wise versus memory gap as a function of budget, separated by slice.
  2. The per-frame versus clip-level mining ablation, with the distribution of mined negatives across frames.
  3. LVEF correlation computed from our predicted end-diastolic and end-systolic masks against GDKVM's 0.904 on CAMUS. Ejection fraction is exactly as sensitive to end-systolic errors as our hypothesis predicts.

As a secondary probe, we add a learned, untruncated spectral mixer to the context path. This tests whether the low-frequency truncation used in S2M-Net hurts boundary metrics in video.