Motion-Tested Skip Connections for Camouflaged Video Segmentation

Video models that find camouflaged animals, such as a flatfish on sand or a stick insect on a branch, locate the animal by its movement. They then draw its outline from blurry, heavily downsampled features, so the thin parts are lost: legs, antennae, fin edges, tail tips. The usual fix for lost detail is the skip connection, a shortcut that copies early, full-resolution features straight to the layers that draw the mask. Camouflage breaks that fix. At full resolution the animal's texture looks the same as the background's, so the shortcut carries background clutter into the mask as readily as it carries true edges.

We propose to decide, pixel by pixel, which high-resolution features are allowed through the shortcut. The test asks whether that pixel's recent history is better explained by how the background moves or by how the animal moves. The test compares two candidate motions. It never estimates a free motion vector at each pixel. This matters because standard motion estimators (optical flow, which assigns a displacement to every pixel) fail on camouflaged edges. Comparing two hypotheses over several frames is a much easier decision, and the evidence builds up over time even when contrast is low.

Core Claim

The claim is that a skip connection gated by a multi-frame, two-hypothesis motion test will recover thin camouflaged structures that current video camouflaged-object models blur away. This should work as a small module added to an existing model, without dense high-resolution flow or 4D correlation volumes.

Concretely, an existing model first produces a coarse mask at low resolution. From that mask we fit two low-dimensional motions over a window of $K$ past frames. The first is a background homography: a single 3×3 perspective warp for the whole background, as in the registration module of Betrayed by Motion. The second is a smooth foreground displacement field computed at the coarse mask's resolution and then upsampled. At a shallow encoder level with stride 4, we warp the earlier frames' features under each hypothesis and compare them with the current features:

$$ E(p)=\frac{\sum_{k=1}^{K} s_k(p)\,\big[\rho\big(f_t(p)-\mathcal{W}^{bg}_k f_{t-k}(p)\big)-\rho\big(f_t(p)-\mathcal{W}^{fg}_k f_{t-k}(p)\big)\big]}{\sum_{k=1}^{K} s_k(p)+\epsilon}. $$

The symbols are as follows. $f_t(p)$ is the shallow feature at pixel $p$ in frame $t$. $\mathcal{W}^{bg}_k$ and $\mathcal{W}^{fg}_k$ warp frame $t-k$ into frame $t$ under the background and foreground hypotheses. $\rho$ is a robust penalty (Charbonnier). $s_k(p)$ is the separability: the distance between the two hypotheses' predicted displacements at $p$. It lets frames where the two motions coincide contribute nothing. $E(p)$ is positive when the pixel moves with the animal rather than the background. The skip gate is then

$$ g(p)=\sigma\big(\alpha E(p)+\beta\, m(p)+\gamma\big), $$

where $m(p)$ is the upsampled coarse-mask logit, $\sigma$ is the logistic function, and $\alpha,\beta,\gamma$ are learned scalars. The decoder receives $g\odot f_t$ in place of $f_t$.

When the total separability $\sum_k s_k(p)$ falls below a threshold, as when the animal pauses and moves no differently from the background, the gate stops using $E$. Instead it uses the previous frame's gate, carried forward by the foreground motion. This gives the fine-scale gating a memory across stop-and-go behaviour, which is the known failure point of motion-only methods.

A thin limb that swings independently of the body may fit the foreground field poorly. It still violates the background hypothesis strongly, so the first term in $E$ carries it through. The coarse-mask term keeps background parallax far from the animal from opening the gate.

Relation to Prior Work

Three video camouflage models motivate the gap. Betrayed by Motion registers the background with a learned homography and segments from the resulting difference images. Its model sees only motion, and its skip connections pass those difference features on without selection. We reuse its registration idea as one of the two hypotheses. The difference is that we use the residual to select appearance features rather than as the input itself. SLT-Net avoids explicit flow with a 4D correlation pyramid and a long-range transformer, and it introduced the pixel-level MoCA-Mask benchmark. The memory cost of that pyramid forces downsampling, and the model has no mechanism for routing detail. Vcamba models motion through Fourier phase shifts with state-space sequence models at low compute cost, but it does so on low-resolution features. All three detect motion well. None of them uses motion to decide which full-resolution features reach the mask.

Gating by motion reliability has precedents at other levels of the network. GRFP shuts a recurrent gate when flow confidence is low, but it gates warped segmentation outputs and never touches the high-resolution skip path. LiVOS routes edge-level "sensory memory" around a compressed temporal memory. Its gates are driven by appearance matching on non-camouflaged benchmarks. MoNet warps deep features with flow and adds a border-connected background prior, and it works only at deep layers. We build on the reliability-gating idea from these papers and move it to the shallow, full-detail features where camouflage does its damage.

Two static detail-preservation papers act as constraints on the design. Dual-SAM shows that predicting connectivity between neighbouring pixels sharpens marine-animal boundaries by about 2 mIoU on MAS3K, but it relies on image gradients that exact camouflage removes. gen2seg shows that discriminative mask decoders lose thin structures badly: SAM scores 16.8 mIoU on iShape against 51.4 for a generative decoder. This warns that a pipeline built on a SAM-style decoder has a ceiling on thin parts. It is also a reminder that decoder choice, not only gating, may limit detail. We control for that below.

Research Program

The module is added to two host models, Vcamba (low compute) and SLT-Net (heavier, flow-free). This lets us show the gain does not depend on one architecture. For each host we train three configurations with the same data, schedule and augmentation: the unmodified host, the host with ordinary identity skips at stride 4, and the host with motion-tested skips.

The shallow feature $f$ is taken from a fixed early layer. We ablate raw RGB against that layer, since learned features may resist the lighting flicker common in underwater clips. $K$ is swept over 2, 4 and 8. Compute is reported in multiply-accumulate operations (MACs) against Vcamba's 16.49G. We expect the module to add only warps and per-pixel residuals, because both hypotheses have few parameters.

The analysis component checks that the gate does what we claim. We measure how much gate mass falls inside the ground-truth mask, and compare this for thin-structure pixels against body pixels. We also measure how the gate behaves in frames where the animal is still.

Baselines

The naive reference is each host model exactly as published, retrained under our schedule so the comparison uses the same data and compute. On MoCA-Mask the reported numbers are an $S_\alpha$ of 0.684 for Vcamba and 0.656 for SLT-Net (Vcamba). No reported $F_\beta^w$ or mIoU was found on Lacuna for Vcamba, so we will measure these ourselves.

The strongest existing system on overall mask quality is ZS-VCOS (ZS-VCOS). It is a training-free pipeline: flow highlights moving regions, an open-vocabulary detector turns them into box prompts, and SAM 2 segments and propagates the mask. It reports $F_\beta^w$ = 0.628 and mIoU = 0.550 on MoCA-Mask, well above the supervised ZoomNeXt at $F_\beta^w$ = 0.476. We will run its public pipeline unchanged on the same test split and score it with our extra metrics. Because it is zero-shot, no tuning budget applies to it. It is the right competitor because it already combines motion with a strong decoder.

Beating ZS-VCOS on the thin-structure and boundary-band metrics is the central target. Matching its overall $F_\beta^w$ would be a bonus. If our best configuration trails it overall but wins on thin structures, we will state the narrower claim.

Three ablations isolate the key ingredient, each with the same tuning budget as the full model. The first replaces the motion evidence $E$ with a learned attention gate computed from decoder features, which tests whether motion specifically matters. The second concatenates the registration residual to the skip features instead of multiplying by a gate, which tests gating against extra input. The third drops the separability weighting and the memory fallback, which tests the stop-and-go handling.

Why not just…?

Why not just use ZS-VCOS, since zero-shot already beats supervised models? This is the real threat, and it is why ZS-VCOS is the headline baseline. Its mask comes from SAM 2's decoder, driven by a box. The gen2seg evidence on iShape suggests this family loses thin structures however good the prompt is. Our thin-structure split tests that prediction directly on camouflaged video. If ZS-VCOS matches us there, the gating idea adds nothing beyond a strong prompted decoder, and we will report that.

Why not just feed the registration difference image into a standard U-Net, as Betrayed by Motion effectively does? The concatenation ablation answers this directly. Our prediction is that adding the residual as an input helps localize the animal, but it still lets textured background leak through the skip, because nothing forces the decoder to discount those features. The boundary-band precision measured in background pixels near the animal is where this difference should show.

Why not just run the host at higher resolution, or add a learned attention gate on the skips? Higher resolution alone multiplies compute. SLT-Net's correlation volume is already the reason it downsamples, and SLT-Net's authors report that explicit flow costs over 10% accuracy, so high-resolution flow is not a cheap substitute either. The learned-attention ablation covers the second option. If an appearance-driven gate matches motion gating, the claim that motion is the necessary signal fails, and the experiment is designed so that outcome is visible.

Proposed Experiments

The main benchmark is MoCA-Mask, scored with $S_\alpha$, $F_\beta^w$ and mIoU for comparability with the numbers above. We add two splits derived automatically from the ground-truth masks, each with an evaluation metric. The thin-structure split contains pixels within a small distance of mask skeleton segments whose local width is under a few pixels, and is scored by thin-part IoU. The boundary-band split is scored by a boundary F-measure within a narrow band around the true contour. We also define a stationary split of frames where the host's own registration residual inside the mask falls below a threshold.

Secondary checks cover transfer and robustness. On CAD2016 no specific reported numbers were found on Lacuna, so we report our own runs for all methods. On MoCA we report box-level mIoU against the 39.4% of Betrayed by Motion. DAVIS 2016 serves as a do-no-harm check on non-camouflaged video, where MoNet reports a mean Jaccard of 84.7%.

Three outcomes decide the proposal. First, on both hosts, motion-tested skips must beat identity skips and the learned-attention gate on thin-part IoU and boundary F by a margin larger than the variation across three training seeds. Second, the full model must beat ZS-VCOS on the thin-structure split. Third, on the stationary split, removing the memory fallback must hurt, which shows that the separability logic handles pauses rather than hiding them.

If the first outcome holds but the second fails, the remaining limit is the decoder and not the routing of features. The natural next step would then be to give a generative decoder of the gen2seg kind the gated features as its input.