Where the Background Disappears: Occlusion Evidence as a Boundary Signal for Low-Contrast Video Segmentation

When a camouflaged animal crawls across sand, or a polyp sits on colon wall of nearly the same color, a single frame often holds no visible edge at all. Motion still gives the boundary away, through a cue other than the object's speed. As the object moves relative to what lies behind it, background texture vanishes along its leading edge ("deletion") and reappears along its trailing edge ("accretion"). Optical flow is the per-pixel estimate of where each point moves between frames. In optical flow, these vanishing and reappearing pixels are exactly the occluded pixels, the ones with no match in the next frame. They lie on a thin band that hugs the true object outline. They also carry a sign: the side of the band that keeps its texture and moves with the band is the object, and the side that loses texture is background. This note proposes treating occlusion maps as a boundary-and-ownership signal for segmentation in low-contrast video, not as nuisance masks to discard. We would accumulate them over several frame gaps and many frames into a dense map of "who owns this edge."

The opportunity matters because the two existing ways of using motion for camouflaged or endoscopic video both use the wrong part of the motion signal. Zero-shot pipelines highlight pixels with large flow magnitude and prompt a segmenter from that. Surgical pipelines threshold angular differences in flow. Both depend on the flow field being correct near the edge, which is precisely where flow "bleeds" across low-contrast boundaries. Matching failure is a more local, more contrast-robust event than matching success at the right displacement. The core claim is that a learned occlusion estimator trained for low contrast, with its output integrated over time into a signed boundary-ownership map, gives a segmentation prior. That prior should beat flow magnitude, flow-gradient boundaries, and static edge filters on low-contrast video, and the gain should grow as image contrast at the boundary falls.

Grounding in prior work

The occlusion-aware flow literature supplies the estimator. Modern networks predict occlusion probability alongside flow and use it to switch off the matching cost at pixels with no correspondence (Hur et al.; Jeong et al.). OCTC (Imposing Consistency for Optical Flow Estimation) trains a RAFT-based model to recognize synthetic "cow-mask" occlusions pasted onto images. It reports 4.33% Fl-all on KITTI-2015 and lowers Sintel-Clean error from 1.43 to 1.31. We reuse its synthetic-occlusion training idea, but we change the distribution to camouflage-like composites. The same paper also constrains us: it names textureless and degraded-boundary zones as places where correlation peaks go flat and both flow and occlusion become unresolvable. Our design must therefore show that occlusion evidence survives where flow does not, and the experiments test exactly that. In this whole line of work, occlusion is a by-product that protects the flow. None of it uses occlusion as an output for segmentation.

Motion boundaries have been used as structure before. MoBoNet (Deep Motion Boundary Detection) learns motion boundaries from flow, images and warping error, and uses them to stop flow bleeding. It raises boundary mAP on Sintel Final from 59.7 to 67.9, but only on scenes with moderate to high contrast. HQAM (Rethinking Low-quality Optical Flow in Unsupervised Surgical Instrument Segmentation) builds boundary masks from angular differences between neighboring flow vectors in surgical video. It drops the highest-loss frames, reaching 0.75 mIoU on EndoVis2017 VOS. Both derive boundaries from the flow field's spatial gradient, and both are unsigned. We derive boundaries from matching failure and keep the sign. HQAM's choice to drop bad frames rather than reason about why matching fails is a tension the file highlights. Our method takes the other side of that tension.

On the segmentation side, three systems motivate the gap. ZS-VCOS (ZS-VCOS) runs RAFT open-loop, paints flow magnitude onto the frame, detects the highlighted animal, and prompts SAM 2. It reaches weighted F-measure 0.628 and mIoU 0.550 on MoCA-Mask, but its authors note it fails when the animal stops or is partly occluded. CamSAM2 (CamSAM2) adapts SAM 2 with appearance prototypes and explicitly lists "integrating explicit motion cues" as future work. OASIS (Structure Matters) feeds Canny edges into a structure decoder, which gives an F-score of 91.6 on DAVIS-17, but Canny returns little on soft-tissue gradients. The video polyp models PNS+, MAST and HFSTI-Net (PNS+; MAST; HFSTI-Net) use attention and memory banks with no physical motion or occlusion model. The PNS+ authors flag "indefinable boundaries" as unsolved. The conceptual ancestor is the FormMotionOcclusion model (Computational modeling of depth ordering in occlusion through accretion or deletion of texture). It shows with handcrafted filters on random-dot displays that accretion/deletion establishes figure-ground without luminance contrast. We turn that qualitative principle into a learned, benchmarked component.

Core Claim

Let $O^{+}_{t,k}$ and $O^{-}_{t,k}$ be the predicted probabilities that a pixel in frame $t$ is deleted (visible at $t$, gone at $t+k$) or accreted (absent at $t-k$, visible at $t$). The proposal builds a signed ownership map by warping these into frame $t$ and summing over gaps $k$ and a window of past frames:

$$ B_t(x) = \sum_{s \in W_t} \sum_{k \in K} w_{s,k}(x)\,\mathcal{W}_{s \to t}\!\left[\nabla \!\left(O^{+}_{s,k} - O^{-}_{s,k}\right) \cdot \hat{n}_{s}\right](x) $$

Here $W_t$ is a window of recent frames and $K$ a set of frame gaps (for example 1, 2, 3, following HQAM's observation that wider gaps amplify slow motion). $\mathcal{W}_{s\to t}$ warps a map from frame $s$ into frame $t$ using flow estimated only on non-occluded pixels. $\hat{n}_s$ is the local direction of relative motion, which turns "deletion on this side, accretion on that side" into "object on this side." $w_{s,k}$ is a learned confidence weight. In words, each frame contributes an oriented edge fragment wherever texture is being swallowed or revealed. As the object moves in different directions over time, the fragments close into an outline whose inside is known.

The prediction is testable. When relative motion exists, $B_t$ should keep its boundary localization as image contrast across the true edge goes to zero. Flow-magnitude and flow-gradient cues should degrade with contrast, because they need correct displacements at the edge rather than mere matching failure.

Research Program

The work has three pieces. The first is a low-contrast occlusion estimator. We take a RAFT-style network with an occlusion head and train it OCTC-style on synthetic composites. In each composite, the foreground texture is cut from the background image itself and pasted with randomly reduced contrast. Foreground and background move independently, and camera parallax is simulated with a two-layer depth model. Occlusion ground truth comes free from the compositing. This is the piece most likely to fail, because of OCTC's warning about flat correlation peaks. The first milestone is therefore a pure measurement: occlusion-band recall on held-out composites as a function of contrast, compared against RAFT forward-backward consistency.

The second piece is the temporal ownership accumulator defined above, a small recurrent module with learned weights $w_{s,k}$. It also outputs an evidence-strength channel, so that downstream components know when the scene gives no motion evidence, for instance when an animal is frozen or the endoscope is still.

The third piece injects the map into a segmenter in two regimes. In the zero-shot regime on MoCA, we replace ZS-VCOS's flow-magnitude highlight with $B_t$. The rest stays the same: Owl-v2 detection, SAM 2 propagation, and the same bidirectional merge. $B_t$ also enters as a dense mask prompt. This swap isolates the cue. In the supervised regime on SUN-SEG, we add $B_t$ and its strength channel as inputs to a CamSAM2-style lightweight adapter on frozen SAM 2. Where evidence is absent, the model falls back on appearance prototypes. We also run a closed-loop variant. There, the current mask is fed back to the flow estimator to suppress matching costs inside the predicted object, and flow under occluders is inpainted from the visible part of the object. This step answers Gap 4 directly: occlusion-gated reasoning in place of HQAM's case dropping.

Baselines

The naive reference is SAM 2 with the same prompts and no motion input. For SUN-SEG it is CamSAM2's starting point; the file reports CamSAM2 improves on it by 19.6 mDice on SUN-SEG-Hard and 12.2 mDice on MoCA-Mask. This tells us how much any adapter recovers.

On MoCA-Mask the strongest motion-based competitor is ZS-VCOS (ZS-VCOS), which reports weighted F-measure 0.628 and mIoU 0.550. It is the strongest because it already exploits motion with the same foundation segmenter, so it is the direct test of whether occlusion beats magnitude. We rerun it with identical RAFT weights, detector, SAM 2 checkpoint, and evaluation code, changing only the prompt source. Beating it means exceeding 0.628 weighted F-measure under our rerun, and at least matching our reproduced ZS-VCOS number if that differs from the reported one. The gain must be concentrated in the occluded and slow-motion clips where the authors say it fails.

On SUN-SEG-Hard the strongest numbers in the file are HFSTI-Net at 86.27% Dice (HFSTI-Net) and MAST at 0.803 Dice (MAST). CamSAM2 is the most direct architectural competitor, since our supervised variant is CamSAM2 plus one input. The file gives CamSAM2 only as a relative +19.6 mDice over SAM 2 and no absolute Dice, so we retrain it ourselves with the same training split, schedule and tuning budget, with and without the ownership channel. Beating it means a Dice gain on SUN-SEG-Hard that is largest on the "Indefinable Boundaries" attribute subset. We report against HFSTI-Net's 86.27% as the absolute bar but do not claim to have to exceed it, since it uses a different backbone. For surgical video we compare against HQAM+LQCD on EndoVis2017 VOS (0.75 mIoU reported) in the unsupervised RCF setting, substituting $B_t$ for HQAM's angular boundary mask.

The ablations isolate the key ingredient by holding the injection point fixed and swapping the cue. The variants are: flow magnitude; HQAM-style angular flow boundaries; MoBoNet-style learned motion boundaries; Canny edges as in OASIS; off-the-shelf RAFT forward-backward inconsistency maps; our occlusion maps unsigned; and our maps signed but without temporal accumulation. Each gets the same tuning budget.

Why not just…?

Why not just use RAFT's forward-backward inconsistency as the occlusion map? This is the most serious threat, because it is free and might capture most of the signal. The OCTC paper's case for learned occlusion is that forward-backward checks are costly and noisy. That argument was made on natural scenes, not low-contrast ones. Our ablation feeds forward-backward maps through the same accumulator and injection point. If they match our learned estimator, the contribution shrinks honestly to the signed temporal accumulation and the use of occlusion as a segmentation prior. That is still a finding, and we would report it as such.

Why not just concatenate off-the-shelf flow to the RGB input? HQAM's authors report that raw RAFT flow in surgical video produces hallucinated segmentations until it is filtered. MoBoNet documents flow bleeding across boundaries. Both suggest that a network given raw flow inherits errors exactly at the edge. Our ablation includes a flow-concatenation variant of the CamSAM2 adapter, and the contrast-stratified analysis shows whether its errors concentrate in the low-contrast band as predicted.

Why not just rely on appearance adapters like CamSAM2? CamSAM2 already gains a lot without motion. Its own authors identify long occlusions and shot changes as failures and call for motion cues. The real question is whether motion adds anything on top. The with-and-without comparison under identical training answers it. If the gain is zero on clips with relative motion, the idea fails.

Why not just a stronger single-frame boundary model? PolypFlow reaches 0.971 mDice on Kvasir-SEG, but that is a static benchmark. HFSTI-Net documents shape collapse and lost tracking in video, and PNS+ shows video modeling beats image models on SUN-SEG (0.756 vs. 0.713 Dice for ACSNet). Single-frame models cannot use a cue that exists only between frames. They remain a reference row, not the main competitor.

Proposed Experiments

The decisive check is a stratified analysis rather than a headline number. For every ground-truth boundary pixel on MoCA-Mask and SUN-SEG-Hard, we measure local contrast across the edge and local relative motion, the latter estimated from ground-truth masks across frames. Boundary F-score is plotted against contrast within each motion bin, for every cue in the ablation. The hypothesis predicts that occlusion-based ownership keeps its boundary F-score flat as contrast falls, provided relative motion exists, while magnitude, angular-flow and Canny cues fall off. In the zero-motion bin, all cues should collapse to the appearance-only baseline, and the evidence-strength channel should correctly report low confidence. If the occlusion curve falls with contrast just like the others, the central claim is refuted.

The second check validates the estimator in isolation. On the synthetic camouflage composites we measure occlusion-band precision and recall against contrast, compared with RAFT forward-backward consistency. On Sintel we confirm that low-contrast training does not ruin ordinary flow. No reported result was found on Lacuna for Sintel occlusion-map accuracy, so we report our RAFT baseline alongside OCTC's reported 1.31 Clean EPE as the flow sanity bar.

The third check tests ownership specifically. We evaluate signed versus unsigned maps on clips where foreground and background textures are statistically identical, the MoCA-Filtered subset; ZS-VCOS reports a detection success rate of 0.697 there. On those clips an unsigned edge cannot tell inside from outside, so the sign should matter most. Finally, we measure latency. CamSAM2 reports 6 ms per frame, and the iterative solvers in the file are criticized for latency. We therefore report frames per second for the full pipeline and for a cached-flow variant, so the cost of the extra flow pass is visible rather than hidden.