Occlusion Schedules Instead of Video Diffusion: Cheap Synthetic Clips for Consistent Video Instance Segmentation

Video instance segmentation means finding every object in a video, outlining it pixel by pixel, and keeping its identity fixed from frame to frame. Models that do this well still fail in a predictable place. When an object is mostly hidden and later reappears, it often loses its identity or flickers out of existence, and labeled videos showing such events are rare and expensive to annotate. A recent fix is to generate training videos: a text-to-video diffusion model (a generator that turns random noise into frames by denoising it step by step) produces moving objects, which are cut out and pasted onto real background videos. This works, but it is costly. Each clip takes many denoising steps, clips are capped at 16 frames, and every generated object has to pass through a segmenter and a quality filter before it can be used.

We propose to test a cheaper hypothesis. What models lack is not realistic generated motion. It is controlled occlusion events with known identities. Such events can be built almost for free by layered compositing. We take object cutouts from images, deform them over time by borrowing real object-shape trajectories from existing annotated videos, and place them in depth-ordered layers on real backgrounds under an explicit schedule of partial occlusion, full disappearance, and reappearance. Because we place every layer ourselves, the visible mask and identity of each object are exactly known in every frame, including frames where the object is completely hidden. Clips can also be any length. The central experiment crosses where the motion comes from (video diffusion, borrowed real shape trajectories, or rigid image transforms) with how objects move (random linear paths versus scheduled occlusions). It measures accuracy on heavily occluded objects and boundary stability per GPU-hour spent generating data.

Core Claim

Synthetic data helps temporal consistency mainly by supplying occlusion and reappearance events with exact identity labels, rather than by supplying realistic appearance change. If the occlusion structure is made explicit, a generator costing a small fraction of video diffusion should match or beat diffusion-based synthetic clips on heavy-occlusion accuracy, at equal or lower generation compute.

The motivating evidence comes from three places. SDI-Paste generates 16-frame object clips with AnimateDiff, extracts masks with TokenCut, filters them with CLIP, and pastes them along "linear-random" trajectories. It reports +2.9 AP for CTVIS and +2.1 AP for IDOL on YouTube-VIS 2021. Its margin over static image pasting (X-Paste) is only 0.4 AP, and its authors find that generation glitches do no harm and that simple trajectories beat physically elaborate ones. Together these suggest that the expensive part of the pipeline, video diffusion, buys little. Second, OVIS documents that models lose about 80% of their accuracy on heavily occluded objects. Third, the memory-based models GenVIS and TCOVIS are built to carry identity across long clips. Yet the existing synthetic data, short clips with random paths, almost never asks them to carry identity through an object's complete disappearance.

The method borrows from BVDM. That work separates appearance from motion: it generates a single textured frame and then warps it forward with predicted flow fields. Trained from only one real microscopy video, it produced cell tracking data that outperformed real-data training (SEG 0.829 vs 0.823). BVDM relies on a statistical shape model that cannot cover natural object motion. We replace that model with shape trajectories taken from real annotated videos.

Research Program

The generator has three parts, and none of them runs a video model at sampling time.

The first part is the appearance bank. It holds object cutouts with masks, drawn from two sources. One is the X-Paste pipeline (Stable Diffusion, then U2Net, then a CLIP filter), which costs one image generation per instance. The other is real instance masks from image datasets, which cost nothing to generate. This lets us measure how much even single-image generation contributes.

The second part is motion transplant. From YouTube-VIS training annotations we collect real per-object mask sequences and normalize each into a "shape track." To animate a cutout, we fit a smooth thin-plate-spline warp from the cutout's mask to each frame's mask in a sampled track, and apply that warp to the texture. Objects therefore inherit real non-rigid shape change, such as turning, stretching, or limbs moving, at the cost of a geometric warp. This addresses the stated weakness of MASA, whose 2D rigid transforms of static images never show genuine shape change. It also addresses the 16-frame ceiling in SDI-Paste, because tracks can be chained or looped to any length.

The third part is the occlusion scheduler, which is the ingredient we expect to matter. Each composited clip receives a depth ordering of the pasted objects relative to each other and to the real objects in the background. Each clip also receives a sampled event program built from three kinds of events. In a crossing, two objects pass with one occluding the other. In a disappearance, an object goes fully behind a layer for a sampled number of frames, from a few up to dozens, and then reappears. In a near-identical distractor, a second copy of the same category, sometimes the same cutout recolored, passes nearby to force identity discrimination. Visible masks are computed exactly from the layer order. Identity is preserved through invisible frames, so the supervision used by GenVIS and TCOVIS for long-clip label assignment sees reappearance events directly. We control how many frames are heavily occluded so that the distribution can be pushed toward OVIS's heavy-occlusion regime.

We count generation cost in GPU-seconds per accepted training frame for every pipeline, including SDI-Paste re-run on the same hardware. Results are then reported as accuracy against generation compute curves rather than as single points.

Baselines

The naive reference point is the same video model trained only on real data, with an identical schedule and number of iterations. We use GenVIS with a ResNet-50 backbone as the main model because its prototype memory and long-clip assignment are the components that occlusion schedules should train. GenVIS reports 35.8 AP on OVIS with ResNet-50 GenVIS, and we will reproduce this number before adding any data. Beating the naive baseline means a gain beyond seed variance, which we estimate from three seeds, on both OVIS AP and heavy-occlusion AP.

The strongest existing competitor is SDI-Paste SDI-Paste, whose reported result is +2.9 AP over CTVIS on YouTube-VIS 2021. No absolute CTVIS number was found on Lacuna, and no SDI-Paste result on OVIS or with GenVIS was found on Lacuna. We will therefore run SDI-Paste ourselves on both CTVIS (to check that we reproduce its reported gain) and GenVIS. The comparison will use the same real backgrounds, the same number of synthetic frames (300,800, matching the paper), the same mixing ratio with real data, and the same tuning budget for each method. We will also run a second comparison at equal generation GPU-hours, which favors whichever pipeline is cheaper. We beat SDI-Paste if we match its YouTube-VIS 2021 gain within seed noise and exceed it on OVIS heavy-occlusion AP, at no more than one tenth of its generation compute.

The static-paste baseline is X-Paste-style cutouts moved by rigid affine transforms along linear-random paths, in the spirit of X-Paste and MASA. It is the cheapest plausible competitor.

The central ablation is a full crossing of motion source (video diffusion clips from SDI-Paste, motion transplant, or rigid affine) with trajectory type (linear-random or occlusion schedule). This isolates which ingredient produces the gain. For architecture-only context we cite TCOVIS at 46.7 AP on OVIS with Swin-L TCOVIS and run the full study at ResNet-50 scale. A smaller confirmation run with TCOVIS tests whether the effect holds for a second memory design.

Why not just…?

Why not just paste static cutouts with rigid motion, which is nearly free? This is the most serious threat, because SDI-Paste beat X-Paste by only 0.4 AP. We take it seriously by including the rigid-affine plus occlusion-schedule cell in the factorial design. If that cell matches motion transplant, then non-rigid shape change is unnecessary, and the cheapest version of our generator is the result we report. In either case the occlusion scheduler is tested separately from the realism of the motion.

Why not just use game-engine data like SAIL-VOS, which already has exact amodal masks? SAIL-VOS draws 111k GTA-V frames and raised DAVIS16 IoU from 55% to 74%. However, it is limited to the game's assets and depends on fragile graphics hooks, and it was tested only with frame-by-frame Mask R-CNN. We include SAIL-VOS pretraining as an extra arm on OVIS. If it beats our generator on heavy-occlusion AP for overlapping categories, we will conclude that full rendered 3D occlusion matters beyond 2D layering. That would be a real finding against us, and we would report it as such.

Why not just rely on better architectures? Memory-based models are exactly the components that need occlusion events to learn from. The additive gains reported by SDI-Paste on top of CTVIS and IDOL show that data and architecture are not substitutes. The naive GenVIS baseline measures directly how much the architecture achieves without our data.

Proposed Experiments

The main benchmark is OVIS, where we report AP and the heavy-, medium-, and light-occlusion breakdowns. We also count identity switches that occur within a window after an object reappears, using the ground-truth tracks to locate reappearance events. YouTube-VIS 2021 is used for comparability with SDI-Paste, and YouTube-VIS 2022 tests long videos, which matters because our clips are not length-limited.

To address the concern that pasted seams inflate AP while worsening outline stability, we evaluate on HQ-YTVIS with the Tube-Boundary AP and pooled AP introduced by RoCoVIS. We also evaluate seam-feathered versions of every pasting pipeline.

The decisive checks are three. First, at equal synthetic frame counts, the occlusion-schedule rows of the factorial design should beat the linear-random rows on OVIS heavy-occlusion AP regardless of motion source. If they do not, the core claim fails. Second, at equal generation GPU-hours, motion transplant should dominate video diffusion on the accuracy-versus-compute curve. Third, a sweep over the maximum disappearance length should show heavy-occlusion AP for GenVIS rising with disappearance length in training while a no-memory model stays flat. That pattern would tie the gain specifically to learning to remember identity rather than to generic augmentation.