Splitting Video Pseudo-Label Noise into Its Spatial and Temporal Parts

When a video model teaches itself from its own guesses on unlabeled footage, the guesses (pseudo-labels) carry two different kinds of error. Some are mistakes of where and what: a mask that bleeds into the background, or a missed object in a single frame. Others are mistakes of when and which: the same object given two identities across frames, or a track that jumps to a neighbor. Current self-training pipelines treat a video pseudo-label as one object, score it with one quality number, and update one jointly trained network. This proposal argues that the well-known early saturation of video self-training comes from mixing these two error types. Their gradients interfere in a shared network, and each error type then corrupts the representation that should have caught the other.

We propose to train the per-frame (spatial) part and the cross-frame (temporal) part of a video segmenter as separate modules. Each module's pseudo-labels are verified by the other module rather than by itself. Each module is fed the cheapest synthetic data that can repair its own error type: synthetic still images for spatial errors, and synthetic motion made by moving pasted image masks along random trajectories for temporal errors. Neither requires running a video diffusion model, the expensive step that dominates synthetic-video pipelines. In short, the proposal factorizes the noise, cross-checks each factor with the other, and pays for synthetic data per factor. The prediction is that this keeps self-training improving past the point where joint pipelines collapse, at lower total compute.

Core Claim

AutoQ-VIS is the closest existing work and our main point of departure. It does unsupervised video instance segmentation (finding and tracking every object in a video without labels). It warms up a jointly trained VideoMask2Former on synthetic videos built from ImageNet pseudo-labels. It then self-trains on unlabeled YouTubeVIS-2019 videos, using a frozen quality predictor that estimates each pseudo-mask's overlap with the truth, and a loss that drops low-quality masks. Its accuracy peaks at round two (+4.4 AP50) and degrades afterwards, leaving a 10.1 AP50 gap to the fully supervised model.

Our claim is that the decline after round two comes mainly from temporal association errors. These pass a mask-quality filter that looks at masks frame by frame, and they then contaminate the shared features that produce masks. If that is true, three changes should push the peak later and raise it: decouple the modules, gate each factor's pseudo-labels using the other factor, and target synthetic data at each factor. If it is false, meaning spatial mask errors dominate the late-round decline, the decoupling buys little. We will find this out in the first experiment, before building anything else.

How the Prior Work Shapes This

Three lines of work motivate the decoupling.

Generative video models already separate appearance from motion to avoid interference. MoTrans documents that joint training drags identity along with motion. Learning Temporally Consistent Video Depth from Video Diffusion Priors learns temporal behavior as a residual on a frozen per-frame model without losing single-frame accuracy. SurV-Gen (Mission Balance) freezes a frame-level diffusion model and trains only inserted temporal attention. All three use decoupling for generation. We use it for self-training a perception model, where the payoff is error isolation rather than identity preservation.

The second line, from pseudo-labeling, supplies the verification tools. OMRS-I2V adapts first at the image level and then at the video level. It keeps a pseudo-label only if a cycle walk through neighboring features returns to the same class. PTCL trusts pseudo-labels near verified anchors first and assimilates distant ones later. We borrow the cycle check for verifying track links and the anchor-outward curriculum for spreading masks through time.

The difference from OMRS-I2V matters. It trains two separate backbones in sequence and runs its stages once. We keep one shared image encoder with two lightweight heads and alternate the stages over self-training rounds, with each stage checking the other's output. Neither OMRS-I2V nor PTCL uses synthetic data.

The third line constrains the design on cost. Consistent Video-to-Video Transfer Using Synthetic Dataset generates about 400,000 synthetic video pairs and filters them with CLIP. Depth Any Video needs game-engine depth hooks and, even with flow matching, runs at 0.37 s per frame. SurV-Gen takes 6.55 s per generated clip, plus a rejection classifier that discards samples. These figures argue against putting a video generator inside a self-training loop. They also motivate our claim that, once errors are factorized, video synthesis becomes unnecessary: each factor can be repaired with synthetic data that is orders of magnitude cheaper. TrueTeacher and DISCO show the same generation-plus-filtering tax in text, where the filter's cost can rival generation itself. We therefore count verification compute in every budget.

Research Program

The model. It has a shared image encoder, a spatial head that predicts instance masks for each frame independently, and a temporal head that links per-frame instance queries across time. The temporal head works by attending over the queries of neighboring frames.

Spatial training. Training proceeds in alternating rounds. In a spatial round, the spatial head and encoder train on real video frames treated as an unordered set. As in Dynamic Concepts Personalization, ignoring order keeps motion out of this stage. They also train on CutLER-style synthetic still images. A frame's pseudo-mask is kept when two conditions hold. Its predicted quality must be high, and it must agree with the masks the frozen temporal head propagates into that frame from nearby high-confidence anchor frames. Propagation runs outward from anchors in PTCL's order. The temporal head is thus the spatial head's second opinion.

Temporal training. In a temporal round, the encoder and spatial head are frozen and only the temporal head trains. Its data are real videos with pseudo-tracks, plus synthetic trajectories made by pasting image instances onto backgrounds and moving them with random affine motion, occlusion and crossings. These trajectories have exact identity labels at almost no cost. A pseudo-link between two frames is kept only if both endpoint masks pass the spatial gate and the forward-backward association cycle returns to the starting instance. This adapts OMRS-I2V's cycle walk to instance tracks.

Breaking the confirmation loop. Because neither head verifies its own outputs, the self-confirmation loop described in Tri-net is broken: each head is checked by a model that does not share its blind spot.

Analysis. The analytic part of the program is an error decomposition run on every round's pseudo-labels against held-out ground truth, used only for analysis and never for training. It separates per-frame mask error (frame-level mask AP) from association error (identity switches and fragmentation). It also tracks the compute cost per surviving label for each synthetic source: synthetic images, synthetic trajectories, and, as a comparator, diffusion-generated clips.

Baselines

The naive reference point is the synthetic-only model. This is the VideoMask2Former warmed up on VideoCutLER-style synthetic videos with no self-training. It sets the floor and stands in for the zero-shot synthetic-transfer strategy of Depth Any Video.

The strongest existing method is AutoQ-VIS itself, run with its quality predictor and loss-dropping on YouTubeVIS-2019. Its reported result is +4.4 AP50 after two rounds and a remaining 10.1 AP50 gap to fully supervised training. The file does not give the absolute AP50, so we will reproduce it, and treat the published gain and gap as a sanity check on that reproduction. We run it with the same encoder size, the same unlabeled videos, and the same total GPU-hour budget. That budget counts synthesis, quality-predictor training and all self-training rounds. Both methods get an equal hyperparameter search. We run five rounds for both, so that saturation is visible. Beating it means a higher best-round AP50 with no later-round collapse, at equal or lower total compute. A single-seed gain inside the round-to-round variance does not count; we will use three seeds.

Three ablations isolate the key ingredients. The first keeps the joint VideoMask2Former but replaces its single quality score with our two factor-specific gates. This tests whether factorized verification alone accounts for the gains, without architectural decoupling. The second keeps the decoupled architecture but lets each head gate its own pseudo-labels. This tests whether cross-checking matters. The third swaps our cheap synthetic trajectories for SurV-Gen-style diffusion clips with classifier rejection at matched compute. This tests whether expensive video synthesis is worth its cost.

Why not just…?

Why not just stop self-training at round two, or tighten AutoQ-VIS's threshold? Early stopping caps the gain at the 4.4 AP50 already reported and leaves the 10.1 AP50 gap untouched. Tightening a single threshold throws away hard-but-correct masks, a known weakness of hard filters, and does not separate the error type that is accumulating. This is the default we must beat. If AutoQ-VIS with a tuned threshold matches us at round two and our method only avoids the collapse without exceeding its peak, the contribution shrinks to robustness, and we will report it as such.

Why not just add a mean teacher (an averaged copy of the model that produces the pseudo-labels) to the joint model? This is a real threat. Smoothing the teacher slows confirmation drift without any decoupling. We will run AutoQ-VIS plus an averaged teacher as an additional baseline. If it closes most of the gap, the decoupling is not doing the work. Our error decomposition makes a distinct prediction here. An averaged teacher should slow both error types equally, whereas decoupling should specifically flatten the growth of association error.

Why not just generate more synthetic video with a decoupled video diffusion model, as SurV-Gen does? At 6.55 s per clip plus rejection losses, a matched budget buys far fewer labeled examples than paste-and-move trajectories. The third ablation measures accuracy per GPU-hour directly. If diffusion clips win at equal compute, perhaps because pasted motion is too unrealistic for the temporal head, we will adopt them for the temporal factor only and report the cost.

Why not just reuse OMRS-I2V's image-then-video curriculum once? A single pass verifies pseudo-labels only at the end. Our hypothesis is that the errors appear across rounds. The comparison is our method run for one round versus five.

Proposed Experiments

The main benchmark is unsupervised video instance segmentation on YouTubeVIS-2019, following the AutoQ-VIS protocol. We report AP and AP50 per round, frame-level mask AP, identity-switch counts, and total GPU-hours broken into synthesis, verification and training.

The first decisive check comes before any method building. We rerun AutoQ-VIS for five rounds and decompose its pseudo-label errors. If association error does not grow faster than mask error after round two, the core claim fails and we stop.

The second decisive check is the round-by-round AP50 curve of our method against the AutoQ-VIS and averaged-teacher baselines. The test is whether the peak moves past round two and rises above AutoQ-VIS's best round.

The third decisive check is the cost curve: accuracy against total GPU-hours for image-plus-trajectory synthesis versus diffusion-clip synthesis.

As a secondary probe of the transfer from image models to video, we will fine-tune the temporal head on Something-Something-v2. In this motion-heavy action benchmark, BEVT (BEVT) reports 71.4% top-1 with joint image and video pretraining. We do not aim to beat that number. The goal is to test whether a temporal head trained on synthetic trajectories over a frozen image encoder captures real motion order, which the main experiments rely on.