Learning Where to Re-Annotate a Video Matte

Video matting separates a moving foreground, including its hair, smoke or glass, from the background by predicting a per-pixel opacity value called alpha. The best current systems ask a person to draw a trimap on the first frame. A trimap is a rough map marking definite foreground, definite background, and an uncertain band in between. The system then carries that guidance forward through the clip. Over long clips the propagated guidance drifts. Errors compound frame by frame, and a single first-frame annotation stops being enough. The practical fix is to annotate a few more frames. Hand-drawing a trimap costs about 200 seconds per image (Learning Trimaps via Clicks for Image Matting), so the real question is which few frames deserve that effort.

We propose to treat this choice as a sparse-reward decision problem. Annotating frame $t$ helps only after its effect has propagated through every later frame. A frame that already looks bad may be a poor choice if the error would heal by itself. A frame that looks fine may be the start of a slow drift that ruins the second half of the clip. We train a critic that predicts the downstream error reduction from annotating each candidate frame, the analogue of a step-level Q-value in agent reinforcement learning. The critic sees only the video and the current propagated mattes, with no ground truth. At test time, annotations are spent greedily on the frames the critic scores highest. The claim is that credit assigned through the propagation, rather than estimated from immediate per-frame uncertainty, is what makes a small annotation budget go far.

Core Claim and Relation to Prior Work

The closest prior work is Active Matting. It trains a recurrent policy to suggest 20×20 patches in a single image for a human to label. Its reward is the immediate drop in alpha RMSE after a non-differentiable solver re-runs. We keep its central insight, that "the informativeness of a region changes once you've already labeled another nearby area," which is also why fixed heuristics underperformed in that paper. We change three things. The decision unit is a video frame, not an image patch. The payoff is delayed through a learned propagator, not immediate. And at deployment the critic must judge matte quality with no reference.

Three lines of work supply the method. The Q-value approach of Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models annotates steps with Monte Carlo rollout values, trains a small critic on them, and acts by scoring candidates. We reuse that recipe, with "candidate frame to annotate" in place of "candidate action." A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots estimates progress from pairs of observations rather than single frames. We borrow that pairwise input so the critic can see drift between frame $t$ and $t+\Delta$. That paper also constrains our design: it reports that progress critics break when progress is not monotonic over time. Matte error is exactly such a quantity, because occlusions come and go. So our critic target is the realized downstream error reduction, not elapsed time. STeCa motivates the problem framing. It shows that the step where a trajectory deviates is hidden by terminal feedback and must be located explicitly.

The matting side supplies the environment and motivates the gap. One-Trimap Video Matting jointly propagates trimap and alpha to reduce drift. Video Object Matting via Hierarchical Space-Time Semantic Guidance works from a single first-frame mask. Matting Anything 2 builds on SAM2, a promptable tracking segmenter, and adds a memory-free trimap branch because tracking memory collapses semi-transparent regions into solid masks. All three improve what one annotation can do. None asks where a second or fifth annotation should go. RePainter is the only other work in this space that combines matting-style masks with reinforcement learning. It uses those masks as a fixed attention prior for inpainting and never estimates alpha.

Research Program

The environment is a promptable propagator that accepts trimap or mask guidance on any frame. We use MAM2, whose SAM2 base natively accepts prompts at arbitrary frames, and OTVM as a second propagator to test transfer. An episode is one clip with a budget of $K$ annotations beyond frame 0. Each annotation is simulated by giving the propagator a trimap derived from ground-truth alpha at the chosen frame. A cheaper variant uses Click2Trimap-style simulated clicks. Cost is measured in seconds using the 200-second-per-trimap figure.

To build critic targets on training clips, we run Monte Carlo rollouts. For a partial annotation set $S$ and candidate frame $t$, the target is

$$Q(S,t) = E(S) - E(S \cup \{t\}),$$

where $E(S)$ is the total alpha error summed over all frames after propagating with annotations $S$. This target is dense over candidate frames, but every value requires a full propagation pass. Rollout cost is the known weakness of this family of methods. We control it by subsampling candidates to every eighth frame and by limiting $|S| \le 4$ during data generation.

The critic is a small network on top of the frozen propagator features. Its inputs are the frame, the current propagated alpha and trimap, a pairwise feature contrasting frame $t$ with neighbours at fixed offsets, and an encoding of $S$. It outputs a predicted $Q(S,t)$. At test time we pick $\arg\max_t Q(S,t)$, annotate, re-propagate, and repeat. Re-scoring after each pick is what lets the critic account for redundancy between nearby annotations.

The analysis focuses on two failure modes. One is drift that the critic must anticipate before it becomes visible. The other is the transparency collapse documented by MAM2, which a critic trained mostly on opaque human foregrounds might not recognize as error.

Baselines

The naive reference point is no extra annotation: frame 0 only. On published single-guidance settings, the strongest numbers in the file are a MAD of 14.72 on NOVM for MAM2, an MSE of 5 on DVM for HSTSG, and a 56.7% MSE improvement for OTVM on VideoMatting108. The file gives no absolute VM108 figure for OTVM. Our $K=0$ runs should reproduce the first two before any comparison is trusted.

The simplest annotation schedule is uniform spacing, which places $K$ annotations evenly through the clip. It is the obvious default and is surprisingly hard to beat when drift grows steadily.

The strongest heuristic is uncertainty selection. It annotates the frame with the largest predicted unknown-band area, or with the most alpha values near 0.5, after each re-propagation. This is the non-learned counterpart that Active Matting compared against in the image setting.

The key ablation is an immediate-error critic. It is identical in architecture, inputs, data and training compute to our critic, but is trained to predict the error at frame $t$ only, rather than the downstream reduction $Q(S,t)$. It isolates the proposal's central ingredient, credit assigned through propagation.

We also report an oracle that greedily picks frames using true $Q$ from rollouts, as an upper bound.

All methods share the propagator, clips and annotation simulator. The two learned critics also share the training budget and hyperparameter search. Beating a baseline means a lower area under the error-versus-annotation-seconds curve for $K$ from 1 to 6. The difference must hold across three seeds and on held-out clips. The main target is closing at least half the gap between the immediate-error critic and the oracle.

Why not just…?

Why not just space annotations uniformly? If drift is roughly linear in time, uniform spacing is near optimal and our critic adds nothing. This is a real threat on short, steady portrait clips such as YouTubeMatte, and we expect small gains there. The proposal's case rests on clips with events: occlusions, a second person entering, or smoke thinning. RVM's authors (Robust High-Resolution Video Matting with Temporal Guidance) report that the network gets confused about which person is the subject in exactly such clips. We will report results separately for event-rich and steady clips, so a uniform-spacing win on steady clips is visible rather than averaged away.

Why not just annotate wherever the model is least certain? Uncertainty measures current error, not future payoff. Active Matting found heuristic selection consistently worse than a learned sequential policy in images, but that evidence comes from immediate rewards and patches, not video. The immediate-error-critic ablation is the fair test. If it matches the downstream critic, then propagated credit is unnecessary and the simpler predictor should be adopted. We would report that as the finding.

Why not just use a stronger propagator? MAM2 already shrinks single-annotation error substantially. A better propagator lowers the curve but does not remove drift on long clips, which OTVM identifies as the core failure. Running the critic on both MAM2 and OTVM tells us whether the selection gain survives a stronger base model, or shrinks as the propagator improves.

Proposed Experiments

We train critics on rollouts over VideoMatting108 training clips. Evaluation uses the VM108 test split (MSE, SAD and dtSSD, a temporal-coherence error), DVM (MSE), NOVM (MAD, with per-category breakdown for smoke, fire and bubbles) and YouTubeMatte (MAD).

The decisive check is the downstream-versus-immediate critic comparison at matched budgets on event-rich clips. The second check is transfer. A critic trained with MAM2 rollouts is applied unchanged to OTVM. If its picks still beat uncertainty selection, the critic has learned something about where videos are hard, not quirks of one propagator. The third check is on NOVM transparent categories. There we measure whether the critic detects transparency collapse, judged by the oracle's picks there, or misses it because collapsed mattes look confident. If it misses, the result argues for adding MAM2's memory-free branch output as a critic input.