To train a model to segment objects in static images, an annotator draws a boundary around an object once. To do the same for video, that boundary must be drawn dozens of times per second across thousands of frames. This is the dense annotation bottleneck: the reality that manual pixel-level masking for video sequences is prohibitively expensive and fundamentally unscalable. As noted by Huiling Wang et al. in Semi-Supervised Domain Adaptation for Weakly Labeled Semantic Video Object Segmentation and Yu Zhang 0035 et al. in Semantic object segmentation via detection in weakly labeled video, this data scarcity severely limits the development of fully supervised video models.
To bypass this bottleneck, a growing body of research focuses on a simple insight: video contains massive temporal redundancy. We do not need to label every pixel in every frame because the physical world changes continuously. By combining sparse human inputs (like clicks, points, or boxes) with temporal propagation and self-supervised spatial priors, models can achieve dense, pixel-perfect video understanding at a fraction of the annotation cost.
Propagating Sparse Labels Across Time
The most intuitive way to beat the dense annotation bottleneck is to label a single frame and let the model figure out the rest. Because adjacent frames in a video are highly correlated, information can be "warped" or tracked across time.
For instance, David Nilsson et al. demonstrate in Semantic Video Segmentation by Gated Recurrent Flow Propagation that optical flow can be used to propagate labels from sparsely annotated frames to neighboring unannotated frames. By using a gated recurrent unit (GRU) to fuse these warped labels with current frame feature estimates, their framework maintains temporal consistency without requiring dense manual annotations. Similarly, in Query-Memory Re-Aggregation for Weakly-supervised Video Object Segmentation, Fanchao Lin et al. use a memory re-aggregation mechanism to bridge the gap between a target's initial sparse location and its subsequent frame-by-frame pixel masks.
Even when user interaction is required dynamically, minimizing the initialization cost is paramount. Traditional semi-supervised video object segmentation (VOS) requires a perfect mask for the first frame, which can take minutes per object. To solve this, Pinxue Guo et al. introduce ClickVOS: Click Video Object Segmentation, which initializes tracking with a single click. Their framework uses an "improvement memory" that allows initially imprecise masks to self-heal as the video progresses, demonstrating that temporal context can correct sparse initialization errors.
Lifting Weak Supervision to Dense Masks
If frame-by-frame tracking is possible, do we even need a full mask in the initial frame? Modern architectures show that we can replace masks entirely with cheaper alternatives like bounding boxes, points, or scribbles. The core mechanism involves lifting these low-dimensional annotations into dense masks by leveraging spatial boundaries.
In Mask-Free Video Instance Segmentation, Lei Ke et al. eliminate mask annotations entirely during training, relying only on bounding boxes. To prevent the model from simply segmenting the entire box (including the background), they enforce consistency across frames, forcing the network to isolate the moving object from its background. Liqi Yan et al. take a similar approach in Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration, combining unsupervised depth estimation and optical flow into a "puzzle loss" that forces box annotations to resolve into precise object boundaries.
We can go even sparser. Shuaiyi Huang et al. introduce What is Point Supervision Worth in Video Instance Segmentation?, proving that a single annotated point per object per frame is sufficient. By pairing a spatio-temporal point matcher with strong image-based segmentation priors, their framework achieves up to 97% of the performance of fully supervised models.
Similarly, Sabarinath Mahadevan et al. present Point-VOS: Pointing Up Video Object Segmentation, showing that point-based supervision can be up to 40 times faster to annotate than dense masks while retaining high tracking fidelity. Even alternative modalities like eye-tracking fixations can serve as weak supervision, as shown by Qi Qin et al. in From Sight to Insight: Unleashing Eye-Tracking in Weakly Supervised Video Salient Object Detection.
Self-Supervised Objectness and Pseudo-Labeling
What if we have no human annotations at all for the vast majority of frames? The solution lies in exploiting the inherent structures within deep networks and generating pseudo-labels to guide training.
Vision Transformers (ViTs) trained via self-supervision (such as DINO) naturally learn to cluster semantic objects without any mask labels. In Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation, Rui Qian et al. exploit this "objectness" hidden within attention maps. By adding a simple spatio-temporal block over these self-supervised features, they achieve state-of-the-art multi-object segmentation without using a single manual mask during training.
Alternatively, models can learn to track pixels purely by matching visual properties like color across frames. Youngeun Kim introduces RPM-Net: Robust Pixel-Level Matching Networks for Self-Supervised Video Object Segmentation, which trains a network to map pixels into an embedding space where temporal color consistency is maximized. This allows the network to learn robust tracking representations completely unsupervised.
When some labels are available, pseudo-labeling frameworks can fill the gaps: Yongri Piao et al. use uncertainty-guided pseudo-labels in Semi-Supervised Video Salient Object Detection Based on Uncertainty-Guided Pseudo Labels to filter out noise and scale training using 95% unlabeled data.
Synthesizing the Missing Labels
When real-world video annotations are too scarce or expensive to acquire—such as in specialized domains like video matting or surgical AI—researchers turn to data synthesis and simulation.
In Lucid Data Dreaming for Video Object Segmentation, Anna Khoreva et al. show that a model can be trained effectively by synthesizing thousands of plausible "future" video frames from just a single annotated static image. This in-domain data generation bypasses the need for massive, manually labeled video datasets.
For tasks requiring perfect ground truth across complex physical movements, virtual environments offer an elegant escape. Enrico Meloni et al. demonstrate in Evaluating Continual Learning Algorithms by Generating 3D Virtual Environments how simulated 3D streams can automatically generate flawless pixel-level labels like optical flow and semantic segmentation, completely eliminating human labor from the data-generation loop.
Key Foundational Work
Semantic object segmentation via detection in weakly labeled video (2015) — contributed 1 concept elements to this direction.
Semi-Supervised Domain Adaptation for Weakly Labeled Semantic Video Object Segmentation (2016) — contributed 1 concept elements to this direction.
Multi-class Semantic Video Segmentation with Exemplar-Based Object Reasoning (2015) — contributed 1 concept elements to this direction.