Mask Selection and Propagation for Unsupervised Video Object Segmentation

Shubhika Garg, Vidit GoelWACV 2021 · OpenReview

Abstract

In this work we present a novel approach for Unsupervised Video Object Segmentation, that is automatically generating instance level segmentation masks for salient objects and tracking them in a video. We efficiently handle problems present in existing methods such as drift while temporal propagation, tracking and addition of new objects. To this end, we propose a novel idea of improving masks in an online manner using ensemble of criteria whose task is to inspect the quality of masks. We introduce a novel idea of assessing mask quality using a neural network called Selector Net. The proposed network is trained is such way that it is generalizes across various datasets. Our proposed method is able to limit the noise accumulated along the video, giving state of the art result on Davis 2019 Unsupervised challenge dataset with $\mathcal{J}\& {\mathcal{F}}$ mean 61.6%. We also tested on datasets such as FBMS and SegTrack V2 and performed better or on par compared to the other methods.

Introduction

Unsupervised Video Object Segmentation (UVOS) aims to automatically identify and segment salient objects in a video sequence without any human intervention or initial ground-truth masks. This task is significantly more challenging than semi-supervised VOS because the algorithm must determine which objects to track while simultaneously maintaining mask quality over time. Two primary challenges plague current UVOS methods: drift, where errors accumulate during temporal propagation, and noise, where background elements are incorrectly identified as objects.

Methodology

The proposed framework addresses these challenges through a modular three-stage pipeline that combines object detection with intelligent temporal propagation.

Overview of the proposed 3-stage pipeline for Unsupervised Video Object Segmentation.

Stage 1: Object Mask Generation

The process begins by using Mask R-CNN to detect potential objects in each frame. To minimize initial noise, the system retains only the top 10 detections and applies filters to remove small, likely erroneous fragments.

Stage 2: Temporal Propagation and Online Selection

This stage is the core innovation of the research. The system leverages Space-Time Memory Networks (STM) to propagate masks from previous frames to the current frame $t$. This results in two candidate masks for each object: one from the detector (Mask R-CNN) and one from the propagator (STM).

To select the optimal mask, the authors introduce Selector Net, a ResNet-18-based architecture designed to act as a "judge." Selector Net evaluates the relationship between the RGB image and the candidate mask to assign a quality score.

Architecture of the Selector Net used to evaluate mask quality.

In addition to Selector Net, an "Area Criterion" is employed as a secondary check, favoring masks that maintain consistent surface area across frames. This ensemble approach ensures that the most stable and accurate mask is propagated forward, effectively mitigating drift.

Stage 3: Offline Selection

A final offline pass is conducted after processing the entire video. This stage helps recover objects that may have been lost due to temporary occlusions and ensures global consistency of object identities throughout the sequence.

Experiments and Results

The model was evaluated on several benchmark datasets, including DAVIS 2019 (Unsupervised Challenge), FBMS, and SegTrack V2.

  • DAVIS 2019: The method achieved a state-of-the-art $\mathcal{J}\& {\mathcal{F}}$ mean of 61.6%, representing a 5.2% improvement over previous benchmarks.
  • Ablation Study: Removing the Selector Net (Stage 2) resulted in an 8% drop in the J&F score, confirming that the online selection process is critical for preventing error accumulation.
  • Generalization: Selector Net demonstrated strong generalizability, performing well across different datasets without requiring video-specific retraining.

Conclusion

By reframing UVOS as a selection problem between detection and propagation, this work provides a robust solution to the problem of drift. The modular "judge" approach using Selector Net allows for the integration of future advancements in detection and propagation models. While the current pipeline's inference speed (approx. 2.2 seconds per frame) remains a bottleneck for real-time applications, the significant gains in accuracy establish a new baseline for unsupervised segmentation tasks. Future work may explore using Selector Net's feedback to refine masks rather than just selecting from existing candidates.

Summary updated

Abstract

In this work we present a novel approach for Unsupervised Video Object Segmentation, that is automatically generating instance level segmentation masks for salient objects and tracking them in a video. We efficiently handle problems present in existing methods such as drift while temporal propagation, tracking and addition of new objects. To this end, we propose a novel idea of improving masks in an online manner using ensemble of criteria whose task is to inspect the quality of masks. We introduce a novel idea of assessing mask quality using a neural network called Selector Net. The proposed network is trained is such way that it is generalizes across various datasets. Our proposed method is able to limit the noise accumulated along the video, giving state of the art result on Davis 2019 Unsupervised challenge dataset with $\mathcal{J}\& {\mathcal{F}}$ mean 61.6%. We also tested on datasets such as FBMS and SegTrack V2 and performed better or on par compared to the other methods.
BibTeX
@inproceedings{garg2021mask,
  author = {Shubhika Garg and Vidit Goel},
  title = {{Mask Selection and Propagation for Unsupervised Video Object Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=k6LiN184qF},
}
Figure