Key Instance Selection for Unsupervised Video Object Segmentation

Donghyeon Cho, Sungeun Hong, Sungil Kang, Jiwon KimarXiv 2019 · arXiv

Abstract

Unsupervised video object segmentation (UVOS) requires identifying and tracking salient objects without initial human-provided masks. This paper presents a "Key Instance Selection" strategy that analyzes video saliency and appearance frequency in early frames to select a fixed number of primary targets. This approach mitigates the "ID explosion" problem common in unsupervised methods, where the number of tracked objects grows uncontrollably. The system achieved a 3rd place ranking in the DAVIS 2019 Unsupervised Challenge.

Introduction

In semi-supervised video object segmentation (SVOS), a human provides a mask for the first frame, defining the tracking targets. In contrast, unsupervised video object segmentation (UVOS) requires the system to autonomously determine which objects are significant.

A naive UVOS approach—tracking every object proposal—faces two major hurdles:

  1. Complexity Explosion: The number of tracked IDs increases indefinitely, leading to excessive memory and processing demands.
  2. Noise: Many proposals are background clutter or transient artifacts that do not represent meaningful objects.

The authors address these by identifying a fixed number ($K$) of "key" instances early in the video, focusing resources on the most salient targets.

Methodology

The proposed pipeline follows a "propose, link, and select" architecture.

Overview of the proposed UVOS pipeline featuring instance segmentation, mask propagation, and the key instance selection mechanism.

Candidate Generation and Linking

The system utilizes Mask R-CNN and DeepLab to generate object proposals. To maintain continuity during motion blur or occlusions, masks are propagated from previous frames using RGMP. An online tracker links new proposals to existing IDs based on:

  • IoU (Intersection over Union): Spatial overlap between consecutive frames.
  • Trajectory: Predicted bounding box locations.
  • Re-identification (ReID): Comparison of appearance features against a historical "object pool."
  • Semantic Co-segmentation: Category-level feature consistency.

Key Instance Selection

This core component operates during the first $M$ frames. The system collects all potential IDs and, at frame $M$, calculates a Selection Score for each based on:

  1. Video Saliency: How much the object stands out or moves relative to the background.
  2. Frequency: The consistency of the ID's appearance across the initial frames.

Only the top $K$ IDs are retained for tracking throughout the remainder of the video. Hyperparameters for this selection were optimized using a Meta AI search on a 144-GPU cluster.

Experimental Results

The method was evaluated on the UVOS DAVIS challenge dataset (30 videos), which lacks initial masks.

Performance comparison on the DAVIS 2019 Unsupervised Challenge leaderboard. The proposed method (SK T-Brain) achieved 3rd place.

The system ranked 3rd overall with a Global Mean ($J\&F$) score of 0.516. Key findings include:

  • Selection Efficiency: The "Key Instance" method significantly outperformed random selection. For $K=5$, the method achieved a score of 0.576, compared to 0.474 for random selection.
  • Robustness: Qualitative analysis showed the model successfully handling fast motion (e.g., "mbike-trick") and multiple similar objects (e.g., "lab-coat").

Discussion and Key Insights

The primary insight is that "importance" is as critical as "objectness." By leveraging video saliency, the model distinguishes primary actors (e.g., a scooter rider) from background elements (e.g., trees). Furthermore, the use of an Object Pool—comparing candidates to a history of ReID descriptors rather than just the previous frame—enhances robustness against occlusions.

Limitations

  • Temporal Bottleneck: Objects entering after the initial $M$ frames may be ignored.
  • Fixed $K$: A static number of instances may not suit all videos; an adaptive $K$ could be more effective.
  • Initial Computational Load: The selection phase requires running multiple heavy models (Mask R-CNN, Saliency, etc.) simultaneously.

Summary updated

Abstract

This paper proposes key instance selection based on video saliency covering objectness and dynamics for unsupervised video object segmentation (UVOS). Our method takes frames sequentially and extracts object proposals with corresponding masks for each frame. We link objects according to their similarity until the M-th frame and then assign them unique IDs (i.e., instances). Similarity measure takes into account multiple properties such as ReID descriptor, expected trajectory, and semantic co-segmentation result. After M-th frame, we select K IDs based on video saliency and frequency of appearance; then only these key IDs are tracked through the remaining frames. Thanks to these technical contributions, our results are ranked third on the leaderboard of UVOS DAVIS challenge.
BibTeX
@misc{cho2019key,
  author = {Donghyeon Cho and Sungeun Hong and Sungil Kang and Jiwon Kim},
  title = {{Key Instance Selection for Unsupervised Video Object Segmentation}},
  year = {2019},
  eprint = {1906.07851},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.07851},
}
Figure