Towards Segmenting Anything That Moves

Achal Dave, Pavel Tokmakov, Deva Ramanan

Abstract

Detecting and segmenting individual objects, regardless of their category, is crucial for many applications such as action detection or robotic interaction. While this problem has been well-studied under the classic formulation of spatio-temporal grouping, state-of-the-art approaches do not make use of learning-based methods. To bridge this gap, we propose a simple learning-based approach for spatio-temporal grouping. Our approach leverages motion cues from optical flow as a bottom-up signal for separating objects from each other. Motion cues are then combined with appearance cues that provide a generic objectness prior for capturing the full extent of objects. We show that our approach outperforms all prior work on the benchmark FBMS dataset. One potential worry with learning-based methods is that they might overfit to the particular type of objects that they have been trained on. To address this concern, we propose two new benchmarks for generic, moving object detection, and show that our model matches top-down methods on common categories, while significantly out-performing both top-down and bottom-up methods on never-before-seen categories.

Introduction

Modern computer vision systems excel in "closed-world" scenarios where the categories of interest are predefined. However, autonomous agents operating in the real world must handle "open-world" perception—the ability to detect and segment any physical object, even those never encountered during training. Traditional methods relied on bottom-up spatio-temporal grouping based on the principle of "common fate," but these have been surpassed by deep learning models that are typically category-specific. This research introduces a learning-based framework that generalizes motion segmentation to any moving object.

Methodology: Two-Stream Spatio-Temporal Grouping

The proposed architecture extends the Mask R-CNN framework with a novel Two-Stream Region Proposal Network (RPN) that fuses appearance and motion cues.

  1. Appearance Stream: Trained on the MS COCO dataset, this stream collapses 80 specific categories into a single "object" class. This forces the network to learn a generic "objectness" prior rather than class-specific features.
  2. Motion Stream: This stream processes optical flow. Due to the scarcity of labeled real-world video, it is trained on the synthetic FlyingThings3D dataset. It learns to group pixels based purely on motion disparities.
  3. Fusion and Tracking: Features from both streams are concatenated in a joint RPN. This allows the model to use motion as a filter to identify potential objects and appearance as a refiner to delineate their precise boundaries.
Figure 1: Comparison of initial motion-based proposals (left) versus the final fused segmentation (right) for novel categories like an octopus and a rat.

Experimental Results

The model was evaluated on the classic FBMS-59 benchmark and two new, more rigorous benchmarks: DAVIS-Moving and YTVOS-Moving. These new benchmarks require exhaustive labeling of all moving objects, providing a more realistic assessment of open-world performance.

  • Benchmark Performance: The approach outperformed the previous state-of-the-art on FBMS-59 by 11.4% using a stricter F-measure that penalizes over-segmentation.
  • Generalization to Unseen Categories: On the YTVOS-Moving dataset, the model achieved a 68.2% F-measure on novel objects, significantly higher than a standard Mask R-CNN (40.6%), which failed to detect categories outside its training set.
  • Motion vs. Appearance: The results confirm that while motion cues are excellent for discovery, the appearance stream is vital for capturing the full extent of an object.

Conclusion

By combining synthetic motion training with generic appearance priors, this work demonstrates that learning-based methods can achieve robust open-world segmentation. Future work will focus on improving tracking through complex occlusions and resolving ambiguities caused by camera-induced parallax.

Summary updated

Abstract

Detecting and segmenting individual objects, regardless of their category, is crucial for many applications such as action detection or robotic interaction. While this problem has been well-studied under the classic formulation of spatio-temporal grouping, state-of-the-art approaches do not make use of learning-based methods. To bridge this gap, we propose a simple learning-based approach for spatio-temporal grouping. Our approach leverages motion cues from optical flow as a bottom-up signal for separating objects from each other. Motion cues are then combined with appearance cues that provide a generic objectness prior for capturing the full extent of objects. We show that our approach outperforms all prior work on the benchmark FBMS dataset. One potential worry with learning-based methods is that they might overfit to the particular type of objects that they have been trained on. To address this concern, we propose two new benchmarks for generic, moving object detection, and show that our model matches top-down methods on common categories, while significantly out-performing both top-down and bottom-up methods on never-before-seen categories.
BibTeX
@inproceedings{dave2019towards,
  author = {Achal Dave and Pavel Tokmakov and Deva Ramanan},
  title = {{Towards Segmenting Anything That Moves}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  doi = {10.1109/iccvw.2019.00187},
}
Figure