Towards Segmenting Everything That Moves

Achal Dave, Pavel Tokmakov, Deva Ramanan

Abstract

Detecting and segmenting individual objects, regardless of their category, is crucial for many applications such as action detection or robotic interaction. While this problem has been well-studied under the classic formulation of spatio-temporal grouping, state-of-the-art approaches do not make use of learning-based methods. To bridge this gap, we propose a simple learning-based approach for spatio-temporal grouping. Our approach leverages motion cues from optical flow as a bottom-up signal for separating objects from each other. Motion cues are then combined with appearance cues that provide a generic objectness prior for capturing the full extent of objects. We show that our approach outperforms all prior work on the benchmark FBMS dataset. One potential worry with learning-based methods is that they might overfit to the particular type of objects that they have been trained on. To address this concern, we propose two new benchmarks for generic, moving object detection, and show that our model matches top-down methods on common categories, while significantly out-performing both top-down and bottom-up methods on never-before-seen categories.

Introduction

Modern computer vision systems excel in "closed-world" scenarios, where they are trained to recognize a predefined set of categories, such as the 80 classes in MS COCO. However, real-world applications like autonomous driving or robotics require systems to perceive "unseen" objects—ranging from highway debris to specialized equipment. To achieve true autonomy, systems must move beyond category-specific recognition toward generic spatio-temporal grouping. This approach relies on the principle of "common fate": pixels that move together likely belong to the same object. While a classic problem, existing learning-based methods often struggle to generalize beyond their training data.

Methodology

The proposed architecture utilizes a two-stream approach to integrate motion and appearance cues:

  1. Motion Stream: This stream identifies moving regions using optical flow. To avoid category bias and overcome the scarcity of real-world motion labels, it is trained on the synthetic FlyingThings3D dataset. This allows the model to learn low-level motion geometry applicable to any object.
  2. Appearance Stream: A modified Mask R-CNN provides a "generic objectness" prior. By collapsing all training categories into a single "thing" class, the model learns to identify the full extent of objects based on general appearance rather than specific class features.
  3. Fusion and Tracking: A novel Region Proposal Network (RPN) fuses features from both streams. A stage-wise training strategy, including weight freezing, preserves the general knowledge of each stream. Finally, an overlap-based tracker maintains object identity over time, even during temporary pauses in motion.

Experimental Setup

The model was evaluated on the classic FBMS (Freiburg Berkeley Motion Segmentation) benchmark and DAVIS 2016. To specifically test open-world performance, the authors introduced DAVIS-Moving and YTVOS-Moving. These benchmarks require the detection of every moving object, providing a rigorous test for generalization to categories not seen during training.

Key Results

The proposed method achieves state-of-the-art results on the FBMS benchmark, outperforming previous methods by 11.4% on a more rigorous F-measure that penalizes false positives.

Qualitative comparison of segmentation results on novel categories (octopus and rat).

On the YTVOS-Moving dataset, the model demonstrates significant generalization capabilities. It achieved an F-score of 67.7 on novel objects, compared to 40.6 for a standard Mask R-CNN baseline. Qualitative results (see figure above) show successful segmentation of complex, never-before-seen objects like octopuses and rats, where traditional models typically fail.

Discussion and Future Work

A key insight is that focusing on "objectness" rather than classification provides a nearly 8% boost in mean Average Precision (mAP) for unseen categories. By leveraging synthetic data for motion, the model sidesteps the "semantic bottleneck" and relies on physical principles of motion.

Current limitations include tracking failures during long occlusions and occasional "guilt by association" errors where static objects near moving ones are misclassified. Future work will explore more sophisticated temporal modeling, such as Recurrent Neural Networks, to improve long-term tracking and the discovery of novel stationary objects.

Summary updated

Abstract

Detecting and segmenting individual objects, regardless of their category, is crucial for many applications such as action detection or robotic interaction. While this problem has been well-studied under the classic formulation of spatio-temporal grouping, state-of-the-art approaches do not make use of learning-based methods. To bridge this gap, we propose a simple learning-based approach for spatio-temporal grouping. Our approach leverages motion cues from optical flow as a bottom-up signal for separating objects from each other. Motion cues are then combined with appearance cues that provide a generic objectness prior for capturing the full extent of objects. We show that our approach outperforms all prior work on the benchmark FBMS dataset. One potential worry with learning-based methods is that they might overfit to the particular type of objects that they have been trained on. To address this concern, we propose two new benchmarks for generic, moving object detection, and show that our model matches top-down methods on common categories, while significantly out-performing both top-down and bottom-up methods on never-before-seen categories.
BibTeX
@inproceedings{dave2019towards,
  author = {Achal Dave and Pavel Tokmakov and Deva Ramanan},
  title = {{Towards Segmenting Everything That Moves}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  doi = {10.1109/iccvw.2019.00187},
}
Figure