Abstract
We consider the problem of providing dense segmentation masks for object discovery in videos. We formulate the object discovery problem as foreground motion clustering, where the goal is to cluster foreground pixels in videos into different objects. We introduce a novel pixel-trajectory recurrent neural network that learns feature embeddings of foreground pixel trajectories linked across time. By clustering the pixel trajectories using the learned feature embeddings, our method establishes correspondences between foreground object masks across video frames. To demonstrate the effectiveness of our framework for object discovery, we conduct experiments on commonly used datasets for motion segmentation, where we achieve state-of-the-art performance.
Introduction
Standard object detection and tracking models are typically constrained by their training on predefined categories like pedestrians or vehicles. However, for autonomous systems to truly navigate and understand novel environments, they must be capable of object discovery—identifying and segmenting entities they have never seen before. This paper defines an object by its motion: an entity that moves or can be moved. The primary challenge lies in separating these moving foreground objects from the background and maintaining consistent segmentation across frames, especially during occlusions or when motion temporarily ceases.
Methodology
The proposed framework treats object discovery as a foreground motion clustering problem. It leverages a multi-stage pipeline to transform raw video data into distinct object instances.
Architecture Overview
The system utilizes a two-stream encoder-decoder called Y-Net, which performs mid-level fusion of RGB images and optical flow. This network extracts dense pixel-wise embeddings and predicts a binary foreground mask for each frame.
Pixel Trajectory RNN (PT-RNN)
The core innovation is the PT-RNN. First, foreground pixels are linked across frames using flow consistency, guided by the predicted foreground masks to prevent background drift. The PT-RNN then processes these variable-length trajectories, aggregating Y-Net embeddings into fixed-length feature vectors on a unit sphere. This allows for direct comparison and clustering of trajectories regardless of their lifespan.
Clustering and Discovery
Final object discovery is achieved by clustering these trajectory embeddings using von Mises-Fisher mean shift (vMF-MS). A contrastive loss function ensures that trajectories belonging to the same object are pulled together in the embedding space, while those from different objects are pushed apart.
Experimental Results
The framework was evaluated across several benchmark datasets, including FBMS, DAVIS2016, and specialized sets like Complex Background (CB) and Camouflaged Animal (CA).
Performance Metrics
As shown in the results table, the method achieves state-of-the-art F-scores on the FBMS dataset (87.7%) and the CA dataset (77.1%). On DAVIS2016, it achieves a J-measure of 74.2 without requiring CRF post-processing. The system is also computationally efficient, running at approximately 15 FPS on a single GPU once optical flow is computed.
Conclusion and Future Work
This approach demonstrates that motion is a powerful, universal signal for object discovery. By mapping trajectories to a common embedding space, the system bypasses the need for category-specific labels. However, challenges remain regarding GPU memory consumption for long videos and the heavy reliance on the initial foreground mask quality. Future work may explore more robust clustering techniques to prevent "cluster collapse" where similar motions are incorrectly merged.
Summary updated