Abstract
We describe an unsupervised method to detect and segment portions of images of live scenes that, at some point in time, are seen moving as a coherent whole, which we refer to as objects. Our method first partitions the motion field by minimizing the mutual information between segments. Then, it uses the segments to learn object models that can be used for detection in a static image. Static and dynamic models are represented by deep neural networks trained jointly in a bootstrapping strategy, which enables extrapolation to previously unseen objects. While the training process requires motion, the resulting object segmentation network can be used on either static images or videos at inference time. As the volume of seen videos grows, more and more objects are seen moving, priming their detection, which then serves as a regularizer for new objects, turning our method into unsupervised continual learning to segment objects. Our models are compared to the state of the art in both video object segmentation and salient object detection. In the six benchmark datasets tested, our models compare favorably even to those using pixel-level supervision, despite requiring no manual annotation.
Introduction
In computer vision, teaching a machine to segment an object—identifying exactly which pixels belong to it—traditionally requires massive amounts of human-annotated data. This reliance on manual labeling is expensive and fails to scale to the infinite variety of objects in the real world.
Nature provides a "free" teacher: motion. Animals and infants use common motion to identify objects; if a group of pixels moves together independently of the background, it is likely a single entity. While unsupervised motion segmentation is not a new concept, it often fails when objects stop moving or when the camera itself is in motion. DyStaB (Dynamic-Static Bootstrapping) addresses these challenges by building a robust object detector that learns from motion but can eventually recognize objects even when they are perfectly still.
Methods: Dynamic-Static Bootstrapping
The core of the DyStaB framework is a dual-model architecture that "bootstraps" itself through a synergistic loop between motion and appearance.
1. The Dynamic Model ($\phi$)
This network analyzes optical flow patterns using a novel Temporally Consistent Mutual Information Minimization strategy. It seeks to find a mask such that the motion inside the mask and the motion outside are as independent as possible. To ensure stability, the model enforces temporal consistency, preventing the object's shape from radically changing between consecutive frames.
2. The Static Model ($\chi$)
This network processes single RGB images. It is trained using the masks generated by the Dynamic Model as "pseudo-labels," allowing it to learn the appearance (color, texture, shape) of objects without manual supervision.
3. The Synergistic Loop
Once the static model learns the appearance of an object (e.g., a lizard) because it saw it move, it can assist the dynamic model in future frames. If the object stops moving or the motion data becomes noisy, the static model provides a "top-down" prior to maintain accurate segmentation. The system uses mutual information loss as a reliability gauge; if motion data is low-confidence, it relies more on the static model's previous knowledge.
Key Insights and Results
DyStaB demonstrates that the physics of independent motion provides a sufficient signal to train deep neural networks for complex segmentation tasks.
- Continual Learning: The model exhibits a "the more you see, the more you know" property. As it watches more videos, its internal definition of "objectness" refines, allowing it to detect objects it has never seen move before (like a sloth) simply because they share appearance properties with previously observed moving objects.
- Performance: DyStaB outperformed the previous state-of-the-art unsupervised method (CIS) by approximately 7% across major benchmarks. Remarkably, it achieved 80.0 mIoU on the DAVIS dataset, surpassing several models trained with thousands of manual annotations (e.g., FSEG and PDB).
- Scalability: Performance steadily improved as the training data increased from 12 to 108 videos, suggesting a high ceiling for improvement with larger datasets.
Conclusion
While DyStaB represents a significant step forward, several challenges remain. The definition of an "object" can be subjective (e.g., a rider vs. a motorcycle), and the model does not yet handle hierarchical membership (understanding that a helmet is part of a rider). Future work may involve incorporating other signals like sound or tactile feedback and integrating attention mechanisms to focus on specific salient targets in complex scenes.
Summary updated