Abstract
We describe an unsupervised method to detect and segment portions of images of live scenes that, at some point in time, are seen moving as a coherent whole, which we refer to as objects. Our method first partitions the motion field by minimizing the mutual information between segments. Then, it uses the segments to learn object models that can be used for detection in a static image. Static and dynamic models are represented by deep neural networks trained jointly in a bootstrapping strategy, which enables extrapolation to previously unseen objects. While the training process requires motion, the resulting object segmentation network can be used on either static images or videos at inference time. As the volume of seen videos grows, more and more objects are seen moving, priming their detection, which then serves as a regularizer for new objects, turning our method into unsupervised continual learning to segment objects. Our models are compared to the state of the art in both video object segmentation and salient object detection. In the six benchmark datasets tested, our models compare favorably even to those using pixel-level supervision, despite requiring no manual annotation.
Introduction
In computer vision, "objectness" is traditionally defined through manual annotations—thousands of hours spent by humans tracing boundaries. However, biological systems, such as human infants, learn to perceive objects largely through motion: if a group of pixels moves coherently against a background, it likely constitutes an object.
The challenge in artificial intelligence has been two-fold: motion-based segmentation typically fails the moment an object stops moving, and learning static appearance from scratch without labels is notoriously difficult. DyStaB (Dynamic-Static Bootstrapping) addresses these issues by using motion as a natural supervisor to "prime" a static image detector, which then feeds back to improve motion segmentation.
Methodology
The core of DyStaB is a synergistic loop between two deep neural networks:
- The Dynamic Model ($\phi$): Operating on optical flow (motion fields), this module utilizes Mutual Information (MI) Minimization. The principle is that a correct partition of "object" and "background" results in motion within the object being maximally independent of the motion outside. To ensure stability, a Temporal Consistency Constraint is applied, keeping object masks stable across consecutive video frames.
- The Static Model ($\chi$): This standard image segmentation network takes a single RGB frame as input and is trained using the masks generated by the Dynamic Model as "pseudo-labels."
- The Bootstrapping Loop: This is the system's "secret sauce." Through a Confidence-Aware Update, the static model avoids blindly trusting noisy motion masks. If motion is uninformative (e.g., the object is still), the system relies on its own previous knowledge. Conversely, once the static model learns object appearances, its predictions are fed back into the dynamic model as a "top-down prior," helping segment objects even when their motion is subtle or ambiguous.
Key Insights
- Motion as a "Free" Teacher: Motion provides a high-quality supervisory signal inherent in video data, bypassing the need for human annotators.
- Solving Camouflage: As illustrated in the case of a camouflaged lizard, objects invisible when still become obvious when they move. By learning appearance during motion, the static model can continue to recognize the object after it stops.
- Unsupervised Continual Learning: Performance improves as the model processes more diverse videos. There is a clear correlation between the variety of videos "watched" and the model's ability to segment objects in novel static images.
Experimental Results
DyStaB was evaluated across several rigorous benchmarks, including DAVIS, FBMS, and SegTrackV2 for video, and MSRA-B, ECCSD, and DUT for static saliency detection.
- State-of-the-Art Performance: DyStaB outperformed previous unsupervised methods like CIS by approximately 7% on the DAVIS benchmark, achieving a mean Intersection-over-Union (mIoU) of 80.0.
- Competitive with Supervision: Remarkably, this unsupervised approach performed comparably to, and sometimes better than, models trained with thousands of manual labels (e.g., FSEG and LVO).
- Temporal Robustness: The temporal consistency constraint significantly reduced "label flipping," a common failure point where models confuse foreground and background across frames.
Discussion and Limitations
While DyStaB represents a significant step toward unsupervised learning, it faces challenges with hierarchical or nested objects (e.g., a person on a motorcycle). The definition of an "object" remains somewhat tied to the motion patterns seen during training. Future work may involve incorporating multi-modal signals, such as sound or touch, to further refine the concept of objectness. Despite these challenges, DyStaB demonstrates that watching the world move is a powerful, label-free way to learn to see.
Summary updated