Time-Supervised Primary Object Segmentation

Yanchao Yang, Brian Lai, Stefano Soatto

Abstract

We describe an unsupervised method to detect and segment portions of images of live scenes that, at some point in time, are seen moving as a coherent whole, which we refer to as objects. Our method first partitions the motion field by minimizing the mutual information between segments. Then, it uses the segments to learn object models that can be used for detection in a static image. Static and dynamic models are represented by deep neural networks trained jointly in a bootstrapping strategy, which enables extrapolation to previously unseen objects. While the training process requires motion, the resulting object segmentation network can be used on either static images or videos at inference time. As the volume of seen videos grows, more and more objects are seen moving, priming their detection, which then serves as a regularizer for new objects, turning our method into unsupervised continual learning to segment objects. Our models are compared to the state of the art in both video object segmentation and salient object detection. In the six benchmark datasets tested, our models compare favorably even to those using pixel-level supervision, despite requiring no manual annotation.

Introduction

In computer vision, object segmentation—the task of outlining every pixel belonging to an object—is notoriously expensive, typically requiring thousands of hours of human labor to manually trace boundaries. However, biological systems learn to segment objects without pixel-perfect labels by observing motion. In "DyStaB: Unsupervised Object Segmentation via Dynamic-Static Bootstrapping," researchers present a method that mimics this process. By leveraging the fact that objects tend to move as coherent wholes, the system learns to segment objects in both videos and static images without a single manual label.

Methodology: The Synergistic Loop

The core of the DyStaB framework is a "synergistic loop" between two neural networks that allows the system to bootstrap its own learning process.

The DyStaB architecture showing the synergistic loop between the Dynamic model (phi) and the Static model (chi).
  1. The Dynamic Model ($\phi$): This network analyzes optical flow—the pattern of apparent motion between frames. It partitions the motion field by minimizing the mutual information between segments, effectively finding boundaries where motion on one side provides minimal information about motion on the other. A temporal consistency constraint ensures these masks remain stable over time.
  2. The Static Model ($\chi$): A standard segmentation network (based on DeepLab) that processes single RGB images. It learns from the "pseudo-labels" generated by the dynamic model.
  3. The Bootstrapping Loop: The dynamic model identifies moving objects, and these masks train the static model to recognize them based on appearance alone. Subsequently, the static model’s predictions are fed back into the dynamic model as an "objectness prior," allowing the system to handle cases where motion is noisy or absent.

Key Insights: Motion as the Ultimate Teacher

Motion provides a high-confidence signal that "primes" the model to learn visual features like color, texture, and shape. This is particularly powerful for identifying objects that are otherwise difficult to distinguish from their environment.

Motion cues reveal objects that are nearly invisible in static images due to camouflage (a) or complex backgrounds (b).

Even a perfectly camouflaged animal becomes easy to spot once it moves. By using the dynamic model's reconstruction loss as a confidence measure, the system only updates the static model when the motion signal is clear. This leads to a form of unsupervised continual learning: as the system sees more videos, its "vocabulary" of objects grows, improving its ability to segment new, unseen objects.

Experimental Results

DyStaB was evaluated across several benchmarks, including DAVIS, FBMS, and SegTrackV2 for video, and MSRA-B and ECCSD for static images. The results demonstrate that the model achieved state-of-the-art performance for unsupervised methods and, in several instances, outperformed supervised models:

  • Video Object Segmentation: On the DAVIS dataset, DyStaB achieved a 62.4 mIoU, outperforming the previous unsupervised baseline (CIS) by over 7%.
  • Static Saliency: When tested on static images without motion, the model achieved an 88.1 score on ECCSD, surpassing many specialized saliency models trained with explicit human supervision.

Conclusion

While DyStaB defines objects purely by common motion—meaning it may struggle to distinguish between a rider and a motorcycle if they always move together—it proves that a synergistic loop between motion and appearance can rival manual supervision. This research opens a promising path toward AI that learns about the world's physical structure simply by watching it move.

Summary updated

Abstract

We describe an unsupervised method to detect and segment portions of images of live scenes that, at some point in time, are seen moving as a coherent whole, which we refer to as objects. Our method first partitions the motion field by minimizing the mutual information between segments. Then, it uses the segments to learn object models that can be used for detection in a static image. Static and dynamic models are represented by deep neural networks trained jointly in a bootstrapping strategy, which enables extrapolation to previously unseen objects. While the training process requires motion, the resulting object segmentation network can be used on either static images or videos at inference time. As the volume of seen videos grows, more and more objects are seen moving, priming their detection, which then serves as a regularizer for new objects, turning our method into unsupervised continual learning to segment objects. Our models are compared to the state of the art in both video object segmentation and salient object detection. In the six benchmark datasets tested, our models compare favorably even to those using pixel-level supervision, despite requiring no manual annotation.
BibTeX
@inproceedings{yang2021time,
  author = {Yanchao Yang and Brian Lai and Stefano Soatto},
  title = {{Time-Supervised Primary Object Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=I2THQBNUxl},
}
Figure