Unsupervised Learning from Videos for Object Discovery in Single Images

Dong Zhao, Baoqing Ding, Yulin Wu, Lei Chen, Hongchao ZhouSymmetry 2020 · DOI

TL;DR

This paper introduces an unsupervised learning framework that leverages video data to discover primary objects in single images. By training an encoder-decoder network to decompose video frames into foreground, background, and a segmentation mask, the model learns to identify and segment objects without any manual annotations, offering a significant step towards truly unsupervised visual understanding.

What Problem Does This Paper Address?

The core challenge this paper tackles is unsupervised object discovery and segmentation in images. Traditional computer vision tasks often rely heavily on large, meticulously labeled datasets. However, acquiring such labels is time-consuming and expensive. While unsupervised learning aims to alleviate this, existing methods often still require some form of supervision during training or rely on pre-trained models. Furthermore, many unsupervised object discovery methods are designed for collections of images or videos, making them slow for inference on single images. This paper addresses the need for a method that can learn to identify and segment primary objects from single images, using only unlabeled video data for training, and then perform inference efficiently on individual frames.

Methods

The proposed method, termed UnsupOD (Unsupervised Object Discovery), employs an encoder-decoder architecture trained on videos. The key idea is to factorize each video frame into three components: a foreground, a background, and a segmentation mask. This factorization is achieved through a neural network that learns to reconstruct the original input frame.

Here's a breakdown of the novel aspects:

  • Video-based Unsupervised Learning: The training process exclusively uses unlabeled video sequences. The inter-frame structure and motion cues within videos are crucial for learning.
  • Sparsity Representation: The paper hypothesizes that images are sparse, meaning they primarily consist of distinct objects. The network is designed to exploit this sparsity.
  • Encoder-Decoder Architecture: A fully convolutional encoder-decoder network is used. Crucially, it lacks skip connections to encourage the network to learn more abstract representations.
  • Component-wise Constraints: To guide the unsupervised learning of foreground, background, and mask, specific regularization losses are introduced:
  • Foreground/Background Model: The background network has a narrower bottleneck than the foreground network, forcing it to learn a lower-dimensional representation of the background. A gradient regularization is applied to the background to ensure it's "clean" and free of foreground elements.
  • Segmentation Mask Model: A U-Net architecture is used for the mask. It's regularized to be close to a binary image (using a binary loss) and to have smooth contours without holes (using a closed regularization loss). An "area regularization" is also introduced to control the mask's size and prevent it from encompassing the entire background.
  • Reconstruction Objective: The ultimate goal is to reconstruct the input image by combining the predicted foreground, background, and mask. This reconstruction loss drives the learning process.
  • Efficient Single-Image Inference: After training on videos, the network can directly process single images to output foreground, background, and a segmentation mask, making it efficient for inference.

Datasets

The paper utilizes several datasets for training and evaluation:

  • YouTube Objects (YTO) dataset: Used for unsupervised training. This dataset contains videos of 10 diverse object classes with significant clutter, viewpoint changes, and occlusions.
  • Object Discovery dataset: Used for evaluating object discovery and segmentation on single images. It contains internet images of airplanes, cars, and horses with detailed segmentation masks.
  • MSRC dataset: Used for evaluating generalization on unseen classes. It comprises photographs of 21 object classes with hand-labeled segmentation masks.
  • iCoseg dataset: Also used for evaluating generalization on unseen classes. It's built from Flickr images and contains challenging groups with pixel-level segmentations.

Key Insights and Ideas

  • Videos as a Rich Source of Unsupervised Supervision: The paper strongly advocates for using the inherent temporal structure and motion in videos as a powerful source of self-supervision for learning object representations, even when the ultimate goal is single-image analysis.
  • Sparsity is Key: The assumption that images are sparse (composed of distinct objects) is fundamental. By forcing the network to represent these components separately and then reconstruct the whole, it implicitly learns to identify what constitutes an "object."
  • Decomposition for Discovery: The core idea of decomposing an image into foreground, background, and a mask, even without explicit labels for these components, is a clever way to force the network to learn meaningful visual features that distinguish objects from their surroundings.
  • Auxiliary Tasks for Better Masks: Using foreground/background separation and reconstruction as auxiliary tasks helps to produce cleaner and more refined segmentation masks, even though these intermediate outputs are not the primary focus.
  • Generalization to Unseen Classes: The success on datasets with classes not seen during training highlights the learned high-level semantic features, suggesting the model learns more than just specific object appearances.

Key Results

  • Quantitative Performance:
    • On the YouTube Objects dataset, the UnsupOD model achieved the best overall performance in terms of Correct Localization (CorLoc) across 10 classes, outperforming several state-of-the-art methods in six classes.
  • It achieved a significant improvement over co-segmentation methods on the Object Discovery dataset, particularly for the challenging "horse" category.
  • On the Object Discovery dataset, UnsupOD achieved the best P and J metrics for segmentation masks across all three categories.
  • The model is remarkably fast during inference, processing each image in 0.03 seconds, which is orders of magnitude faster than many compared methods.
  • Qualitative Findings:
    • The model successfully separates primary objects from their backgrounds.
  • The background model can effectively fill in "missing" image parts.
  • Produced object masks are smooth, capture fine details, and accurately represent figure-ground contrast.
  • The model can detect multiple objects within a single image.
  • The learned bounding boxes can sometimes be more encompassing than ground truth, reflecting the motion-based grouping of pixels.
  • The model demonstrates strong generalization capabilities on unseen classes from MSRC and iCoseg datasets, producing decent segmentation masks even in challenging scenarios.

Limitations and Future Work

  • Limitations:
    • The current model is primarily designed for scenarios with a single primary foreground object and a background.
  • The unsupervised training phase requires significant computational resources and large amounts of video data.
  • The paper notes that minimizing background gradients can sometimes lead to blurred or unreal backgrounds, as these are auxiliary tasks.
  • Future Work:
    • The authors suggest extending the model to handle more than two object instances (i.e., multiple foreground objects) to address unsupervised object detection.
  • Further exploration of applying the learned features to other computer vision tasks is implied.

Summary updated

Abstract

This paper proposes a method for discovering the primary objects in single images by learning from videos in a purely unsupervised manner—the learning process is based on videos, but the generated network is able to discover objects from a single input image. The rough idea is that an image typically consists of multiple object instances (like the foreground and background) that have spatial transformations across video frames and they can be sparsely represented. By exploring the sparsity representation of a video with a neural network, one may learn the features of each object instance without any labels, which can be used to discover, recognize, or distinguish object instances from a single image. In this paper, we consider a relatively simple scenario, where each image roughly consists of a foreground and a background. Our proposed method is based on encoder-decoder structures to sparsely represent the foreground, background, and segmentation mask, which further reconstruct the original images. We apply the feed-forward network trained from videos for object discovery in single images, which is different from the previous co-segmentation methods that require videos or collections of images as the input for inference. The experimental results on various object segmentation benchmarks demonstrate that the proposed method extracts primary objects accurately and robustly, which suggests that unsupervised image learning tasks can benefit from the sparsity of images and the inter-frame structure of videos.
BibTeX
@inproceedings{zhao2020unsupervised,
  author = {Dong Zhao and Baoqing Ding and Yulin Wu and Lei Chen and Hongchao Zhou},
  title = {{Unsupervised Learning from Videos for Object Discovery in Single Images}},
  booktitle = {Symmetry},
  year = {2020},
  doi = {10.3390/sym13010038},
}
Figure