Cosegmentation and Cosketch by Unsupervised Learning

Jifeng Dai, Ying Nian Wu, Jie Zhou, Song-Chun ZhuICCV 2013 · OpenReview

Abstract

Cosegmentation refers to the problem of segmenting multiple images simultaneously by exploiting the similarities between the foreground and background regions. The key issue in cosegmentation is to align common objects between these images. To address this issue, we propose an unsupervised learning framework for cosegmentation by coupling it with what we call "cosketching." The goal of cosketching is to automatically discover a codebook of deformable shape templates shared by the input images. These shape templates capture distinct image patterns and help align foreground objects, providing crucial information for cosegmentation. We present a statistical model whose energy function couples cosketch and cosegmentation, solved via an unsupervised learning algorithm. Experiments show that our method outperforms state-of-the-art methods on the challenging MSRC and iCoseg datasets. We also illustrate our method on a new dataset called Coseg-Rep where cosegmentation can be performed within a single image with repetitive patterns.

Introduction

Cosegmentation is the task of segmenting the same object type across multiple different images. The central challenge is alignment: how do you know which pixels in Image A correspond to the foreground object in Image B when the objects appear in different positions, scales, and poses?

Existing methods typically relied on local features like SIFT or color histograms. However, these features are often too "local"—they lack the global structural information needed to provide a strong prior for what an object actually looks like. This paper addresses the gap by introducing "cosketching," which discovers shared, deformable shape templates to provide the structural "glue" that local features lack.

Methodology: Coupling Sketch and Segmentation

The proposed framework uses a joint statistical model that minimizes an energy function consisting of three main components:

  1. Sketch Model: Uses an Active Basis Model to represent objects as a codebook of deformable shape templates (e.g., the head or body of a deer). These templates are composed of Gabor wavelets that can slightly shift and rotate to account for deformations.
  2. Region Model: A Markov Random Field (MRF) that models "non-sketchable" patterns like color distributions (using Gaussian Mixture Models) and spatial consistency between neighboring pixels.
  3. Coupling: This is the core innovation. Each shape template is associated with a segmentation template (a probability map). The sketch identifies where an object is, and the segmentation template provides a top-down prior for which pixels belong to the foreground.
Overview of the cosegmentation and cosketch framework, showing the learned model components and parsing results.

The learning process is entirely unsupervised and iterative. It alternates between Image Parsing (using the current templates to segment images via Graph Cuts) and Re-learning (updating the shape and segmentation templates based on the latest segmentation results). Remarkably, the model starts with random initializations and no pre-processing.

Key Insights

  • Mutual Supervision: Sketching and segmenting are mutually beneficial. Discovered shapes provide the "top-down" structural cues to guide segmentation, while the resulting segmentations provide "bottom-up" data to refine the shape templates.
  • Beyond Local Features: By using 100x100 deformable templates rather than point-based features, the model captures much larger, more distinctive patterns, making it more robust to cluttered backgrounds.
  • Intra-image Cosegmentation: The framework is flexible enough to handle "repetitive" patterns. If an image contains many instances of the same leaf, the model treats these as a "collection" and cosegments them within the single frame.

Experiments and Results

The authors evaluated their approach on three benchmarks:

  • MSRC: A standard dataset where they tested 14 main object categories.
  • iCoseg: A dataset featuring 38 categories with varying backgrounds and lighting.
  • Coseg-Rep: A new dataset introduced in this paper containing 23 categories and 572 images, specifically including "repetitive" categories where cosegmentation is performed within a single image containing repeating patterns.

Quantitative Performance

On the MSRC dataset, the method achieved an average accuracy of 63.0% (Intersection-over-Union), significantly outperforming the previous state-of-the-art (Joulin et al.) which sat at 50.2%. It achieved the best performance in 13 out of 14 categories.

Comparison of cosegmentation accuracy (IoU) on the MSRC dataset against state-of-the-art methods.

On the iCoseg dataset, the model reached 89.5% accuracy, surpassing other unsupervised methods by a margin of 4.2% to 10.6%. On the Coseg-Rep dataset, the model achieved a mean accuracy of 67.4% (IoU) and 90.2% (pixel ratio), demonstrating its effectiveness on natural, repeating textures.

Conclusion

This paper presents a novel unsupervised framework that simultaneously learns to sketch and segment. By coupling these two tasks, the model uses discovered structural shapes as top-down priors to achieve state-of-the-art cosegmentation performance without requiring any manual annotations. While the templates currently have fixed constraints on size and basis functions, the mutual supervision between sketching and segmentation provides a powerful path forward for unsupervised object discovery in complex scenes.

Summary updated

Abstract

Co segmentation refers to the problem of segmenting multiple images simultaneously by exploiting the similarities between the foreground and background regions in these images. The key issue in co segmentation is to align common objects between these images. To address this issue, we propose an unsupervised learning framework for co segmentation, by coupling co segmentation with what we call ``co sketch''. The goal of co sketch is to automatically discover a codebook of deformable shape templates shared by the input images. These shape templates capture distinct image patterns and each template is matched to similar image patches in different images. Thus the co sketch of the images helps to align foreground objects, thereby providing crucial information for co segmentation. We present a statistical model whose energy function couples co sketch and co segmentation. We then present an unsupervised learning algorithm that performs co sketch and co segmentation by energy minimization. Experiments show that our method outperforms state of the art methods for co segmentation on the challenging MSRC and iciest datasets. We also illustrate our method on a new dataset called Coseg-Rep where co segmentation can be performed within a single image with repetitive patterns.
BibTeX
@inproceedings{dai2013cosegmentation,
  author = {Jifeng Dai and Ying Nian Wu and Jie Zhou and Song-Chun Zhu},
  title = {{Cosegmentation and Cosketch by Unsupervised Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  url = {https://openreview.net/forum?id=BkZ-7Wzd-H},
}
Figure