Abstract
Cosegmentation refers to the problem of segmenting multiple images simultaneously by exploiting the similarities between the foreground and background regions. The key issue in cosegmentation is to align common objects between these images. To address this issue, we propose an unsupervised learning framework for cosegmentation by coupling it with what we call "cosketching." The goal of cosketching is to automatically discover a codebook of deformable shape templates shared by the input images. These shape templates capture distinct image patterns and help align foreground objects, providing crucial information for cosegmentation. We present a statistical model whose energy function couples cosketch and cosegmentation, solved via an unsupervised learning algorithm. Experiments show that our method outperforms state-of-the-art methods on the challenging MSRC and iCoseg datasets. We also illustrate our method on a new dataset called Coseg-Rep where cosegmentation can be performed within a single image with repetitive patterns.
Introduction
Cosegmentation is the task of segmenting the same object type across multiple different images. The central challenge is alignment: how do you know which pixels in Image A correspond to the foreground object in Image B when the objects appear in different positions, scales, and poses?
Existing methods typically relied on local features like SIFT or color histograms. However, these features are often too "local"—they lack the global structural information needed to provide a strong prior for what an object actually looks like. This paper addresses the gap by introducing "cosketching," which discovers shared, deformable shape templates to provide the structural "glue" that local features lack.
Methodology: Coupling Sketch and Segmentation
The proposed framework uses a joint statistical model that minimizes an energy function consisting of three main components:
- Sketch Model: Uses an Active Basis Model to represent objects as a codebook of deformable shape templates (e.g., the head or body of a deer). These templates are composed of Gabor wavelets that can slightly shift and rotate to account for deformations.
- Region Model: A Markov Random Field (MRF) that models "non-sketchable" patterns like color distributions (using Gaussian Mixture Models) and spatial consistency between neighboring pixels.
- Coupling: This is the core innovation. Each shape template is associated with a segmentation template (a probability map). The sketch identifies where an object is, and the segmentation template provides a top-down prior for which pixels belong to the foreground.
The learning process is entirely unsupervised and iterative. It alternates between Image Parsing (using the current templates to segment images via Graph Cuts) and Re-learning (updating the shape and segmentation templates based on the latest segmentation results). Remarkably, the model starts with random initializations and no pre-processing.
Key Insights
- Mutual Supervision: Sketching and segmenting are mutually beneficial. Discovered shapes provide the "top-down" structural cues to guide segmentation, while the resulting segmentations provide "bottom-up" data to refine the shape templates.
- Beyond Local Features: By using 100x100 deformable templates rather than point-based features, the model captures much larger, more distinctive patterns, making it more robust to cluttered backgrounds.
- Intra-image Cosegmentation: The framework is flexible enough to handle "repetitive" patterns. If an image contains many instances of the same leaf, the model treats these as a "collection" and cosegments them within the single frame.
Experiments and Results
The authors evaluated their approach on three benchmarks:
- MSRC: A standard dataset where they tested 14 main object categories.
- iCoseg: A dataset featuring 38 categories with varying backgrounds and lighting.
- Coseg-Rep: A new dataset introduced in this paper containing 23 categories and 572 images, specifically including "repetitive" categories where cosegmentation is performed within a single image containing repeating patterns.
Quantitative Performance
On the MSRC dataset, the method achieved an average accuracy of 63.0% (Intersection-over-Union), significantly outperforming the previous state-of-the-art (Joulin et al.) which sat at 50.2%. It achieved the best performance in 13 out of 14 categories.
On the iCoseg dataset, the model reached 89.5% accuracy, surpassing other unsupervised methods by a margin of 4.2% to 10.6%. On the Coseg-Rep dataset, the model achieved a mean accuracy of 67.4% (IoU) and 90.2% (pixel ratio), demonstrating its effectiveness on natural, repeating textures.
Conclusion
This paper presents a novel unsupervised framework that simultaneously learns to sketch and segment. By coupling these two tasks, the model uses discovered structural shapes as top-down priors to achieve state-of-the-art cosegmentation performance without requiring any manual annotations. While the templates currently have fixed constraints on size and basis functions, the mutual supervision between sketching and segmentation provides a powerful path forward for unsupervised object discovery in complex scenes.
Summary updated