Abstract
Joint segmentation of image sets has great importance for object recognition, image classification, and image retrieval. In this paper, we aim to jointly segment a set of images starting from a small number of labeled images or none at all. To allow the images to share segmentation information with each other, we build a network that contains segmented as well as unsegmented images, and extract functional maps between connected image pairs based on image appearance features. These functional maps act as general property transporters between the images and, in particular, are used to transfer segmentations. We define and operate in a reduced functional space optimized so that the functional maps approximately satisfy cycle-consistency under composition in the network. A joint optimization framework is proposed to simultaneously generate all segmentation functions over the images so that they both align with local segmentation cues in each particular image, and agree with each other under network transportation. This formulation allows us to extract segmentations even with no training data, but can also exploit such data when available. The collective effect of the joint processing using functional maps leads to accurate information sharing among images and yields superior segmentation results, as shown on the iCoseg, MSRC, and PASCAL data sets.
Introduction
Co-segmentation is the task of jointly segmenting common objects across a collection of images. This task is inherently challenging because objects of the same class (e.g., different cows or cars) can exhibit significant variations in appearance due to lighting, viewpoint, and intra-class diversity. Traditional methods often rely on point-based correspondences, such as SIFT matching, which frequently fail when visual appearance changes drastically. Furthermore, ensuring "global consistency" across a large image network—where mapping information from Image A to B to C and back to A remains unchanged—is computationally difficult. Without such consistency, errors in pairwise matching accumulate, leading to degraded segmentation performance.
Methodology
The proposed framework addresses these challenges by representing relationships between images as linear "functional maps" rather than point-to-point correspondences. The process is divided into three main stages:
1. Reduced Functional Space
Instead of raw pixels, images are decomposed into super-pixels. Segmentations are represented as functions over these regions. By using the first few eigenvectors of the Graph Laplacian of the super-pixel graph, the authors create a low-dimensional "functional space" that captures the essential geometric structure of the image.
As shown in the figure above, the reconstruction error decreases rapidly as the number of bases increases, justifying the use of a reduced space for efficiency and robustness.
2. Consistent Functional Maps
Linear maps ($X_{ij}$) are computed between these functional spaces. The optimization for these maps considers feature alignment (preserving descriptors like RGB and color histograms) and regularization (favoring similar spectral components). Crucially, the method enforces cycle-consistency by introducing a latent basis. Every pairwise map is constrained to be consistent with this shared latent space, providing a tractable way to ensure global network agreement.
3. Joint Segmentation
Finally, segmentation functions for all images are jointly optimized. These functions are encouraged to align with local image cues and maintain consistency when transported across the network via the functional maps.
Results and Discussion
The "FMaps" approach was evaluated on standard benchmarks including iCoseg, MSRC, and PASCAL VOC 2012.
- iCoseg: The method achieved an average accuracy of 90.5% in an unsupervised setting, significantly outperforming previous unsupervised state-of-the-art methods and rivaling supervised approaches.
- MSRC: Significant improvements were noted in difficult classes; for instance, "cow" segmentation accuracy reached 89.7%, compared to the previous best of 81.6%.
- PASCAL VOC: The framework demonstrated scalability to hundreds of images and robustness to the high variance typical of this dataset.
Conclusion
By shifting from point-based matching to functional maps and enforcing global cycle-consistency, this framework provides a robust solution for image co-segmentation. While the method relies on initial graph construction and currently focuses on single-class scenarios, its ability to aggregate information across diverse image sets marks a significant advancement in joint image analysis. Future work may explore multi-class extensions and further computational optimizations for web-scale applications.
Summary updated