Abstract
The first main contribution of this paper is a novel method for representing images based on a dictionary of shape epitomes. These shape epitomes represent the local edge structure of the image and include hidden variables to encode shift and rotations. They are learnt in an unsupervised manner from ground truth edges. This dictionary is compact but is also able to capture the typical shapes of edges in natural images. In this paper, we illustrate the shape epitomes by applying them to the image labeling task. We apply shape epitomes to image labeling by using Conditional Random Field (CRF) Models. They are alternatives to the super pixel or pixel representations used in most CRFs. In our approach, the shape of an image patch is encoded by a shape epitome from the dictionary. Unlike the super pixel representation, our method avoids making early decisions which cannot be reversed. Our resulting hierarchical CRFs efficiently capture both local and global class co-occurrence properties. We demonstrate its quantitative and qualitative properties of our approach with image labeling experiments on two standard datasets: MSRC-21 and Stanford Background.
Introduction
Image labeling—the task of assigning a semantic category (like "cow," "sky," or "grass") to every pixel—is a cornerstone of computer vision. Most systems rely on either pixels or superpixels. Pixels are computationally expensive and ignore local context. Superpixels (groups of similar pixels) are more efficient but suffer from "early commitment": if the unsupervised algorithm that creates the superpixel makes a mistake and crosses an object boundary, the labeling system can almost never recover.
A third alternative is segmentation templates, which provide a prior model for edge structures. However, previous template-based methods were hand-crafted and rigid. To cover all possible shifts and rotations of an edge, thousands of static templates would be required. This paper introduces "shape epitomes," a compact, unsupervised dictionary of mid-level edge structures designed to represent local image geometry with shift and rotation invariance.
Methodology
The proposed "Shape epitome CRF" (SeCRF) framework relies on two main innovations:
1. Learning Shape Epitomes
Instead of hand-coding templates, the authors extract edge patches from the BSDS500 dataset and use affinity propagation to cluster them into a dictionary. Crucially, these "epitomes" include hidden variables for shift and rotation. A single epitome acts as a "master" that can generate hundreds of specific templates by shifting or rotating its internal structure. This approach reduced the error in representing ground-truth labels from 2% (using previous hand-crafted methods) to just 0.27%.
2. Hierarchical CRF Models
To integrate these epitomes into a labeling framework, the authors propose three levels of model complexity:
- Model 1 (One-level): A flat graph of overlapping local nodes where each node represents a patch encoded by a shape epitome.
- Model 2 (Two-level): Adds global nodes to enforce image-level consistency (e.g., if a "boat" is detected globally, local "water" labels become more likely).
- Model 3 (Three-level): Adds pairwise co-occurrence between global nodes and an auxiliary node to encourage sparsity, ensuring only a few object classes are active in any single image.
Key Insights and Results
One of the most striking findings is that a dictionary of only five shape epitomes is sufficient to capture the majority of mid-level shape structures in natural images. Furthermore, the dictionary learned on BSDS500 worked "out of the box" for both MSRC and Stanford Background datasets, suggesting that these shape epitomes capture fundamental geometric primitives.
- MSRC-21: The full Model 3 achieved a global accuracy of 81.7%, outperforming previous state-of-the-art template methods.
- Stanford Background: The model achieved 77.4% accuracy, performing better than more complex dynamic superpixel models.
Visual results demonstrate that shape epitomes are particularly effective at "snapping" to object boundaries, such as the legs of a cow or the edges of a building, compared to standard pixel-wise classifiers.
Conclusion
Shape epitomes offer a powerful, compact alternative to traditional image representations. By explicitly encoding invariance and utilizing hierarchical CRFs for global context, the method achieves high accuracy while avoiding the pitfalls of early commitment inherent in superpixel methods. Future work may involve integrating these shape priors into modern deep learning architectures to further enhance boundary sensitivity.
Summary updated