Memory‐color segmentation and classification using class‐specific eigenregions

Clément Fredembach, Francisco Estrada, Sabine SüsstrunkJournal of the Society for Information Display 2009 · DOI

TL;DR

This paper introduces "class-specific eigenregions" – geometric features derived from human-segmented memory color regions (like sky, vegetation, skin) – to better evaluate and improve image segmentation algorithms. They demonstrate that these tailored features significantly outperform general eigenregions for classifying these important image regions.

What Problem Does This Paper Address?

Accurately segmenting images into meaningful regions is crucial for many computer vision tasks, especially those involving color correction or rendering. Certain image regions, termed "memory colors" (e.g., blue sky, green vegetation, skin tones), are particularly important because human observers perceive their colors consistently and expect them to be reproduced faithfully. However, existing segmentation evaluation methods often treat all regions equally, failing to account for the varying importance and distinct characteristics of these memory color regions. Furthermore, it's unclear if a single segmentation algorithm is optimal for all types of memory color regions, given their diverse shapes, sizes, and textures. This paper aims to address this by developing a class-specific approach to segmentation evaluation and feature extraction.

Methods

The core of the paper's methodology lies in the concept of eigenregions, which are PCA-based geometrical features capturing shape, size, and position information of image regions. The novelty here is the extension to class-specific eigenregions.

  1. Human Segmentation: A large dataset of real-world images was manually segmented by experts for specific memory color classes (blue sky, green vegetation, skin tones).
  2. Region Representation: Each segmented region was represented as a binary mask, which was then downsampled to a small grid (e.g., 6x8 pixels). This downsampled grid captures the region's coverage.
  3. PCA for Eigenregions: Principal Component Analysis (PCA) was applied to these downsampled region representations. The eigenvectors (eigenregions) and eigenvalues from this PCA form the geometric features. Eigenvalues indicate the importance of each eigenvector.
  4. Class-Specific vs. General Eigenregions: The authors calculated eigenregions for each specific memory color class (class-specific) and also for all regions pooled together (general).
  5. Segmentation Evaluation Metric: A novel distance measure is proposed to quantify the difference between the eigenregions of a human segmentation (ground truth) and those produced by an automatic segmentation algorithm. This measure uses the angle between corresponding eigenvectors, weighted by the eigenvalues from the human segmentation, to prioritize errors in more important orientations.
  6. Classification: These eigenregions (both general and class-specific) were then used as features for a multivariate Gaussian classifier to identify memory color regions.

Datasets

  • Database: A large database of 55,000 real-world images was used.
  • Annotated Subset: From this, 9,000 images were manually annotated by photographic experts for containing blue sky, green vegetation, or skin tones.
  • Experimental Subset: For experiments, 900 images (300 per class) were randomly selected from the annotated subset.
  • Ground Truth: Manual segmentations of these 900 images were created for the specific classes.
  • Automatic Segmentations: Four segmentation algorithms were applied: Meanshift, Felzenszwalb and Huttenlocher, k-means, and edgeflow.

Key Insights and Ideas

  • Memory Colors are Not Uniform: The paper strongly emphasizes that memory color regions are not all alike. Their geometric properties (shape, size, position) vary significantly, making a one-size-fits-all segmentation approach suboptimal.
  • Class-Specific Features Matter: Tailoring geometric features (eigenregions) to specific classes (sky, vegetation, skin) leads to a much better representation of their underlying structure compared to general features.
  • Segmentation Quality is Class-Dependent: Different segmentation algorithms perform better on different memory color classes. There isn't a single "best" algorithm for all tasks.
  • Eigenregions as a Diagnostic Tool: The proposed eigenregions and the distance metric can not only evaluate segmentation accuracy but also reveal why an algorithm performs well or poorly for a specific class (e.g., over-segmentation, under-segmentation, sensitivity to certain features).
  • Localization is Key: The degree to which a class is "localized" (consistently found in similar positions/shapes) directly impacts how well PCA-based features can represent it and how easily it can be segmented. Sky and vegetation are more localized than skin tones.

Key Results

  • Improved Classification: Using class-specific eigenregions resulted in a noticeable improvement in classification rates across all memory color classes compared to using general eigenregions. For example, classification rates for skin tones increased from 0.89 to 0.95.
  • Segmentation Performance Varies: Table 1 shows significant differences in the distance metric (lower is better) between automatic segmentations and human ground truth for each class and algorithm. For instance, k-means had a distance of 24.3 for sky but 44.15 for skin tones.
  • Meanshift as Overall Best (with caveats): Meanshift generally performed best overall, particularly for vegetation and skin tones, but surprisingly had a higher distance for sky compared to other algorithms.
  • FH Algorithm Struggles: The FH algorithm consistently performed poorly across all classes, attributed to chronic over-segmentation.
  • Dataset Size Matters: The experiments on dataset size suggest that around 200 images are needed to form a representative subset for a given class to achieve a distance close to that obtained with 300 images. Using fewer than 200 images introduces non-negligible approximations.

Limitations and Future Work

  • Parameter Sensitivity: The authors acknowledge that the performance of segmentation algorithms is highly dependent on their parameters, and further optimization for each method and class could yield even better results.
  • Computational Cost: While feature extraction and classification can be fast, image segmentation itself can be time-consuming, presenting a trade-off between accuracy and speed.
  • Future Work: The authors suggest that their distance measure can be used to optimize algorithm parameters for specific classes and to select representative subsets of data more efficiently. They also imply that the framework could be extended to other types of image regions beyond memory colors.

Summary updated

Abstract

Memory colors refer to the color of specific image regions that have the essential attribute of being perceived in a consistent manner by human observers. In color correction — or rendering — tasks, this consistency implies that they have to be faithfully reproduced; their importance, in that respect, is greater than that for other regions in an image. There are various schemes and attributes to detect memory colors, but the preferred method remains to segment the images into meaningful regions, a task for which many algorithms exist. Memory‐color regions are not, however, similar in their attributes. Significant variations in shape, size, and texture exist. As such, it is unclear whether a single segmentation algorithm is the most adapted for all of these classes. By using a large database of real‐world images, class‐specific geometrical features, eigenregions, were calculated. They can be used to evaluate how well an algorithm is adapted to segment a given class. A measure of localization of memory colors is given. The performance of class‐specific eigenregions were compared to general ones in the task of memory‐color‐region classification and it was observed that they provide a noticeable improvement in classification rates.
BibTeX
@article{fredembach2009memorycolor,
  author = {Clément Fredembach and Francisco Estrada and Sabine Süsstrunk},
  title = {{Memory‐color segmentation and classification using class‐specific eigenregions}},
  journal = {Journal of the Society for Information Display},
  year = {2009},
  doi = {10.1889/jsid17.11.921},
}

Loading related content...

Figure