Texture-Aware Superpixel Segmentation

Rémi Giraud, Vinh-Thong Ta, Nicolas Papadakis, Yannick Berthoumieu

Abstract

Most superpixel algorithms compute a trade-off between spatial and color features at the pixel level. Hence, they may need fine parameter tuning to balance the two measures, and highly fail to group pixels with similar local texture properties. In this paper, we address these issues with a new Texture-Aware SuperPixel (TASP) method. To accurately segment textured and smooth areas, TASP automatically adjusts its spatial constraint according to the local feature variance. Then, to ensure texture homogeneity within superpixels, a new pixel to superpixel patch-based distance is proposed. TASP outperforms the segmentation accuracy of the state-of-the-art methods on texture and also natural color image datasets.

Introduction: The Challenge of Texture

Superpixel algorithms are a fundamental pre-processing step in computer vision, grouping pixels into meaningful atomic regions to reduce computational complexity. Most state-of-the-art methods, such as the ubiquitous SLIC, rely on a trade-off between color similarity and spatial proximity. However, these methods face two major hurdles:

  1. Texture Sensitivity: Because they operate at the pixel level, high-frequency textures (like grass, fabric, or stripes) appear as "noise" to the algorithm. This causes superpixels to bleed across boundaries or shatter into irregular shapes.
  2. Manual Tuning: Most algorithms require a "regularity" parameter to be set globally. A setting that works for a smooth blue sky will likely fail on a detailed forest floor, forcing researchers to manually tune parameters for every new dataset.

Methodology

TASP introduces three key innovations to the standard iterative K-means clustering framework used in superpixel generation:

  • Local Adaptation of Regularity: Instead of a global regularity parameter, TASP calculates a local constraint for each superpixel. This value scales exponentially based on the feature variance within that specific region. In textured areas (high variance), the spatial constraint increases to maintain shape; in smooth areas, it relaxes to better adhere to subtle color boundaries.
  • Patch-based Distance Measure: To truly "see" texture, the algorithm moves beyond individual pixels. It compares a square patch around a pixel to the content of a candidate superpixel. This allows the algorithm to recognize that a pixel belongs to a region not because its color matches the average, but because its neighborhood matches the local pattern.
  • Texture Unicity Constraint: To prevent a superpixel from "grabbing" similar-looking textures from distant parts of the image, the authors introduce a spatial penalty during the patch search. This ensures that the patches used to justify a pixel's inclusion are physically close to the superpixel's center, promoting compact and coherent regions.

To keep the computational cost manageable, the authors utilize PatchMatch, a fast randomized algorithm, to find these similar patches across the image.

Experimental Results

The authors evaluated TASP against several state-of-the-art methods (SLIC, LSC, SNIC, etc.) using three distinct datasets: the Berkeley Segmentation Dataset (BSD) for natural images, and two new challenging synthetic datasets, mix-Stripes and mix-Brodatz, designed specifically to test segmentation on complex patterns.

Qualitative comparison of TASP against state-of-the-art superpixel methods on the synthetic mix-Stripes dataset.

Quantitative Performance

TASP consistently outperformed existing methods across all metrics, particularly in textured environments. On the synthetic mix-Stripes dataset, TASP achieved an Achievable Segmentation Accuracy (ASA) of 0.8706, significantly higher than SLIC (0.7256) and the previous best-performer ETPS (0.7769).

Quantitative comparison of TASP against state-of-the-art methods across three datasets using ASA and F-measure metrics.

Even on the standard BSD natural image set, TASP maintained a slight edge (ASA of 0.9503), proving that its texture-awareness doesn't come at the cost of performance on "normal" images. Visual results show TASP successfully capturing the boundaries of striped objects that other algorithms completely ignored or "leaked" through.

Conclusion

TASP demonstrates that texture is a neighborhood property that cannot be defined at a single pixel. By using patches and making the algorithm self-adjusting based on local variance, TASP achieves superior results on natural images and synthetic textures simultaneously using the exact same internal settings. While the use of PatchMatch involves more computational overhead than simple pixel-wise comparisons, the robustness gained makes it a powerful tool for complex image segmentation tasks.

Summary updated

Abstract

Most superpixel algorithms compute a trade-off between spatial and color features at the pixel level. Hence, they may need fine parameter tuning to balance the two measures, and highly fail to group pixels with similar local texture properties. In this paper, we address these issues with a new Texture-Aware SuperPixel (TASP) method. To accurately segment textured and smooth areas, TASP automatically adjusts its spatial constraint according to the local feature variance. Then, to ensure texture homogeneity within superpixels, a new pixel to superpixel patch-based distance is proposed. TASP outperforms the segmentation accuracy of the state-of-the-art methods on texture and also natural color image datasets.
BibTeX
@inproceedings{giraud2019texture,
  author = {Rémi Giraud and Vinh-Thong Ta and Nicolas Papadakis and Yannick Berthoumieu},
  title = {{Texture-Aware Superpixel Segmentation}},
  booktitle = {ICIP},
  year = {2019},
  url = {https://openreview.net/forum?id=4jO3uV4Qzl},
}
Figure