Saliency-guided adaptive seeding for supervoxel segmentation

Ge Gao, Mikko Lauri, Jianwei Zhang, Simone Frintrop

TL;DR

This paper introduces Saliency-guided Adaptive Seeding (SSV), a novel method for 3D supervoxel segmentation that leverages visual saliency to guide the seeding process. By concentrating more supervoxels in salient, object-rich regions and fewer in less salient backgrounds, SSV significantly improves boundary adherence and reduces undersegmentation errors compared to traditional uniform seeding methods.

What Problem Does This Paper Address?

Accurate segmentation of 3D point clouds into supervoxels is crucial for many robotics and computer vision tasks, as it simplifies downstream processing by reducing the number of primitives. However, existing methods like Voxel Cloud Connectivity Segmentation (VCCS) often use a uniform seeding strategy, which can lead to supervoxels that either cut across object boundaries or are too small in less important areas, hindering overall segmentation quality. The challenge is to create supervoxels that are not only spatially connected but also respect object boundaries more effectively, especially in complex scenes with varying levels of visual importance.

Methods

The core innovation of SSV lies in its saliency-guided adaptive seeding approach. Instead of distributing seeds uniformly, SSV first generates a pixel-level saliency map from the input RGB-D data using a computational visual saliency model (specifically, VOCUS2). This saliency map is then segmented into K clusters using k-means. Each cluster is assigned a "saliency-specific seeding resolution," meaning regions with higher average saliency receive a denser distribution of seeds, resulting in smaller, more precise supervoxels. Conversely, less salient regions get sparser seeding and thus larger supervoxels. The VCCS algorithm is then applied independently to each cluster with its assigned seeding resolution, and the results are merged to form the final supervoxel segmentation. This approach can be seen as a generalization of VCCS, where setting K=1 effectively reverts to the original VCCS method.

Datasets

The authors evaluate their method on two publicly available RGB-D datasets:

  • NYUV2: Contains 1449 color images with depth information, collected using Microsoft Kinect v1, depicting indoor scenes. The test set of 399 images was used for evaluation.
  • SUNRGBD: A larger dataset with 10335 images of cluttered indoor scenes, collected using various sensors including Intel RealSense, Asus Xtion, and Microsoft Kinect. 400 randomly chosen images were used for evaluation.

Key Insights and Ideas

  • Saliency as a Prior for Segmentation: The paper demonstrates that visual saliency is a strong indicator of object presence and importance. By leveraging this prior, SSV can allocate segmentation resources more effectively, leading to better boundary preservation.
  • Adaptive Seeding for Efficiency: The adaptive seeding strategy allows SSV to achieve a better trade-off between segmentation quality and computational complexity. It generates detailed segmentations in critical areas without over-segmenting less important background regions.
  • Generalization of Existing Methods: SSV builds upon the robust VCCS algorithm, extending its capabilities by introducing an intelligent seeding mechanism. This makes it a more versatile tool for 3D scene understanding.

Key Results

  • Improved Boundary Recall (REC): SSV consistently achieves higher boundary recall than VCCS across both datasets, indicating better adherence to object boundaries. For instance, at similar superpixel counts, SSV shows a noticeable improvement.
  • Reduced Undersegmentation Error (UE): The method also demonstrates lower undersegmentation error, meaning fewer objects are incorrectly merged into single supervoxels.
  • Comparable Explained Variation (EV): While REC and UE show significant improvements, the explained variation (a measure of segmentation quality independent of ground truth) is comparable or slightly better for SSV.
  • Qualitative Improvements: Visual comparisons show SSV producing a higher density of supervoxels in salient areas (likely containing objects) and larger supervoxels in background regions. This is particularly evident in capturing fine details of objects, such as handles on a sink.
  • Computational Cost: SSV is computationally more expensive than VCCS due to the saliency computation and k-means clustering. For K=5 clusters, the average processing time was 2.2 seconds per image, compared to 0.5 seconds for VCCS. However, the authors note that the saliency computation can be parallelized.

Limitations and Future Work

  • Computational Overhead: The current implementation of SSV is slower than VCCS due to the added saliency processing and clustering steps.
  • Future Work: The authors plan to investigate other priors, such as edge information and geometric depth cues, to further guide the adaptive seeding process. They also aim to explore the applicability of their supervoxel method to tasks like object discovery and are preparing to release a software implementation.

Summary updated

Abstract

We propose a new saliency-guided method for generating supervoxels in 3D space. Rather than using an evenly distributed spatial seeding procedure, our method uses visual saliency to guide the process of supervoxel generation. This results in densely distributed, small, and precise supervoxels in salient regions which often contain objects, and larger supervoxels in less salient regions that often correspond to background. Our approach largely improves the quality of the resulting supervoxel segmentation in terms of boundary recall and under-segmentation error on publicly available benchmarks.
BibTeX
@inproceedings{gao2017saliency,
  author = {Ge Gao and Mikko Lauri and Jianwei Zhang and Simone Frintrop},
  title = {{Saliency-guided adaptive seeding for supervoxel segmentation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2017},
  doi = {10.1109/iros.2017.8206374},
}
Figure