TL;DR
This paper introduces a novel approach to semantic instance labeling in indoor RGBD scenes by leveraging hierarchical superpixel segmentation. By grouping superpixels into more meaningful, object-like segments, their method achieves state-of-the-art results on the NYU Depth Dataset, outperforming previous pixel- or superpixel-based methods.
What Problem Does This Paper Address?
Accurate semantic understanding of indoor environments is crucial for robots to navigate and interact autonomously. While RGBD sensors have made dense 3D data more accessible, robustly labeling every pixel with its corresponding object category (semantic labeling) and distinguishing individual instances of objects remains a challenge. Existing methods often rely on pixel-level or superpixel-level features, which can be noisy and may not fully capture the characteristics of whole objects. This paper aims to bridge this gap by proposing a hierarchical segmentation approach that better aligns with object boundaries, leading to more accurate and instance-aware semantic labeling.
Methods
The core of the proposed method lies in a hierarchical segmentation strategy. First, they estimate surface normals from the RGBD data, which are then used in conjunction with color information to define a non-linear edge weight for graph-based segmentation. This process over-segments the scene into supervoxels. Crucially, these supervoxels are then hierarchically clustered into larger, more meaningful segments that better correspond to entire objects.
Feature vectors are constructed for each of these hierarchical segments using histograms of CIE Lab color, 3D position, and 3D surface normals (LABXYZNxNyNz histograms). These features, along with geometric properties like size, height, and width, are used to train a random forest classifier. The authors specifically chose a random forest due to the independence of their histogram-based features.
Datasets
The primary dataset used for training and evaluation is the NYU Depth Dataset V2. The authors evaluate their method on two tasks using this dataset:
- 4-class semantic labeling: classifying scenes into floor, structure, furniture, and props.
- 14-class semantic labeling: classifying into more specific object categories (e.g., bed, chair, table) plus an "other" category, as proposed by Couprie et al. [5].
Key Insights and Ideas
- Hierarchical segmentation is key: Grouping superpixels into larger, object-aligned segments provides richer, more discriminative features than using individual superpixels or pixels alone.
- Object-centric features matter: Features derived from whole objects (shape, size, color, surface normals) are more informative for semantic labeling than patch-based features.
- Random forests are effective: The independence and robustness of histogram-based features make random forests a suitable and high-performing classifier for this task.
- Surface normals enhance segmentation: Incorporating surface normals into the segmentation process improves the quality of the resulting segments.
Key Results
- State-of-the-art performance: The proposed method achieves state-of-the-art results on the NYU Depth Dataset for both 4-class and 14-class semantic labeling.
- Improved accuracy: For the 4-class task, their method (without SIFT features) achieved 69.6% class accuracy and 69.5% pixel accuracy, outperforming previous methods. With SIFT features, they achieved 68.2% class accuracy and 71.8% pixel accuracy.
- Strong performance on specific objects: In the 14-class task, they achieved 47.6% class accuracy, significantly outperforming unsupervised methods like Wang et al. [28] (42.2%) and Couprie et al. [5] (36.2%).
- Qualitative improvements: Visual results show that their method excels at recognizing planar objects like floors, ceilings, and furniture, and is superior at distinguishing chairs, tables, and windows due to their consistent shapes and surface normals.
Limitations and Future Work
- Object segmentation challenges: The hierarchical segmentation can sometimes merge small objects into larger segments, leading to misclassifications. The generic features might not generalize well to all object types, and the lack of textural information could also be a contributing factor.
- Future work suggestions: The authors suggest exploring GPU optimization for faster processing, incorporating structural constraints similar to other methods, and integrating their approach with SLAM systems for complete scene modeling. They also mention the potential benefit of combining their hand-crafted features with learned features.
Summary updated