Developing a Semantic-Driven Hybrid Segmentation Method for Point Clouds of 3D Shapes

Xiaowen Yang, Xie Han, Qingde Li, Ligang He, Min Pang, Caiqin JiaIEEE Access 2020 · OpenReview

Abstract

With the rapid development of point cloud processing technologies and the availability of a wide range of 3D capturing devices, a geometric object from the real world can be directly represented digitally as a dense and fine point cloud. Decomposing a 3D shape represented in point cloud into meaningful parts has very important practical implications in the fields of computer graphics, virtual reality and mixed reality. In this paper, a semantic-driven automated hybrid segmentation method is proposed for 3D point cloud shapes. Our method consists of three stages: semantic clustering, variational merging, and region remerging. In the first stage, a new feature of point cloud, called Local Concave-Convex Histogram, is introduced to first extract saddle regions complying with the semantic boundary feature. All other types of regions are then aggregated according to this extracted feature. This stage often leads to multiple over-segmentation convex regions, which are then remerged by a variational method established based on the narrow-band theory. Finally, in order to recombine the regions with the approximate shapes, order relation is introduced to improve the weighting forms in calculating the conventional Shape Diameter Function. We have conducted extensive experiments with the Princeton Dataset. The results show that the proposed algorithm outperforms the state-of-the-art algorithms in this area. We have also applied the proposed algorithm to process the point cloud data acquired directly from the real 3D objects. It achieves excellent results too. These results demonstrate that the method proposed in this paper is effective and universal.

Introduction

3D point clouds—massive collections of $(x, y, z)$ coordinates captured by LiDAR or depth cameras—are the "raw material" of modern computer vision. However, turning a disorganized swarm of points into meaningful parts (like the handle of a cup or the leg of a chair) remains a significant challenge. Traditional methods often struggle with two extremes: they either require thousands of manually labeled examples (supervised learning) or they are unsupervised but highly sensitive to noise, leading to "over-segmentation"—breaking one object into dozens of tiny, meaningless fragments.

The core challenge is identifying semantic boundaries. In cognitive science, the "minima rule" suggests that humans perceive boundaries at negative minima of principal curvature—essentially, the "saddle" regions where a shape dips inward. Most algorithms struggle to identify these subtle transitions in the irregular, disordered data of a point cloud.

Proposed Methodology

The proposed method uses a "bottom-up" hierarchical pipeline consisting of three distinct stages:

1. Semantic Clustering (LCCH)

The researchers introduced the Local Concave-Convex Histogram (LCCH). This feature analyzes the relationship between a point and its neighbors to classify regions as planar, convex, or concave. Crucially, it identifies "saddle" regions first, treating them as the primary candidates for object boundaries. By looking for concave saddle regions first, the algorithm aligns its logic with how human brains naturally perceive parts.

2. Variational Merging

Initial clustering often leaves the shape over-segmented. To address this, the team uses a variational method based on narrow-band theory. They fit quadric surfaces (mathematical proxies) to the regions and merge adjacent ones if the "error" between the points and the fitted surface is low. This approach is significantly faster than global search methods because it only looks at the "narrow-band" around existing boundaries.

3. Region Remerging (Improved SDF)

Finally, to catch regions that look different but belong to the same part, they use an improved Shape Diameter Function (SDF). SDF measures the "local thickness" of an object. The authors added an "order relation analysis" to weight the thickness measurements, making the algorithm much more robust when dealing with non-cylindrical or complex organic shapes.

Experimental Results

The method was evaluated using the Princeton Segmentation Benchmark, a standard dataset containing 380 objects across 19 categories (e.g., humans, animals, furniture), and real-world scans acquired via KinectFusion.

The proposed algorithm achieved a lower (better) Rand Index (RI) across almost all categories in the Princeton dataset compared to classical and modern algorithms like K-means, Normalized Cuts, and SSF. It performed particularly well on complex shapes such as "Ant," "Octopus," and "Glasses."

Segmentation result of an ant model showing distinct parts identified by the algorithm.

In many cases, the algorithm's segmentation was nearly indistinguishable from the ground-truth segments provided by human participants. Furthermore, the hybrid approach maintained high performance on raw data captured from real-world 3D scanners, demonstrating robustness against noise.

Conclusion and Future Work

This semantic-driven hybrid method successfully combines geometric features for initial boundaries with thickness-based analysis for final merging. While powerful, the algorithm currently relies on fixed thresholds that can lead to over-segmentation in highly detailed organic shapes. Future work will focus on developing an adaptive parameter-setting scheme to automatically adjust thresholds based on the specific geometry of the object, further reducing the need for human intervention.

Reference: Yang, X., Han, X., Li, Q., He, L., Pang, M., & Jia, C. (2020). Developing a Semantic-Driven Hybrid Segmentation Method for Point Clouds of 3D Shapes. IEEE Access, 8, 40861-40879.

Summary updated

Abstract

With the rapid development of point cloud processing technologies and the availability of a wide range of 3D capturing devices, a geometric object from the real world can be directly represented digitally as a dense and fine point cloud. Decomposing a 3D shape represented in point cloud into meaningful parts has very important practical implications in the fields of computer graphics, virtual reality and mixed reality. In this paper, a semantic-driven automated hybrid segmentation method is proposed for 3D point cloud shapes. Our method consists of three stages: semantic clustering, variational merging, and region remerging. In the first stage, a new feature of point cloud, called Local Concave-Convex Histogram, is introduced to first extract saddle regions complying with the semantic boundary feature. All other types of regions are then aggregated according to this extracted feature. This stage often leads to multiple over-segmentation convex regions, which are then remerged by a variational method established based on the narrow-band theory. Finally, in order to recombine the regions with the approximate shapes, order relation is introduced to improve the weighting forms in calculating the conventional Shape Diameter Function. We have conducted extensive experiments with the Princeton Dataset. The results show that the proposed algorithm outperforms the state-of-the-art algorithms in this area. We have also applied the proposed algorithm to process the point cloud data acquired directly from the real 3D objects. It achieves excellent results too. These results demonstrate that the method proposed in this paper is effective and universal.
BibTeX
@inproceedings{yang2020developing,
  author = {Xiaowen Yang and Xie Han and Qingde Li and Ligang He and Min Pang and Caiqin Jia},
  title = {{Developing a Semantic-Driven Hybrid Segmentation Method for Point Clouds of 3D Shapes}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=1uGBti8TsY},
}
Figure