Abstract
3D point cloud segmentation is an important function that helps robots understand the layout of their surrounding environment and perform tasks such as grasping objects, avoiding obstacles, and finding landmarks. Current segmentation methods are mostly class-specific, many of which are tuned to work with specific object categories and may not be generalizable to different types of scenes. This research proposes a learnable region growing method for class-agnostic point cloud segmentation, specifically for the task of instance label prediction. The proposed method is able to segment any class of objects using a single deep neural network without any assumptions about their shapes and sizes. The deep neural network is trained to predict how to add or remove points from a point cloud region to morph it into incrementally more complete regions of an object instance. Segmentation results on the S3DIS and ScanNet datasets show that the proposed method outperforms competing methods by 1%-9% on 6 different evaluation metrics.
Introduction
Segmenting 3D point clouds—the process of grouping individual points into distinct object instances—is a foundational challenge in robotics and computer vision. Existing approaches typically fall into two categories: class-specific models and traditional region-growing heuristics. Class-specific models are often limited to predefined categories like "chair" or "table," failing when they encounter clutter or novel object types. Conversely, traditional region-growing methods rely on hand-tuned thresholds for features like color or surface normal similarity. These are often brittle; a slight threshold error can lead to under-segmentation (merging distinct objects) or over-segmentation (fragmenting a single object).
LRGNet addresses these limitations by introducing a data-driven, class-agnostic segmenter. Instead of relying on fixed heuristics, LRGNet learns the geometric "rules" of object composition, allowing it to segment objects of arbitrary shapes and sizes without prior semantic knowledge.
Methodology
The core innovation of LRGNet is treating region growing as a learnable, iterative "morphing" process. Starting from a single seed point, the system repeatedly applies a deep neural network to update and refine the current region.
Architecture and Dual-Mask Prediction
LRGNet utilizes a PointNet-inspired encoder-decoder architecture. The model takes two primary inputs: the current "inlier" set (points already in the region) and a "neighbor" set (nearby points being considered for inclusion). Unlike traditional additive methods, LRGNet predicts two distinct masks at each step:
- Add Mask: Identifies which neighbor points should be incorporated into the region.
- Remove Mask: Identifies points currently in the region that were added erroneously and should be removed.
This self-correction mechanism allows the model to recover from noise and navigate complex boundaries where objects are in close contact.
Iterative Growth and Robust Training
To train the network, the authors simulate the region-growing process on labeled data, intentionally introducing "mistakes" (randomly adding or removing points). This forces the network to learn recovery strategies, ensuring the process converges toward a complete object instance. During inference, a "random restart" strategy is employed: the process is run multiple times from the same seed with slight probabilistic variations, and the most "complete" final segment is selected.
Experimental Results
LRGNet was evaluated on three major benchmarks: S3DIS (indoor offices), ScanNet (indoor meshes), and Semantic KITTI (outdoor driving scenes).
Performance and Generalization
The model outperformed competing class-agnostic and class-specific methods—including PointNet++, JSIS3D, and 3D-BoNet—by 1% to 9% across six evaluation metrics. A key strength of LRGNet is its cross-dataset generalization. When trained on ScanNet and tested on S3DIS, LRGNet achieved a mean Intersection-over-Union (mIOU) of 0.45, significantly higher than the 0.35 achieved by 3D-BoNet. This suggests that the model successfully learns geometric "object-ness" rather than specific semantic features.
Qualitative Success
In outdoor tests on Semantic KITTI, LRGNet demonstrated superior performance in separating roads from sidewalks and identifying large, complete building segments. Traditional methods often over-segmented these large structures due to their reliance on local geometric consistency.
Discussion and Conclusion
LRGNet represents a significant step toward robust, general-purpose 3D segmentation. By learning the dynamics of object growth rather than predicting static masks, it handles objects of arbitrary scales and recovers from errors via its unique dual-mask system.
The primary trade-off for this accuracy is computational overhead. As an iterative process, LRGNet requires multiple passes through the network per scene, making it slower than one-shot methods like PointNet++. Future research will focus on parallelizing the process for different seeds and implementing adaptive step sizes to improve efficiency in less complex areas of a scene.
Summary updated