LRGNet: Learnable Region Growing for Class-Agnostic Point Cloud Segmentation

Jingdao Chen, Zsolt Kira, Yong Kwon Cho

Abstract

3D point cloud segmentation is an important function that helps robots understand the layout of their surrounding environment and perform tasks such as grasping objects, avoiding obstacles, and finding landmarks. Current segmentation methods are mostly class-specific, many of which are tuned to work with specific object categories and may not be generalizable to different types of scenes. This research proposes a learnable region growing method for class-agnostic point cloud segmentation, specifically for the task of instance label prediction. The proposed method is able to segment any class of objects using a single deep neural network without any assumptions about their shapes and sizes. The deep neural network is trained to predict how to add or remove points from a point cloud region to morph it into incrementally more complete regions of an object instance. Segmentation results on the S3DIS and ScanNet datasets show that the proposed method outperforms competing methods by 1%-9% on 6 different evaluation metrics.

Introduction

Segmenting 3D point clouds—the process of grouping individual points into distinct object instances—is a foundational challenge in robotics and computer vision. Existing approaches typically fall into two categories: class-specific models and traditional region-growing heuristics. Class-specific models are often limited to predefined categories like "chair" or "table," failing when they encounter clutter or novel object types. Conversely, traditional region-growing methods rely on hand-tuned thresholds for features like color or surface normal similarity. These are often brittle; a slight threshold error can lead to under-segmentation (merging distinct objects) or over-segmentation (fragmenting a single object).

LRGNet addresses these limitations by introducing a data-driven, class-agnostic segmenter. Instead of relying on fixed heuristics, LRGNet learns the geometric "rules" of object composition, allowing it to segment objects of arbitrary shapes and sizes without prior semantic knowledge.

Methodology

The core innovation of LRGNet is treating region growing as a learnable, iterative "morphing" process. Starting from a single seed point, the system repeatedly applies a deep neural network to update and refine the current region.

Architecture and Dual-Mask Prediction

LRGNet utilizes a PointNet-inspired encoder-decoder architecture. The model takes two primary inputs: the current "inlier" set (points already in the region) and a "neighbor" set (nearby points being considered for inclusion). Unlike traditional additive methods, LRGNet predicts two distinct masks at each step:

  • Add Mask: Identifies which neighbor points should be incorporated into the region.
  • Remove Mask: Identifies points currently in the region that were added erroneously and should be removed.

This self-correction mechanism allows the model to recover from noise and navigate complex boundaries where objects are in close contact.

The LRGNet architecture uses an encoder-decoder framework to predict Add and Remove masks for iterative region growing.

Iterative Growth and Robust Training

To train the network, the authors simulate the region-growing process on labeled data, intentionally introducing "mistakes" (randomly adding or removing points). This forces the network to learn recovery strategies, ensuring the process converges toward a complete object instance. During inference, a "random restart" strategy is employed: the process is run multiple times from the same seed with slight probabilistic variations, and the most "complete" final segment is selected.

Step-by-step visualization of LRGNet's iterative process, demonstrating how the model adds and removes points to refine object instances over time.

Experimental Results

LRGNet was evaluated on three major benchmarks: S3DIS (indoor offices), ScanNet (indoor meshes), and Semantic KITTI (outdoor driving scenes).

Performance and Generalization

The model outperformed competing class-agnostic and class-specific methods—including PointNet++, JSIS3D, and 3D-BoNet—by 1% to 9% across six evaluation metrics. A key strength of LRGNet is its cross-dataset generalization. When trained on ScanNet and tested on S3DIS, LRGNet achieved a mean Intersection-over-Union (mIOU) of 0.45, significantly higher than the 0.35 achieved by 3D-BoNet. This suggests that the model successfully learns geometric "object-ness" rather than specific semantic features.

Qualitative Success

In outdoor tests on Semantic KITTI, LRGNet demonstrated superior performance in separating roads from sidewalks and identifying large, complete building segments. Traditional methods often over-segmented these large structures due to their reliance on local geometric consistency.

Discussion and Conclusion

LRGNet represents a significant step toward robust, general-purpose 3D segmentation. By learning the dynamics of object growth rather than predicting static masks, it handles objects of arbitrary scales and recovers from errors via its unique dual-mask system.

The primary trade-off for this accuracy is computational overhead. As an iterative process, LRGNet requires multiple passes through the network per scene, making it slower than one-shot methods like PointNet++. Future research will focus on parallelizing the process for different seeds and implementing adaptive step sizes to improve efficiency in less complex areas of a scene.

Summary updated

Abstract

3D point cloud segmentation is an important function that helps robots understand the layout of their surrounding environment and perform tasks such as grasping objects, avoiding obstacles, and finding landmarks. Current segmentation methods are mostly class-specific, many of which are tuned to work with specific object categories and may not be generalizable to different types of scenes. This research proposes a learnable region growing method for class-agnostic point cloud segmentation, specifically for the task of instance label prediction. The proposed method is able to segment any class of objects using a single deep neural network without any assumptions about their shapes and sizes. The deep neural network is trained to predict how to add or remove points from a point cloud region to morph it into incrementally more complete regions of an object instance. Segmentation results on the S3DIS and ScanNet datasets show that the proposed method outperforms competing methods by 1%-9% on 6 different evaluation metrics.
BibTeX
@article{chen2021lrgnet,
  author = {Jingdao Chen and Zsolt Kira and Yong Kwon Cho},
  title = {{LRGNet: Learnable Region Growing for Class-Agnostic Point Cloud Segmentation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2021},
  url = {https://openreview.net/forum?id=oM2pn6kFzy},
}
Figure