LRGNet: Learnable Region Growing for Class-Agnostic Point Cloud Segmentation

Jingdao Chen, Zsolt Kira, Yong Kwon Cho

TL;DR

LRGNet introduces a "learnable" region-growing approach for 3D point cloud segmentation that moves away from hand-tuned heuristics. By training a deep neural network to iteratively add and remove points from a region, the method achieves state-of-the-art class-agnostic performance, successfully segmenting objects of any shape or size across both indoor and outdoor environments.

What Problem Does This Paper Address?

Segmenting 3D point clouds—grouping points into individual object instances—is a fundamental task for robotics, yet it remains challenging. Most current methods fall into two camps: class-specific models that only recognize predefined categories (like "chair" or "table") and traditional region-growing methods that use fixed, hand-tuned thresholds (like color or surface normal similarity).

Class-specific models fail when they encounter "clutter" or unknown object types. Traditional region-growing is brittle; if your threshold is slightly off, the algorithm might merge two separate objects (under-segmentation) or break one object into many pieces (over-segmentation). LRGNet aims to bridge this gap by creating a data-driven, class-agnostic segmenter that learns the "rules" of how objects are shaped without needing to know what those objects are.

Methods

The core innovation is treating region growing as a learnable, iterative "morphing" process.

  1. Iterative Growth: Starting from a single "seed" point, the system repeatedly applies a neural network to update the current region.
  2. Dual-Mask Prediction: Unlike traditional methods that only add points, LRGNet predicts two things at each step:
  3. An Add Mask: Which nearby neighbor points should be included?
  4. A Remove Mask: Which points currently in the region were added by mistake and should be removed? This "self-correction" mechanism makes the model significantly more robust to noise.
  5. Architecture: The authors use a PointNet-inspired encoder-decoder. It takes two inputs (the current "inlier" set and the "neighbor" set) and produces the two masks.
  6. Robust Training: To train the model, the authors simulate the region-growing process on labeled data. Crucially, they intentionally introduce "mistakes" (randomly adding/removing points) during training. This forces the network to learn how to recover from errors and converge toward a complete object.
  7. Local Search: At inference time, the authors use a "random restart" strategy. By running the process multiple times from the same seed with slight probabilistic variations, they can select the final segment that appears most "complete" (e.g., has the highest number of points).

Datasets

The model was evaluated on three major benchmarks:

  • S3DIS (Stanford Large-Scale 3D Indoor Spaces): Indoor office environments.
  • ScanNet: A large-scale dataset of 3D indoor mesh models.
  • Semantic KITTI: Used to test generalization in outdoor driving scenes, despite the model being trained primarily on indoor data.

Key Insights and Ideas

  • Learning the Process, Not the Result: Instead of trying to predict the whole object at once, LRGNet learns the dynamics of how an object grows. This allows it to handle objects of arbitrary scale that might not fit into standard fixed-size processing blocks.
  • The Power of Deletion: The "Remove Mask" is a standout feature. Most segmentation algorithms are additive; LRGNet’s ability to "change its mind" about a point allows it to navigate complex boundaries where objects touch.
  • Cross-Dataset Generalization: Because the model learns geometric "object-ness" rather than specific semantic features (like "legs" or "handles"), it generalizes remarkably well. A model trained on ScanNet (homes) performs better on S3DIS (offices) than many models specifically trained for those environments.

Key Results

LRGNet outperformed competing class-agnostic and class-specific methods (including PointNet++, JSIS3D, and 3D-BoNet) by 1% to 9% across six different evaluation metrics.

  • Generalization: When trained on ScanNet and tested on S3DIS, LRGNet achieved a mean Intersection-over-Union (mIOU) of 0.45, significantly higher than the 0.35 achieved by 3D-BoNet.
  • Qualitative Success: In outdoor Semantic KITTI tests, LRGNet was better at separating roads from sidewalks and identifying complete building segments compared to traditional FPFH-based methods, which often over-segmented large structures.

Limitations and Future Work

The primary trade-off for LRGNet’s accuracy is computational cost. Because it is an iterative process, it requires many passes through the neural network to segment a single scene. Table IV in the paper shows that while LRGNet is faster than some complex joint-task networks (like JSIS3D), it is significantly slower than one-shot methods like PointNet++.

The authors suggest that future work could involve parallelizing the region-growing process for different seeds or implementing an adaptive step size to allow the region to grow faster in clear areas and slow down near complex boundaries.

Summary updated

Abstract

3D point cloud segmentation is an important function that helps robots understand the layout of their surrounding environment and perform tasks such as grasping objects, avoiding obstacles, and finding landmarks. Current segmentation methods are mostly class-specific, many of which are tuned to work with specific object categories and may not be generalizable to different types of scenes. This research proposes a learnable region growing method for class-agnostic point cloud segmentation, specifically for the task of instance label prediction. The proposed method is able to segment any class of objects using a single deep neural network without any assumptions about their shapes and sizes. The deep neural network is trained to predict how to add or remove points from a point cloud region to morph it into incrementally more complete regions of an object instance. Segmentation results on the S3DIS and ScanNet datasets show that the proposed method outperforms competing methods by 1%-9% on 6 different evaluation metrics.
BibTeX
@article{chen2021lrgnet,
  author = {Jingdao Chen and Zsolt Kira and Yong Kwon Cho},
  title = {{LRGNet: Learnable Region Growing for Class-Agnostic Point Cloud Segmentation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2021},
  url = {https://openreview.net/forum?id=oM2pn6kFzy},
}

Loading related content...

Figure