Abstract
Most 3D instance segmentation methods exploit a bottom-up strategy, typically including resource-exhaustive post-processing. For point grouping, bottom-up methods rely on prior assumptions about the objects in the form of hyperparameters, which are domain-specific and need to be carefully tuned. On the contrary, we address 3D instance segmentation with TD3D: the pioneering cluster-free, fully-convolutional and entirely data-driven approach trained in an end-to-end manner. This is the first top-down method outperforming bottom-up approaches in the 3D domain. With its straightforward pipeline, it demonstrates outstanding accuracy and generalization ability on standard indoor benchmarks: ScanNet v2, its extension ScanNet200, and S3DIS, as well as on the aerial STPLS3D dataset. Besides, our method is much faster on inference than current state-of-the-art grouping-based approaches: our flagship modification is 1.9x faster than the most accurate bottom-up method, while being more accurate, and our faster modification shows state-of-the-art accuracy running at 2.6x speed.
Introduction
In 3D instance segmentation—the task of identifying and labeling individual objects in a point cloud—the "bottom-up" approach has long reigned supreme. These methods typically learn features for every single point and then use complex clustering algorithms to group them into objects. However, bottom-up methods face several critical challenges:
- Complexity: They rely on resource-heavy post-processing and hand-tuned hyperparameters (like distance thresholds for clustering) that often fail to generalize across different scenes or object scales.
- Efficiency: Clustering is frequently a CPU-bound bottleneck, significantly slowing down inference.
- The 2D-3D Gap: In 2D computer vision, "top-down" methods like Mask R-CNN are the standard. Adapting this to the sparse, unstructured nature of 3D point clouds has historically been difficult, yielding sub-par results until now.
TD3D (Top-Down 3D) sets out to prove that a clean, fully-convolutional top-down approach can outperform the bottom-up status quo.
Methodology
TD3D introduces a two-stage, cluster-free pipeline that shifts the paradigm from point grouping to object detection and refinement.
- Proposal Generation: Instead of grouping points, TD3D utilizes an anchor-free, fully-convolutional 3D object detector (FCAF3D) to predict 3D bounding boxes. These boxes serve as efficient "object proxies."
- RoI Extraction: For each predicted box, the model extracts relevant features from the 3D feature map. A simple RoI Extractor selects voxels whose centers fall within the bounding box.
- Proposal Refinement: These extracted features are passed into a "tiny" 3D U-Net. This lightweight network performs binary segmentation on the voxels to produce the final high-fidelity instance mask.
The entire system is implemented using 3D sparse convolutions and trained end-to-end, allowing it to handle large, sparse 3D scenes efficiently on the GPU.
Experimental Setup
The model was evaluated on four major benchmarks to test accuracy and generalization:
- ScanNet v2: The standard indoor 3D scene dataset (18 categories).
- ScanNet200: A challenging version of ScanNet with a "long-tail" distribution of 200 categories.
- S3DIS: An indoor dataset focusing on large-scale office spaces.
- STPLS3D: A large-scale aerial photogrammetry dataset (outdoor), demonstrating cross-domain robustness.
Key Insights
- Top-Down Viability: The dominance of bottom-up methods was largely due to a lack of efficient top-down architectures, not inherent data limitations.
- Scalable Simplicity: By removing complex clustering heuristics, TD3D handles objects of vastly different sizes—from small chairs to large buildings—without requiring fixed clustering radii.
- GPU-First Design: Avoiding CPU-heavy clustering allows the pipeline to remain entirely on the GPU, leading to massive speedups.
- Efficient Refinement: A lightweight U-Net is sufficient for high-quality mask prediction without the computational burden of a full-sized segmentation network for every proposal.
Results
TD3D sets new benchmarks for both speed and accuracy in 3D instance segmentation.
- Efficiency: The flagship TD3D is 1.9x faster than the previous state-of-the-art (SoftGroup), while a faster variant reaches a 2.6x speedup.
- Accuracy: On ScanNet v2, TD3D achieved 75.1 AP50 on the test set, outperforming all existing competitors.
- Long-Tail Performance: On ScanNet200, it showed significant gains in "common" and "tail" categories, with a +6.9 and +4.3 AP increase respectively.
- Generalization: On the outdoor STPLS3D dataset, TD3D outperformed the best competitor by +8.1 AP, proving its robustness across environments.
Conclusion
TD3D demonstrates that top-down methods are not only viable but superior for 3D instance segmentation. While the method is dependent on the quality of initial bounding box proposals, its simplicity and GPU-native design offer a significant leap in performance. Future work may explore more efficient backbones or transformer-based architectures within this top-down framework to further advance 3D scene understanding.
Summary updated