Top-Down Beats Bottom-Up in 3D Instance Segmentation

Maksim Kolodiazhnyi, Danila Rukhovich, Anna Vorontsova, Anton Konushin

Abstract

Most 3D instance segmentation methods exploit a bottom-up strategy, typically including resource-exhaustive post-processing. For point grouping, bottom-up methods rely on prior assumptions about the objects in the form of hyperparameters, which are domain-specific and need to be carefully tuned. On the contrary, we address 3D instance segmentation with TD3D: the pioneering cluster-free, fully-convolutional and entirely data-driven approach trained in an end-to-end manner. This is the first top-down method outperforming bottom-up approaches in the 3D domain. With its straightforward pipeline, it demonstrates outstanding accuracy and generalization ability on standard indoor benchmarks: ScanNet v2, its extension ScanNet200, and S3DIS, as well as on the aerial STPLS3D dataset. Besides, our method is much faster on inference than current state-of-the-art grouping-based approaches: our flagship modification is 1.9x faster than the most accurate bottom-up method, while being more accurate, and our faster modification shows state-of-the-art accuracy running at 2.6x speed.

Introduction

In 3D instance segmentation—the task of identifying and labeling individual objects in a point cloud—the "bottom-up" approach has long reigned supreme. These methods typically learn features for every single point and then use complex clustering algorithms to group them into objects. However, bottom-up methods face several critical challenges:

  1. Complexity: They rely on resource-heavy post-processing and hand-tuned hyperparameters (like distance thresholds for clustering) that often fail to generalize across different scenes or object scales.
  2. Efficiency: Clustering is frequently a CPU-bound bottleneck, significantly slowing down inference.
  3. The 2D-3D Gap: In 2D computer vision, "top-down" methods like Mask R-CNN are the standard. Adapting this to the sparse, unstructured nature of 3D point clouds has historically been difficult, yielding sub-par results until now.

TD3D (Top-Down 3D) sets out to prove that a clean, fully-convolutional top-down approach can outperform the bottom-up status quo.

Methodology

TD3D introduces a two-stage, cluster-free pipeline that shifts the paradigm from point grouping to object detection and refinement.

Comparison of traditional grouping-based bottom-up methods versus the proposed top-down TD3D pipeline.
  1. Proposal Generation: Instead of grouping points, TD3D utilizes an anchor-free, fully-convolutional 3D object detector (FCAF3D) to predict 3D bounding boxes. These boxes serve as efficient "object proxies."
  2. RoI Extraction: For each predicted box, the model extracts relevant features from the 3D feature map. A simple RoI Extractor selects voxels whose centers fall within the bounding box.
  3. Proposal Refinement: These extracted features are passed into a "tiny" 3D U-Net. This lightweight network performs binary segmentation on the voxels to produce the final high-fidelity instance mask.

The entire system is implemented using 3D sparse convolutions and trained end-to-end, allowing it to handle large, sparse 3D scenes efficiently on the GPU.

Experimental Setup

The model was evaluated on four major benchmarks to test accuracy and generalization:

  • ScanNet v2: The standard indoor 3D scene dataset (18 categories).
  • ScanNet200: A challenging version of ScanNet with a "long-tail" distribution of 200 categories.
  • S3DIS: An indoor dataset focusing on large-scale office spaces.
  • STPLS3D: A large-scale aerial photogrammetry dataset (outdoor), demonstrating cross-domain robustness.

Key Insights

  • Top-Down Viability: The dominance of bottom-up methods was largely due to a lack of efficient top-down architectures, not inherent data limitations.
  • Scalable Simplicity: By removing complex clustering heuristics, TD3D handles objects of vastly different sizes—from small chairs to large buildings—without requiring fixed clustering radii.
  • GPU-First Design: Avoiding CPU-heavy clustering allows the pipeline to remain entirely on the GPU, leading to massive speedups.
  • Efficient Refinement: A lightweight U-Net is sufficient for high-quality mask prediction without the computational burden of a full-sized segmentation network for every proposal.

Results

TD3D sets new benchmarks for both speed and accuracy in 3D instance segmentation.

Performance (AP50) vs. Latency (ms) on ScanNet v2. TD3D achieves higher accuracy with significantly lower latency compared to state-of-the-art grouping-based methods.
  • Efficiency: The flagship TD3D is 1.9x faster than the previous state-of-the-art (SoftGroup), while a faster variant reaches a 2.6x speedup.
  • Accuracy: On ScanNet v2, TD3D achieved 75.1 AP50 on the test set, outperforming all existing competitors.
  • Long-Tail Performance: On ScanNet200, it showed significant gains in "common" and "tail" categories, with a +6.9 and +4.3 AP increase respectively.
  • Generalization: On the outdoor STPLS3D dataset, TD3D outperformed the best competitor by +8.1 AP, proving its robustness across environments.

Conclusion

TD3D demonstrates that top-down methods are not only viable but superior for 3D instance segmentation. While the method is dependent on the quality of initial bounding box proposals, its simplicity and GPU-native design offer a significant leap in performance. Future work may explore more efficient backbones or transformer-based architectures within this top-down framework to further advance 3D scene understanding.

Summary updated

Abstract

Most 3D instance segmentation methods exploit a bottom-up strategy, typically including resource-exhaustive post-processing. For point grouping, bottom-up methods rely on prior assumptions about the objects in the form of hyperparameters, which are domain-specific and need to be carefully tuned. On the contrary, we address 3D instance segmentation with a TD3D: the pioneering cluster-free, fully-convolutional and entirely data-driven approach trained in an end-to-end manner. This is the first top-down method outperforming bottom-up approaches in 3D domain. With its straightforward pipeline, it demonstrates outstanding accuracy and generalization ability on the standard indoor benchmarks: ScanNet v2, its extension ScanNet200, and S3DIS, as well as on the aerial STPLS3D dataset. Besides, our method is much faster on inference than the current state-of-the-art grouping-based approaches: our flagship modification is 1.9x faster than the most accurate bottom-up method, while being more accurate, and our faster modification shows state-of-the-art accuracy running at 2.6x speed. Code is available at https://github.com/SamsungLabs/td3d .
BibTeX
@inproceedings{kolodiazhnyi2024top,
  author = {Maksim Kolodiazhnyi and Danila Rukhovich and Anna Vorontsova and Anton Konushin},
  title = {{Top-Down Beats Bottom-Up in 3D Instance Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00353},
}
Figure