Top-Down Beats Bottom-Up in 3D Instance Segmentation

Maksim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, Danila Rukhovich

3D instance segmentation—the task of identifying and segmenting individual objects within a 3D point cloud—is a cornerstone of spatial AI, with applications ranging from robotics to augmented reality. Historically, this field has been dominated by "bottom-up" approaches that group individual points into clusters. However, a new research paper from Samsung Research, titled "Top-Down Beats Bottom-Up in 3D Instance Segmentation," introduces a significant paradigm shift.

The authors present TD3D, a fully convolutional, cluster-free approach that demonstrates the superiority of the top-down paradigm in both accuracy and computational efficiency.

The Limitations of Bottom-Up Grouping

Most existing 3D instance segmentation methods rely on a bottom-up strategy. These models typically learn per-point embeddings and then employ complex geometric heuristics to group points into object instances. While functional, this approach suffers from several drawbacks:

  1. Computational Complexity: Point-wise clustering is often a resource-intensive post-processing step that is difficult to optimize for real-time applications.
  2. Hyperparameter Sensitivity: Grouping algorithms often require manual tuning of parameters (e.g., distance thresholds) that vary significantly across different datasets and object scales.
  3. Scale Variance: Bottom-up methods frequently struggle to handle scenes containing objects of vastly different sizes simultaneously.

By contrast, 2D computer vision transitioned years ago to top-down methods (like Mask R-CNN) which detect object proxies first. The 3D community has been slow to follow, largely due to the challenges of applying grid-based operations to sparse, unstructured 3D data.

The TD3D Framework: Detect-then-Refine

TD3D addresses these challenges by implementing a streamlined "detect-then-refine" pipeline using 3D sparse convolutions. This architecture eliminates the need for complex clustering entirely.

Comparison between traditional bottom-up grouping methods and the proposed top-down TD3D method.

The TD3D pipeline consists of two primary stages:

  1. Proposal Generation via 3D Object Detection: The model utilizes an anchor-free 3D detector (specifically FCAF3D) to predict 3D bounding boxes, semantic labels, and confidence scores directly from the point cloud. These boxes serve as initial object proposals.
  2. Proposal Refinement: For each predicted box, an RoI (Region of Interest) Extractor identifies the relevant voxels. These are then processed by a lightweight "3D tiny U-Net" that performs binary segmentation to distinguish the foreground object from the background within the box.

Because the entire system is end-to-end and cluster-free, it avoids the pitfalls of manual hyperparameter tuning and achieves a cleaner, more efficient computational graph.

Experimental Validation and Results

The researchers evaluated TD3D across four major benchmarks: ScanNet v2, ScanNet200, S3DIS, and the outdoor aerial dataset STPLS3D. The results consistently showed that TD3D outperforms established bottom-up methods.

Performance (AP50) vs. Latency (ms) on the ScanNet v2 validation set, showing TD3D's superior efficiency and accuracy compared to other models.

Key performance highlights include:

  • State-of-the-Art Accuracy: On ScanNet v2, TD3D achieved 48.9 AP on the test set, surpassing previous leaders like SoftGroup. On the validation set, it reached 47.3 AP compared to SoftGroup's 45.8 AP.
  • Superior Speed: The flagship TD3D model is 1.9x faster than its most accurate competitors. An optimized "fast" version achieves state-of-the-art accuracy with a latency of only ~105ms—a 2.6x speedup over previous methods.
  • Robust Generalization: TD3D showed remarkable performance on the STPLS3D outdoor dataset, improving the AP score by a massive +8.1 points over SoftGroup, demonstrating its ability to handle diverse environments without domain-specific tuning.

Conclusion

TD3D proves that the top-down paradigm is not only viable for 3D instance segmentation but is actually superior to traditional bottom-up grouping. By leveraging sparse convolutions and a simple detection-based pipeline, TD3D offers a faster, more accurate, and more generalizable solution. This work invites the 3D vision community to move away from complex clustering heuristics toward more streamlined, detection-driven architectures.

Summary updated

Abstract

Most 3D instance segmentation methods exploit a bottom-up strategy, typically including resource-exhaustive post-processing. For point grouping, bottom-up methods rely on prior assumptions about the objects in the form of hyperparameters, which are domain-specific and need to be carefully tuned. On the contrary, we address 3D instance segmentation with a TD3D: the pioneering cluster-free, fully-convolutional and entirely data-driven approach trained in an end-to-end manner. This is the first top-down method outperforming bottom-up approaches in 3D domain. With its straightforward pipeline, it demonstrates outstanding accuracy and generalization ability on the standard indoor benchmarks: ScanNet v2, its extension ScanNet200, and S3DIS, as well as on the aerial STPLS3D dataset. Besides, our method is much faster on inference than the current state-of-the-art grouping-based approaches: our flagship modification is 1.9x faster than the most accurate bottom-up method, while being more accurate, and our faster modification shows state-of-the-art accuracy running at 2.6x speed. Code is available at https://github.com/SamsungLabs/td3d.
BibTeX
@inproceedings{kolodiazhnyi2024top,
  author = {Maksim Kolodiazhnyi and Anna Vorontsova and Anton Konushin and Danila Rukhovich},
  title = {{Top-Down Beats Bottom-Up in 3D Instance Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00353},
}
Figure