ICM-3D: Instantiated Category Modeling for 3D Instance Segmentation

Ruihang Chu, Yukang Chen, Tao Kong, Lu Qi, Lei Li

Abstract

Separating 3D point clouds into individual instances is an important task for 3D vision. It is challenging due to the unknown and varying number of instances in a scene. Existing deep learning based works focus on a two-step pipeline: first learn a feature embedding and then cluster the points. Such a two-step pipeline leads to disconnected intermediate objectives. In this paper, we propose an integrated reformulation of 3D instance segmentation as a per-point classification problem. We propose ICM-3D, a single-step method to segment 3D instances via instantiated categorization. The augmented category information is automatically constructed from 3D spatial positions. We conduct extensive experiments to verify the effectiveness of ICM-3D and show that it obtains inspiring performance across multiple frameworks, backbones and benchmarks.

Introduction

In 3D vision, instance segmentation requires identifying not just the semantic class of a point (e.g., "chair"), but also its specific identity (e.g., "chair #1"). Traditionally, this has been approached via an "embed-and-cluster" pipeline. In this paradigm, a network projects points into a high-dimensional feature space where points from the same object are pulled together and points from different objects are pushed apart. A heuristic clustering algorithm, such as Mean-Shift or Breadth-First Search (BFS), is then applied to group these points.

However, this two-step process suffers from two major flaws. First, it is inherently unstable; optimizing relative pairwise distances during training is chaotic, as moving one point in feature space affects its relationship with all others. Second, there is a significant gap between the training objective (distance optimization) and the inference mechanism (rule-based clustering), often requiring fragile, hand-tuned thresholds.

Comparison between traditional embedding learning (a) and the proposed instantiated category modeling (b).

Methodology: Instantiated Category Modeling

The proposed ICM-3D framework reformulates instance segmentation as a single-step, per-point classification problem. The core innovation is the use of spatial position as a fixed anchor for identity. Since instances in 3D space rarely overlap perfectly, their centroids provide a reliable basis for categorization.

The Spatial Grid

ICM-3D divides a 3D scene into a uniform grid of $N_s \times N_s \times N_s$ cubes. Each cube represents a potential "instance category." If an object's centroid falls within a specific cube, every point belonging to that object is assigned to that cube's category. This transforms the task into a standard classification problem using objectives like Dice Loss or Cross-Entropy.

Implementation Paradigms

To handle the memory requirements of a large 3D grid, the authors propose two paradigms:

  1. Flatten Paradigm: The network outputs a matrix where each channel corresponds to one of the $N_s^3$ cubes. While direct, this is memory-intensive.
  2. Project Paradigm: To optimize efficiency, the model predicts three independent projections along the X, Y, and Z axes ($3 \times N_s$ channels). A point is considered part of a specific instance only if it is simultaneously "activated" in the corresponding X, Y, and Z bins.
The spatial grid and projection paradigm used in ICM-3D to identify instances via X, Y, and Z axis projections.

Scoring and Inference

Because most spatial cubes are empty, a parallel Scoring Head predicts which cubes actually contain objects. During inference, the model selects high-scoring cubes, binarizes the resulting masks, and applies Non-Maximum Suppression (NMS) to remove redundant predictions, mirroring the workflow of successful 2D object detectors.

Experimental Evaluation

ICM-3D was evaluated across four major benchmarks, demonstrating significant improvements in both accuracy and efficiency.

  • S3DIS & ScanNet V2: On S3DIS, ICM-3D achieved a +2% gain in Mean Recall. On ScanNet V2, it reached 32.7 mAP, outperforming established methods like PointGroup (30.5 mAP) while maintaining a simpler architecture.
  • PartNet: The model excelled at fine-grained segmentation, showing a massive +6.8% mAP improvement on this challenging dataset.
  • Efficiency: By eliminating iterative clustering, the "Project" version of ICM-3D is up to 17x faster than previous state-of-the-art methods like ASIS.
  • Generalization: Testing on the OCID robotic dataset showed that ICM-3D generalizes well to unseen object shapes, as its reliance on spatial anchors is more robust than learning abstract shape features.

Discussion and Limitations

The primary advantage of ICM-3D is its ability to precisely define boundaries between touching objects, as they are naturally assigned to different spatial bins. However, the method faces challenges when two different instances have centroids that fall into the same spatial cube ("centroid collision"), which can lead to merged predictions. Additionally, the choice of grid granularity ($N_s$) involves a trade-off: a coarse grid risks merging objects, while an overly fine grid reduces the number of training examples per category.

Conclusion

ICM-3D simplifies 3D instance segmentation by anchoring object identity to physical space. By replacing complex clustering heuristics with a unified classification framework, it achieves superior performance and remarkable inference speeds, setting a new direction for efficient 3D scene understanding.

Summary updated

Abstract

Separating 3D point clouds into individual instances is an important task for 3D vision. It is challenging due to the unknown and varying number of instances in a scene. Existing deep learning based works focus on a two-step pipeline: first learn a feature embedding and then cluster the points. Such a two-step pipeline leads to disconnected intermediate objectives. In this paper, we propose an integrated reformulation of 3D instance segmentation as a per-point classification problem. We propose ICM-3D, a single-step method to segment 3D instances via instantiated categorization. The augmented category information is automatically constructed from 3D spatial positions. We conduct extensive experiments to verify the effectiveness of ICM-3D and show that it obtains inspiring performance across multiple frameworks, backbones and benchmarks.
BibTeX
@article{chu2022icm,
  author = {Ruihang Chu and Yukang Chen and Tao Kong and Lu Qi and Lei Li},
  title = {{ICM-3D: Instantiated Category Modeling for 3D Instance Segmentation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  doi = {10.1109/lra.2021.3108483},
}
Figure