Superpoint Transformer for 3D Scene Instance Segmentation

Jiahao Sun, Chunmei Qing, Junpeng Tan, Xiangmin XuAAAI 2023 · DOI

Introduction

3D instance segmentation—the task of identifying and masking individual objects in a 3D point cloud—is a fundamental challenge in computer vision. Existing approaches typically follow one of two paradigms: proposal-based (top-down) methods that fit bounding boxes, or grouping-based (bottom-up) methods that cluster point-wise labels. Both face significant hurdles: 3D boxes are difficult to fit accurately due to surface-only point distributions, while grouping methods often rely on complex, error-prone post-processing.

SPFormer (Superpoint Transformer) addresses these issues by introducing a hybrid, end-to-end framework. It leverages "superpoints" to bridge the gap between low-level geometric features and high-level instance queries, achieving state-of-the-art performance on benchmarks like ScanNetv2.

Methodology: Bridging the Gap with Superpoints

The core innovation of SPFormer lies in its use of superpoints as the primary unit of processing. This approach avoids the quadratic complexity of applying transformers directly to massive point clouds.

Architecture Overview

The SPFormer pipeline consists of several key stages:

  1. Backbone & Superpoint Pooling: A Sparse 3D U-Net extracts point-wise features, which are then aggregated into geometrically homogeneous "superpoints." This step reduces the input dimensionality from hundreds of thousands of points to a few hundred superpoints.
  2. Transformer Query Decoder: A set of learnable query vectors interacts with superpoint features. Each query learns to "attend" to specific superpoints that constitute a single object instance.
  3. Bipartite Matching & Mask Prediction: During training, predicted masks are matched directly to ground truth using the Hungarian algorithm. This enables a truly end-to-end learning process without the need for Non-Maximum Suppression (NMS).
Overview of the SPFormer architecture, illustrating the transition from point clouds to superpoints and the subsequent query-based instance prediction.

The Query Decoder

The heart of the system is the transformer decoder. Unlike traditional top-down methods that rely on bounding boxes, SPFormer uses a mask-based matching strategy. The decoder layers iteratively refine instance queries by alternating between self-attention (to differentiate between objects) and superpoint cross-attention (to group geometric primitives).

Detailed structure of the transformer decoder layer, highlighting the superpoint cross-attention mechanism.

Key Insights

  • Superpoints as Tokens: By treating superpoints as tokens, the model maintains geometric integrity while staying computationally efficient.
  • Attention as Soft Grouping: The cross-attention mechanism replaces hard-coded clustering rules, allowing the model to learn the optimal way to group regions into objects.
  • IoU-Aware Scoring: An auxiliary branch predicts the Intersection over Union (IoU) for each proposal, significantly improving the ranking of final predictions during inference.

Results and Evaluation

SPFormer was evaluated on the ScanNetv2 and S3DIS benchmarks, demonstrating superior accuracy and speed.

  • ScanNetv2 Performance: The model achieved 54.9% mAP, outperforming the previous state-of-the-art (SSTNet) by 4.3%. It showed remarkable gains in geometrically complex categories like "counter" and "sink."
  • Efficiency: With an inference time of 247ms per frame, SPFormer is currently the fastest among top-tier 3D instance segmentation methods.
  • Ablation Success: Experiments confirmed that matching by masks rather than bounding boxes improved performance by 6.4% mAP, validating the design choice to move away from proposal-based box fitting.

Conclusion and Future Directions

SPFormer represents a significant step forward in 3D scene understanding, proving that transformer-based query architectures can be both efficient and highly accurate when paired with superpoint representations. Future work may focus on optimizing the superpoint extraction phase—which currently accounts for a large portion of the inference time—and extending the framework to large-scale outdoor LiDAR datasets.

Summary updated

BibTeX
@inproceedings{sun2023superpoint,
  author = {Jiahao Sun and Chunmei Qing and Junpeng Tan and Xiangmin Xu},
  title = {{Superpoint Transformer for 3D Scene Instance Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25335},
}
Figure