Mask3D: Mask Transformer for 3D Semantic Instance Segmentation

Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian Leibe

TL;DR

Mask3D introduces the first Transformer-based end-to-end approach for 3D semantic instance segmentation, directly predicting instance masks from point clouds without relying on traditional voting or clustering methods. This novel approach achieves state-of-the-art performance across multiple challenging benchmarks, demonstrating the power of Transformers in understanding complex 3D scenes.

What Problem Does This Paper Address?

Segmenting 3D scenes into individual objects with their semantic labels (e.g., identifying each chair as a "chair") is a crucial task for robotics, augmented reality, and scene understanding. Traditional methods for this "3D semantic instance segmentation" often involve complex, multi-stage pipelines. These typically include:

  1. Voting mechanisms: Points in the scene "vote" for potential object centers or other geometric properties. This often requires hand-crafted features and can be sensitive to noise or unusual object shapes.
  2. Geometric clustering: After voting, these votes are grouped together using heuristics and manually tuned parameters (like radii) to form instance masks.

These approaches are brittle, require significant hyperparameter tuning, and don't directly optimize for the final instance masks. The paper aims to overcome these limitations by proposing a unified, end-to-end Transformer-based model that directly predicts instance masks.

Methods

The core of Mask3D is its reliance on Transformer decoders, inspired by successes in 2D vision tasks like DETR and Mask2Former. Here's a breakdown of the key components:

  • Feature Backbone: A sparse convolutional network (specifically, a Minkowski U-Net) processes the raw 3D point cloud to extract multi-scale point features. This backbone is crucial for efficiently handling large, sparse 3D data.
  • Instance Queries: The model uses a set of learnable "instance queries." Each query is intended to represent a single object instance in the scene.
  • Transformer Decoder: These instance queries are iteratively refined by attending to the features extracted by the backbone. This involves:
  • Cross-Attention: Queries attend to point cloud features at various scales.
  • Self-Attention: Queries attend to each other, allowing them to reason about relationships between potential instances and avoid duplicates.
  • Mask Module: For each refined query, a "mask module" predicts a semantic class and a binary instance mask. This is achieved by computing a similarity score between the query's features and the point cloud features, followed by a sigmoid and thresholding.
  • Direct Supervision: Crucially, Mask3D directly optimizes for the predicted instance masks using a combination of Dice loss and binary cross-entropy loss. This avoids the need for proxy losses on intermediate predictions like votes.
  • Bipartite Graph Matching: To establish correspondences between predicted and ground truth masks during training, the authors employ Hungarian matching, a standard technique for optimal assignment.
  • Non-Parametric Queries: The paper explores using non-parametric queries (initialized with sampled point positions) which offer flexibility during inference, allowing for a trade-off between speed and performance without retraining.

Datasets

Mask3D was evaluated on four challenging 3D semantic instance segmentation datasets:

  • ScanNet v2: A widely used dataset of reconstructed indoor scenes with 18 object categories.
  • ScanNet200: An extension of ScanNet with a significantly larger number of classes, designed to test performance on long-tail distributions.
  • S3DIS: A large-scale indoor dataset with 13 classes, evaluated using Area 5 and 6-fold cross-validation.
  • STPLS3D: A synthetic outdoor dataset mimicking aerial photogrammetry point clouds, with 14 instance classes.

Key Insights and Ideas

  • Transformers for 3D Instance Segmentation: The paper demonstrates that generic Transformer building blocks, successfully applied in 2D, can be effectively adapted for the complex task of 3D semantic instance segmentation, achieving state-of-the-art results.
  • End-to-End Mask Prediction: Directly predicting instance masks without intermediate voting or clustering steps simplifies the pipeline and allows for direct optimization of the desired output.
  • Domain-Agnostic Components: Mask3D's reliance on Transformers and sparse convolutions makes its core components largely domain-agnostic, reducing the need for hand-tuned, dataset-specific heuristics.
  • Non-Parametric Queries for Flexibility: The use of non-parametric queries allows for dynamic adjustment of the number of queries at inference time, offering a practical trade-off between speed and accuracy.
  • Masked Cross-Attention: Using intermediate predicted masks to guide the cross-attention mechanism helps queries focus on relevant parts of the scene, improving segmentation quality.

Key Results

Mask3D significantly outperforms existing state-of-the-art methods across all tested benchmarks:

  • ScanNet test: +6.2 mAP improvement.
  • S3DIS 6-fold: +10.1 mAP improvement.
  • STPLS3D: +11.2 mAP improvement.
  • ScanNet200 test: +12.4 mAP improvement.

Qualitatively, Mask3D demonstrates robust performance on diverse scenes, successfully segmenting objects that pose challenges for voting-based methods, such as large, arbitrarily shaped objects. The paper also shows that a smaller feature backbone can still yield competitive results, indicating the robustness of the Transformer-based approach.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Merged Instances: Mask3D can sometimes merge instances of the same class that are far apart, especially if they have similar semantics and geometry. This is attributed to the model's ability to attend to the entire point cloud.
  • Post-processing for Merged Instances: To address the merged instance issue, the authors propose using DBSCAN for post-processing, which splits erroneously merged instances based on spatial contiguity. However, they note that voting-based methods inherently avoid this due to their explicit geometric priors.
  • Computational Cost: While Transformer-based, the model still requires significant computational resources, with training taking around 78 hours on an NVIDIA A40 GPU for 2cm voxelization.

Future work could explore more efficient Transformer architectures for 3D, further investigate query initialization strategies, and potentially integrate explicit geometric reasoning to further mitigate instance merging issues.

Summary updated

Abstract

Modern 3D semantic instance segmentation approaches predominantly rely on specialized voting mechanisms followed by carefully designed geometric clustering techniques. Building on the successes of recent Transformer-based methods for object detection and image segmentation, we propose the first Transformer-based approach for 3D semantic instance segmentation. We show that we can leverage generic Transformer building blocks to directly predict instance masks from 3D point clouds. In our model - called Mask3D - each object instance is represented as an instance query. Using Transformer decoders, the instance queries are learned by iteratively attending to point cloud features at multiple scales. Combined with point features, the instance queries directly yield all instance masks in parallel. Mask3D has several advantages over current state-of-the-art approaches, since it neither relies on (1) voting schemes which require hand-selected geometric properties (such as centers) nor (2) geometric grouping mechanisms requiring manually-tuned hyper-parameters (e.g. radii) and (3) enables a loss that directly optimizes instance masks. Mask3D sets a new state-of-the-art on ScanNet test (+6.2mAP), S3DIS 6-fold (+10.1 mAP), STPLS3D (+11.2 mAP) and ScanNet200 test (+12.4 mAP).
BibTeX
@inproceedings{schult2023mask,
  author = {Jonas Schult and Francis Engelmann and Alexander Hermans and Or Litany and Siyu Tang and Bastian Leibe},
  title = {{Mask3D: Mask Transformer for 3D Semantic Instance Segmentation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160590},
}
Figure