Mask3D: Mask Transformer for 3D Semantic Instance Segmentation

Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian Leibe

TL;DR

Mask3D is the first competitive Transformer-based model for 3D semantic instance segmentation, moving away from hand-crafted geometric grouping and voting heuristics. By representing objects as learnable "instance queries" that attend to multi-scale point cloud features, it directly predicts binary masks and sets a new state-of-the-art across major indoor and outdoor benchmarks, including a massive +12.4 mAP jump on ScanNet200.

What Problem Does This Paper Address?

Semantic instance segmentation in 3D—identifying every individual object (instance) and its category (semantic label) in a point cloud—is a foundational task for robotics and AR. Historically, this field has been dominated by two paradigms:

  1. Bottom-up: Grouping points into instances based on learned feature similarities.
  2. Top-down: Detecting bounding boxes first and then segmenting the interior.

Most modern state-of-the-art methods rely on "center-voting," where points "vote" for the center of the object they belong to, followed by complex, manually-tuned clustering algorithms (like grouping votes within a specific radius). These methods are often brittle, require extensive hyper-parameter tuning for different datasets, and optimize for proxy tasks (like center distance) rather than the final mask itself. Mask3D aims to replace these specialized pipelines with a generic, end-to-end learnable Transformer architecture.

Methods

The Mask3D architecture consists of three main stages:

  1. Feature Backbone: A sparse convolutional U-Net (specifically MinkowskiEngine) processes the raw point cloud to produce a hierarchy of multi-scale features.
  2. Transformer Decoder: This is the heart of the model. It uses a set of instance queries—vectors that each aim to represent one object in the scene. These queries iteratively refine themselves by "attending" to the point cloud features. The authors employ masked cross-attention, where each query is forced to only look at the voxels within its currently predicted mask, helping the model focus on local object details rather than irrelevant background context.
  3. Mask Module: To generate the final output, the model computes the similarity (dot product) between the refined instance queries and the high-resolution point features. This produces a heatmap for each query, which is thresholded to create a binary mask.
The Mask3D architecture uses a sparse convolutional backbone to extract features, which are then refined by a Transformer decoder using instance queries to produce final masks.

During training, the model uses bipartite graph matching (the Hungarian algorithm) to optimally pair predicted masks with ground-truth objects, allowing for a direct loss calculation on the masks using Binary Cross-Entropy and Dice loss.

Datasets

The authors evaluated Mask3D on four diverse benchmarks:

  • ScanNet v2: The standard indoor dataset for 3D reconstruction.
  • ScanNet200: A challenging version of ScanNet with a long tail of 200 object categories.
  • S3DIS: A large-scale indoor dataset covering entire campus buildings.
  • STPLS3D: A synthetic outdoor dataset mimicking aerial photogrammetry.

Key Insights and Ideas

  • Direct Mask Prediction: By treating instance segmentation as a set-prediction problem (similar to DETR in 2D), Mask3D avoids the need for non-maximum suppression (NMS) or geometric grouping heuristics.
  • Non-parametric Initialization: The authors found that initializing instance queries based on actual point positions (using Furthest Point Sampling) outperformed learned, parametric queries. This makes the model more adaptable to the specific geometry of a scene.
  • Domain Agnosticism: Unlike previous methods that required different tuning for indoor vs. outdoor scenes, Mask3D uses the same architecture and hyperparameters across all datasets, demonstrating its robustness.
  • Sampled Cross-Attention: To handle the massive number of points in 3D scenes while staying within memory limits, the authors propose sampling voxels during the cross-attention step in training, which acts as a form of regularization similar to dropout.

Key Results

Mask3D significantly outperformed all prior methods, often by double-digit margins:

  • ScanNet Test: Achieved 56.6 mAP, a +6.2 improvement over the previous best (SoftGroup).
  • ScanNet200 Test: A massive jump to 38.3 mAP (+12.4 improvement), showing the model's strength in handling rare, "long-tail" object classes.
  • S3DIS 6-fold: Improved the state-of-the-art by +10.1 mAP.
  • STPLS3D: Set a new record with 57.3 mAP (+11.2 improvement).
Fig. 4: Qualitative Results on ScanNet. We show pairs of predicted instance masks and predicted semantic labels. On the bottom left, we show the heatmap of a failure case of two windows that are wrongly assigned to a single instance. The corresponding point features are visualized as RGB after projecting them to 3D using PCA.

Limitations and Future Work

The primary limitation identified is a "global attention" artifact: because the Transformer can attend to the entire point cloud, it occasionally merges two distant objects with very similar features (e.g., two identical windows on opposite sides of a room) into a single instance.

To mitigate this, the authors suggest an optional post-processing step using DBSCAN to split spatially disjoint masks. Future work could involve incorporating soft geometric priors directly into the attention mechanism to prevent these distant merges without needing post-processing.

Fig. 1: Mask3D. We train an end-to-end model for 3D semantic instance segmentation on point clouds. Given an input 3D point cloud (left), our Transformer-based model uses an attention mechanism to produce instance heatmaps across all points (center) and directly predicts all semantic object instances in parallel (right).

Summary updated

Abstract

Modern 3D semantic instance segmentation approaches predominantly rely on specialized voting mechanisms followed by carefully designed geometric clustering techniques. Building on the successes of recent Transformer-based methods for object detection and image segmentation, we propose the first Transformer-based approach for 3D semantic instance segmentation. We show that we can leverage generic Transformer building blocks to directly predict instance masks from 3D point clouds. In our model called Mask3D each object instance is represented as an instance query. Using Transformer decoders, the instance queries are learned by iteratively attending to point cloud features at multiple scales. Combined with point features, the instance queries directly yield all instance masks in parallel. Mask3D has several advantages over current state-of-the-art approaches, since it neither relies on (1) voting schemes which require hand-selected geometric properties (such as centers) nor (2) geometric grouping mechanisms requiring manually-tuned hyper-parameters (e.g. radii) and (3) enables a loss that directly optimizes instance masks. Mask3D sets a new state-of-the-art on ScanNet test (+6.2 mAP), S3DIS 6-fold (+10.1 mAP), STPLS3D (+11.2 mAP) and ScanNet200 test (+12.4 mAP).
BibTeX
@inproceedings{schult2023mask,
  author = {Jonas Schult and Francis Engelmann and Alexander Hermans and Or Litany and Siyu Tang and Bastian Leibe},
  title = {{Mask3D: Mask Transformer for 3D Semantic Instance Segmentation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160590},
}
Figure