MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentation

Min Zhong, Xinghao Chen, Xiaokang Chen, Gang Zeng, Yunhe Wang

Introduction

3D instance segmentation is a fundamental task in computer vision with critical applications in robotics, autonomous driving, and augmented reality. The goal is to identify and delineate individual objects within a 3D point cloud. However, indoor environments present significant challenges due to the high complexity of scenes and the extreme variance in object scales—ranging from small decorative items to large architectural elements like walls and floors.

The Limitations of Single-Scale Clustering

Traditional "bottom-up" segmentation methods typically rely on a fixed clustering radius to group points into instances. This approach often encounters a "Goldilocks" dilemma: a small radius may correctly isolate small objects but fragment larger ones (e.g., a long sofa split into multiple segments), while a large radius might merge distinct adjacent objects (e.g., two chairs side-by-side) into a single cluster. Furthermore, initial clusters often contain "noisy" points from the background or neighboring objects, which degrades the precision of the final segmentation.

The MaskGroup Framework

To address these challenges, we propose MaskGroup, a novel framework designed to group and refine 3D instances through a hierarchical approach and a specialized refinement network.

Hierarchical Point Grouping (HPG)

Instead of relying on a single clustering threshold, the HPG algorithm employs a series of increasing radii (e.g., 0.01m, 0.03m, 0.05m). The process begins by learning an offset vector for each point to shift it toward its predicted instance center. The HPG algorithm then groups these points into small, high-confidence clusters, which are progressively merged into larger groups. This multi-scale strategy allows the model to capture geometric information across different levels of granularity, effectively handling instances of varying sizes within the same scene.

MaskScoreNet and Refinement

Recognizing that raw clusters are often imprecise "rough drafts," MaskGroup incorporates a refinement module called MaskScoreNet. This module consists of two primary branches:

  1. Mask Branch: This branch predicts a binary mask for every point within a proposed group, distinguishing the actual object points from background noise or adjacent instances.
  2. Score Branch: This branch evaluates the quality of the predicted mask, providing a confidence score used during the final Non-Maximum Suppression (NMS) phase to filter out low-quality proposals.

A specialized Mask-Pooling layer bridges these branches by using the predicted mask to aggregate features. This ensures the score branch focuses exclusively on the points identified as belonging to the instance, rather than the entire bounding volume.

Experimental Evaluation

The effectiveness of MaskGroup was validated on two industry-standard benchmarks: ScanNetV2 and S3DIS.

ScanNetV2 Performance

On the ScanNetV2 test set, MaskGroup achieved a 66.4% [email protected], outperforming the previous state-of-the-art by 1.9%. Notably, the model ranked first in 10 out of 18 object categories, demonstrating its robustness across diverse object types.

S3DIS Performance

In 6-fold cross-validation on the S3DIS dataset, the model achieved 69.9% AP50. This represents a significant 5.9% improvement over the widely used PointGroup baseline, highlighting the advantages of the hierarchical grouping strategy.

Key Insights

The success of MaskGroup stems from the synergy between multi-scale grouping and learnable masking. Ablation studies indicate that HPG and MaskScoreNet are mutually dependent. While HPG generates a comprehensive set of multi-scale proposals, the masking step acts as an essential filter, cleaning the clusters to achieve high precision. By treating grouping as an iterative process, the model naturally adapts to the geometric diversity of real-world indoor environments.

Conclusion and Future Work

MaskGroup sets a new standard for 3D instance segmentation by combining hierarchical grouping with learnable mask refinement. While the model excels at handling scale variance, challenges remain in distinguishing between geometrically similar objects in close proximity (e.g., a door next to a cabinet) and managing extremely crowded scenes. Future research may focus on incorporating more sophisticated geometric priors or improving the handling of boundary points to further enhance segmentation accuracy.

Summary updated

Abstract

This paper studies the 3D instance segmentation problem, which has a variety of real-world applications such as robotics and augmented reality. Since the surroundings of 3D objects are of high complexity, the separating of different objects is very difficult. To address this challenging problem, we propose a novel framework to group and refine the 3D instances. In practice, we first learn an offset vector for each point and shift it to its predicted instance center. To better group these points, we propose a Hierarchical Point Grouping algorithm to merge the centrally aggregated points progressively. All points are grouped into small clusters, which further gradually undergo another clustering procedure to merge into larger groups. These multi-scale groups are exploited for instance prediction, which is beneficial for predicting instances with different scales. In addition, a novel MaskScoreNet is developed to produce binary point masks of these groups for further refining the segmentation results. Extensive experiments conducted on the ScanNetV2 and S3DIS benchmarks demonstrate the effectiveness of the proposed method. For instance, our approach achieves a 66.4\% mAP with the 0.5 IoU threshold on the ScanNetV2 test set, which is 1.9\% higher than the state-of-the-art method.
BibTeX
@inproceedings{zhong2022maskgroup,
  author = {Min Zhong and Xinghao Chen and Xiaokang Chen and Gang Zeng and Yunhe Wang},
  title = {{MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentation}},
  booktitle = {ICME},
  year = {2022},
  url = {https://openreview.net/forum?id=7hnbNkyuKG},
}
Figure