Introduction
3D instance segmentation is a fundamental task in computer vision with critical applications in robotics, autonomous driving, and augmented reality. The goal is to identify and delineate individual objects within a 3D point cloud. However, indoor environments present significant challenges due to the high complexity of scenes and the extreme variance in object scales—ranging from small decorative items to large architectural elements like walls and floors.
The Limitations of Single-Scale Clustering
Traditional "bottom-up" segmentation methods typically rely on a fixed clustering radius to group points into instances. This approach often encounters a "Goldilocks" dilemma: a small radius may correctly isolate small objects but fragment larger ones (e.g., a long sofa split into multiple segments), while a large radius might merge distinct adjacent objects (e.g., two chairs side-by-side) into a single cluster. Furthermore, initial clusters often contain "noisy" points from the background or neighboring objects, which degrades the precision of the final segmentation.
The MaskGroup Framework
To address these challenges, we propose MaskGroup, a novel framework designed to group and refine 3D instances through a hierarchical approach and a specialized refinement network.
Hierarchical Point Grouping (HPG)
Instead of relying on a single clustering threshold, the HPG algorithm employs a series of increasing radii (e.g., 0.01m, 0.03m, 0.05m). The process begins by learning an offset vector for each point to shift it toward its predicted instance center. The HPG algorithm then groups these points into small, high-confidence clusters, which are progressively merged into larger groups. This multi-scale strategy allows the model to capture geometric information across different levels of granularity, effectively handling instances of varying sizes within the same scene.
MaskScoreNet and Refinement
Recognizing that raw clusters are often imprecise "rough drafts," MaskGroup incorporates a refinement module called MaskScoreNet. This module consists of two primary branches:
- Mask Branch: This branch predicts a binary mask for every point within a proposed group, distinguishing the actual object points from background noise or adjacent instances.
- Score Branch: This branch evaluates the quality of the predicted mask, providing a confidence score used during the final Non-Maximum Suppression (NMS) phase to filter out low-quality proposals.
A specialized Mask-Pooling layer bridges these branches by using the predicted mask to aggregate features. This ensures the score branch focuses exclusively on the points identified as belonging to the instance, rather than the entire bounding volume.
Experimental Evaluation
The effectiveness of MaskGroup was validated on two industry-standard benchmarks: ScanNetV2 and S3DIS.
ScanNetV2 Performance
On the ScanNetV2 test set, MaskGroup achieved a 66.4% [email protected], outperforming the previous state-of-the-art by 1.9%. Notably, the model ranked first in 10 out of 18 object categories, demonstrating its robustness across diverse object types.
S3DIS Performance
In 6-fold cross-validation on the S3DIS dataset, the model achieved 69.9% AP50. This represents a significant 5.9% improvement over the widely used PointGroup baseline, highlighting the advantages of the hierarchical grouping strategy.
Key Insights
The success of MaskGroup stems from the synergy between multi-scale grouping and learnable masking. Ablation studies indicate that HPG and MaskScoreNet are mutually dependent. While HPG generates a comprehensive set of multi-scale proposals, the masking step acts as an essential filter, cleaning the clusters to achieve high precision. By treating grouping as an iterative process, the model naturally adapts to the geometric diversity of real-world indoor environments.
Conclusion and Future Work
MaskGroup sets a new standard for 3D instance segmentation by combining hierarchical grouping with learnable mask refinement. While the model excels at handling scale variance, challenges remain in distinguishing between geometrically similar objects in close proximity (e.g., a door next to a cabinet) and managing extremely crowded scenes. Future research may focus on incorporating more sophisticated geometric priors or improving the handling of boundary points to further enhance segmentation accuracy.
Summary updated