Abstract
Standard instance segmentation models often struggle with large objects in indoor scenes due to fixed-resolution mask heads. This paper presents the winning solution for the 2020 3D AI Challenge’s Instance Segmentation track, which overcomes this "resolution bottleneck" using PointRend—an iterative, point-based rendering approach. By focusing computation on uncertain boundary points, the proposed method produces high-resolution, crisp masks for large-scale furniture.
Introduction
Most modern instance segmentation frameworks, such as Mask R-CNN or Hybrid Task Cascade (HTC), are optimized for datasets like COCO, where objects are typically small. These models usually predict masks on a small, fixed-resolution grid (e.g., $28 \times 28$). While effective for small objects, this approach fails on large-scale indoor data where objects like sofas and dining tables require much higher precision.
The Resolution Bottleneck in 3D-FUTURE
The 3D-FUTURE dataset, focused on indoor furniture, presents a significant distribution shift compared to COCO. As shown in the figure below, the median object size in 3D-FUTURE is approximately five times larger than in COCO.
When a massive object is compressed into a tiny $14 \times 14$ or $28 \times 28$ feature map, the resulting mask boundaries become blurry, losing critical fine-grained details. The authors identified that the primary failure mode for existing models on 3D-FUTURE was not feature extraction, but rather this output resolution bottleneck.
Methodology: Point-Based Rendering
To address the resolution issue, the authors adopted PointRend as their core framework. Unlike traditional convolutional mask heads that operate on a regular grid, PointRend treats segmentation as a rendering problem. It adaptively selects "uncertain" points—typically along object boundaries—and iteratively refines their labels.
Architectural Enhancements
Beyond the base PointRend architecture, several strategic modifications were implemented to maximize performance:
- Advanced Backbones: Transitioning to ResNeXt101-64x4d and Res2Net101 provided richer feature representations.
- FPN Integration: A P6 level was added to the Feature Pyramid Network (FPN) to better handle multi-scale instances.
- Increased Sampling: The number of sampled points during training was increased from 14 to 26, and subdivision points during inference were raised from 28 to 70.
- High-Resolution Training: Input resolutions were increased to a range of $[1200, 1400]$ pixels to preserve fine details.
Results and Qualitative Analysis
The proposed solution secured 1st place in the 3D AI Challenge 2020. A single PointRend model achieved 79.17 mAP on the validation set, significantly outperforming HTC (74.58 mAP) and SOLOv2 (75.29 mAP). An ensemble of five models further solidified the lead with a final test score of 79.03 mAP.
Qualitatively, PointRend excels at handling overlapping instances and complex geometries. The model correctly identifies "holes" in objects (such as the gaps between chair slats) as background, a task where traditional models often fail by incorrectly filling in the gaps. This iterative refinement process allows for crisp boundaries that are essential for high-fidelity 3D scene understanding.
Conclusion
The success of this approach highlights that for large-scale object segmentation, spatial resolution and boundary precision are more critical than additional global context modules. By replacing fixed-grid predictions with iterative point-based refinement, the authors established a new benchmark for high-quality indoor instance segmentation.
Summary updated