Towards Fine-grained Large Object Segmentation 1st Place Solution to 3D AI Challenge 2020 - Instance Segmentation Track

Zehui Chen, Qiaofei Li, Feng Zhao

Abstract

Standard instance segmentation models often struggle with large objects in indoor scenes due to fixed-resolution mask heads. This paper presents the winning solution for the 2020 3D AI Challenge’s Instance Segmentation track, which overcomes this "resolution bottleneck" using PointRend—an iterative, point-based rendering approach. By focusing computation on uncertain boundary points, the proposed method produces high-resolution, crisp masks for large-scale furniture.

Introduction

Most modern instance segmentation frameworks, such as Mask R-CNN or Hybrid Task Cascade (HTC), are optimized for datasets like COCO, where objects are typically small. These models usually predict masks on a small, fixed-resolution grid (e.g., $28 \times 28$). While effective for small objects, this approach fails on large-scale indoor data where objects like sofas and dining tables require much higher precision.

The Resolution Bottleneck in 3D-FUTURE

The 3D-FUTURE dataset, focused on indoor furniture, presents a significant distribution shift compared to COCO. As shown in the figure below, the median object size in 3D-FUTURE is approximately five times larger than in COCO.

Comparison of instance area distributions between 3D-FUTURE and COCO datasets.

When a massive object is compressed into a tiny $14 \times 14$ or $28 \times 28$ feature map, the resulting mask boundaries become blurry, losing critical fine-grained details. The authors identified that the primary failure mode for existing models on 3D-FUTURE was not feature extraction, but rather this output resolution bottleneck.

Methodology: Point-Based Rendering

To address the resolution issue, the authors adopted PointRend as their core framework. Unlike traditional convolutional mask heads that operate on a regular grid, PointRend treats segmentation as a rendering problem. It adaptively selects "uncertain" points—typically along object boundaries—and iteratively refines their labels.

Architectural Enhancements

Beyond the base PointRend architecture, several strategic modifications were implemented to maximize performance:

  • Advanced Backbones: Transitioning to ResNeXt101-64x4d and Res2Net101 provided richer feature representations.
  • FPN Integration: A P6 level was added to the Feature Pyramid Network (FPN) to better handle multi-scale instances.
  • Increased Sampling: The number of sampled points during training was increased from 14 to 26, and subdivision points during inference were raised from 28 to 70.
  • High-Resolution Training: Input resolutions were increased to a range of $[1200, 1400]$ pixels to preserve fine details.

Results and Qualitative Analysis

The proposed solution secured 1st place in the 3D AI Challenge 2020. A single PointRend model achieved 79.17 mAP on the validation set, significantly outperforming HTC (74.58 mAP) and SOLOv2 (75.29 mAP). An ensemble of five models further solidified the lead with a final test score of 79.03 mAP.

Iterative refinement of object boundaries using PointRend, demonstrating superior handling of fine details and 'holes' in complex geometries.

Qualitatively, PointRend excels at handling overlapping instances and complex geometries. The model correctly identifies "holes" in objects (such as the gaps between chair slats) as background, a task where traditional models often fail by incorrectly filling in the gaps. This iterative refinement process allows for crisp boundaries that are essential for high-fidelity 3D scene understanding.

Conclusion

The success of this approach highlights that for large-scale object segmentation, spatial resolution and boundary precision are more critical than additional global context modules. By replacing fixed-grid predictions with iterative point-based refinement, the authors established a new benchmark for high-quality indoor instance segmentation.

Summary updated

Abstract

This technical report introduces our solutions of Team 'FineGrainedSeg' for Instance Segmentation track in 3D AI Challenge 2020. In order to handle extremely large objects in 3D-FUTURE, we adopt PointRend as our basic framework, which outputs more fine-grained masks compared to HTC and SOLOv2. Our final submission is an ensemble of 5 PointRend models, which achieves the 1st place on both validation and test leaderboards. The code is available at https://github.com/zehuichen123/3DFuture_ins_seg.
BibTeX
@misc{chen2020towards,
  author = {Zehui Chen and Qiaofei Li and Feng Zhao},
  title = {{Towards Fine-grained Large Object Segmentation 1st Place Solution to 3D AI Challenge 2020 - Instance Segmentation Track}},
  year = {2020},
  eprint = {2009.04650},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.04650},
}
Figure