PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding

Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, Hao Su

Abstract

We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information. Our dataset consists of 573,585 part instances over 26,671 3D models covering 24 object categories. This dataset enables and serves as a catalyst for many tasks such as shape analysis, dynamic 3D scene modeling and simulation, affordance analysis, and others. Using our dataset, we establish three benchmarking tasks for evaluating 3D part recognition: fine-grained semantic segmentation, hierarchical semantic segmentation, and instance segmentation. We benchmark four state-of-the-art 3D deep learning algorithms for fine-grained semantic segmentation and three baseline methods for hierarchical semantic segmentation. We also propose a novel method for part instance segmentation and demonstrate its superior performance over existing methods.

Introduction

While computer vision has achieved significant success in whole-object identification, understanding the internal structure of 3D objects remains a major challenge. For applications like robotic manipulation, physical simulation, and high-fidelity 3D generation, a machine must perceive an object not as a monolithic block, but as a functional assembly of parts. For instance, a robot needs to distinguish a chair's handle from its backrest to perform a specific action.

Overview of PartNet tasks: fine-grained, instance, and hierarchical segmentation.

Prior 3D datasets were often limited by small scales or coarse annotations. PartNet addresses these gaps by providing a massive, consistently annotated dataset that maps object structures with unprecedented detail, moving beyond simple labels to capture the "nuts and bolts" of 3D geometry.

The PartNet Dataset

PartNet is built upon the ShapeNetCore repository, expanding it with over 570,000 individual part instances. It covers 26,671 3D models across 24 diverse categories, including furniture, appliances, and tools.

Gallery of object categories in PartNet.

Compared to previous standards like ShapeNet Part, which typically provided only 2–5 coarse parts per object, PartNet offers an average of 18 parts per shape. Some complex models contain over 200 distinct components. This granularity allows for a much more nuanced understanding of object composition and variety.

Annotation Methodology

To ensure consistency across such a large scale, the researchers employed a structured annotation framework:

  • Hierarchical Templates: "And-Or-Graph" (AOG) templates guided annotators through a logical decomposition of objects. "And-nodes" represent mandatory components (e.g., a chair has a seat and a base), while "Or-nodes" account for structural variations (e.g., a base can be a pedestal or a set of legs).
  • Custom Interface: A web-based 3D GUI was developed to allow professional annotators to navigate these hierarchies and precisely segment 3D meshes by "cutting" them into parts.

Benchmarking Tasks and Results

The authors established three primary benchmarks to evaluate the current state of 3D part recognition:

  1. Fine-grained Semantic Segmentation: Testing models like PointNet, PointNet++, SpiderCNN, and PointCNN. PointCNN led with a 59.8% mIoU. However, a key insight was that performance drops significantly as granularity increases; smaller, rarer parts remain difficult for current AI to identify.
  2. Hierarchical Semantic Segmentation: Comparing top-down and bottom-up approaches. An ensemble method that utilizes separate networks for different hierarchy levels achieved the best performance (51.7% mIoU), demonstrating that hierarchical context helps narrow down part identities.
  3. Instance Segmentation: The authors proposed a novel "detection-by-segmentation" network that predicts semantic labels and instance masks simultaneously. This method achieved 54.4% mAP, significantly outperforming the SGPN baseline (46.8%) and producing more robust detections.

Conclusion and Future Directions

PartNet reveals that geometry alone is sometimes insufficient; parts with identical shapes can have different semantic meanings based on their global context (e.g., a bed's headboard vs. its footboard). Current limitations include the lack of texture and material data, as well as significant data imbalance for rare parts. Future work aims to incorporate object articulation (how parts move) and affordances (how humans interact with parts), moving closer to the goal of training intelligent agents for real-world environments.

Summary updated

Abstract

We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information. Our dataset consists of 573,585 part instances over 26,671 3D models covering 24 object categories. This dataset enables and serves as a catalyst for many tasks such as shape analysis, dynamic 3D scene modeling and simulation, affordance analysis, and others. Using our dataset, we establish three benchmarking tasks for evaluating 3D part recognition: fine-grained semantic segmentation, hierarchical semantic segmentation, and instance segmentation. We benchmark four state-of-the-art 3D deep learning algorithms for fine-grained semantic segmentation and three baseline methods for hierarchical semantic segmentation. We also propose a novel method for part instance segmentation and demonstrate its superior performance over existing methods.
BibTeX
@inproceedings{mo2019partnet,
  author = {Kaichun Mo and Shilin Zhu and Angel X. Chang and Li Yi and Subarna Tripathi and Leonidas J. Guibas and Hao Su},
  title = {{PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=9IlsbFLY4v},
}
Figure