PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding

Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, Hao Su

Abstract

We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information. Our dataset consists of 573,585 part instances over 26,671 3D models covering 24 object categories. This dataset enables and serves as a catalyst for many tasks such as shape analysis, dynamic 3D scene modeling and simulation, affordance analysis, and others. Using our dataset, we establish three benchmarking tasks for evaluating 3D part recognition: fine-grained semantic segmentation, hierarchical semantic segmentation, and instance segmentation. We benchmark four state-of-the-art 3D deep learning algorithms for fine-grained semantic segmentation and three baseline methods for hierarchical semantic segmentation. We also propose a baseline method for part instance segmentation and demonstrate its superior performance over existing methods.

Introduction

While computer vision has made massive strides in identifying object categories, machines still struggle to understand the internal structural composition of objects. For applications like robotic manipulation or autonomous interaction, a coarse label like "chair" is insufficient; a robot must identify the handle of a door or the specific drawers of a cabinet to interact with them effectively.

Existing 3D datasets, such as ShapeNet, often provide only coarse labels, lacking the fine-grained detail, hierarchical structure, and instance-level separation necessary for complex real-world tasks. PartNet addresses this gap by providing a massive, high-fidelity dataset with over 570,000 individual part instances, establishing a new gold standard for training AI to understand 3D shapes with high precision.

PartNet illustrates the progression from coarse to fine-grained segmentation and defines three core benchmarking tasks.

The PartNet Dataset

The primary contribution of this work is the PartNet dataset, which significantly expands the scale and granularity of available 3D part annotations. It includes 26,671 3D models from ShapeNet across 24 categories, including furniture, electronics, and household items.

On average, a single object in PartNet contains 18 parts, a substantial increase from the 2–5 parts found in previous datasets. These annotations are organized into hierarchies up to seven levels deep, capturing the complex "family tree" of an object's components. This structure allows models to leverage hierarchical priors—for instance, knowing that a wheel is part of a base, which is in turn part of a chair.

PartNet includes 24 diverse object categories, ranging from furniture to household items.

Methodology and Benchmarking

To ensure consistency at scale, the authors developed a rigorous pipeline using "And-Or-Graph" (AOG) templates. "And-nodes" decompose parts into sub-components, while "Or-nodes" represent categorical choices (e.g., a lamp can be a table lamp or a ceiling lamp). Annotations were performed by 66 professional annotators using a custom web-based 3D GUI, allowing for direct interaction with meshes, including cross-sectioning and mesh-cutting to isolate minute components.

The paper establishes three core challenges for the community:

  1. Fine-grained Semantic Segmentation: Labeling every point in a cloud with its specific part type.
  2. Hierarchical Semantic Segmentation: Predicting the entire structural hierarchy of an object.
  3. Instance Segmentation: Detecting and isolating individual part instances (e.g., distinguishing between four identical chair legs).

Key Results and Insights

Benchmarking state-of-the-art 3D deep learning algorithms (PointNet, PointNet++, SpiderCNN, and PointCNN) revealed that current models struggle with the increased complexity of PartNet. For fine-grained semantic segmentation, the best-performing model, PointCNN, achieved a mean IoU (Intersection over Union) of 46.4%, a sharp decline from the ~64% seen on coarser, easier tasks.

Qualitative analysis shows that models frequently miss small functional parts like handles or thin structural bars. For instance segmentation, the authors proposed a new baseline that outperformed previous methods like SGPN (54.4 mAP vs. 46.8 mAP) by producing cleaner boundaries and utilizing structural priors.

Conclusion and Future Work

PartNet serves as a critical "stress test" for 3D deep learning, highlighting the need for models that can handle both local geometric detail and global shape context. While it provides a robust foundation for part-level understanding, future work is needed to incorporate textures, materials, and articulation information (how parts move) to further bridge the gap between digital models and real-world robotic interaction.

Summary updated

Abstract

We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information. Our dataset consists of 573,585 part instances over 26,671 3D models covering 24 object categories. This dataset enables and serves as a catalyst for many tasks such as shape analysis, dynamic 3D scene modeling and simulation, affordance analysis, and others. Using our dataset, we establish three benchmarking tasks for evaluating 3D part recognition: fine-grained semantic segmentation, hierarchical semantic segmentation, and instance segmentation. We benchmark four state-of-the-art 3D deep learning algorithms for fine-grained semantic segmentation and three baseline methods for hierarchical semantic segmentation. We also propose a baseline method for part instance segmentation and demonstrate its superior performance over existing methods.
BibTeX
@inproceedings{mo2019partnet,
  author = {Kaichun Mo and Shilin Zhu and Angel X. Chang and Li Yi and Subarna Tripathi and Leonidas J. Guibas and Hao Su},
  title = {{PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=9IlsbFLY4v},
}
Figure