Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories

Tiange Luo, Kaichun Mo, Zhiao Huang, Jiarui Xu, Siyu Hu, Liwei Wang, Hao Su

Abstract

We address the problem of discovering 3D parts for objects in unseen categories. Being able to learn the geometry prior of parts and transfer this prior to unseen categories pose fundamental challenges on data-driven shape segmentation approaches. Formulated as a contextual bandit problem, we propose a learning-based agglomerative clustering framework which learns a grouping policy to progressively group small part proposals into bigger ones in a bottom-up fashion. At the core of our approach is to restrict the local context for extracting part-level features, which encourages the generalizability to unseen categories. On the large-scale fine-grained 3D part dataset, PartNet, we demonstrate that our method can transfer knowledge of parts learned from 3 training categories to 21 unseen testing categories without seeing any annotated samples. Quantitative comparisons against four shape segmentation baselines shows that our approach achieve the state-of-the-art performance.

Introduction

Most existing 3D shape segmentation models employ a "top-down" approach, analyzing the entire object to identify constituent parts. While effective for known categories, these models often fail to generalize to unseen categories. For instance, a model trained to recognize chair legs may fail to identify legs on a bed or table because the global context—the overall shape—is significantly different.

This research addresses the challenge of zero-shot 3D part discovery. Humans can recognize parts of unfamiliar objects by identifying local geometric cues like convexity or symmetry that are consistent across categories. By shifting from global to local context, the proposed framework aims to segment parts in 21 unseen categories after training on only three.

Methodology: Learning-based Agglomerative Clustering

The core of the approach is a bottom-up framework that learns a policy to group small, "superpixel-like" sub-parts into semantically meaningful parts. This process is formulated as a contextual bandit problem, where the model sequentially decides whether to merge pairs of sub-parts.

The architecture of the Purity Module, Rectification Module, and Verification Network.

The framework consists of several key modules:

  • Purity Module: Predicts the "purity" of a proposed merged part, estimating the likelihood that all points belong to the same ground-truth part.
  • Rectification Module: Explicitly compares two sub-parts before merging to prevent large parts from "swallowing" small, unrelated neighbors—a common failure in standard clustering.
  • Verification Network: Acts as a final check to decide if a merge should proceed, preventing over-grouping.
  • Restricted Context: Crucially, features are extracted only from the local sub-parts under consideration. This prevents the model from overfitting to the global structure of training categories and encourages universal "part-ness" learning.

Experimental Setup

The model was evaluated using PartNet, a dataset with fine-grained annotations for over 26,000 3D models.

  • Training: Conducted on only three categories: Chair, Lamp, and Storage Furniture.
  • Testing: Evaluated on 21 unseen categories, including Bed, Faucet, Dishwasher, and Scissors, to test zero-shot generalization capabilities.

Results and Insights

The proposed method significantly outperformed state-of-the-art (SOTA) deep learning baselines. Quantitatively, it achieved a Mean Recall of 42.1 on unseen categories, compared to 35.7 for the best baseline.

Qualitative comparison showing the discovery of clean, distinct parts in unseen categories (faucet and cart) compared to messy baseline results.

Qualitative results demonstrate that while SOTA methods often produce fragmented or incomplete segments on new objects, this bottom-up approach discovers clean, distinct parts, such as the blades of scissors or the handles of a bag.

Key insights from the study include:

  • Local Context for Generalization: Global context acts as a distractor in zero-shot tasks. Restricting the network to local geometry allows it to learn geometric transitions (e.g., between a handle and a pipe) that are universal across categories.
  • Addressing the "Swallowing" Problem: The Rectification Module proved essential for maintaining high-quality boundaries, as purity scores alone were insufficient for preventing large parts from absorbing small neighbors.

Conclusion and Future Work

This framework demonstrates that learning universal grouping rules from a few categories can successfully transfer to a wide range of unseen 3D objects. Future work may involve moving from a greedy contextual bandit approach to full Reinforcement Learning for long-term planning. Additionally, incorporating a "split" operation could allow the model to correct early grouping errors. The authors also envision applying this "learning to group" logic to 2D image segmentation tasks.

Summary updated

Abstract

We address the problem of discovering 3D parts for objects in unseen categories. Being able to learn the geometry prior of parts and transfer this prior to unseen categories pose fundamental challenges on data-driven shape segmentation approaches. Formulated as a contextual bandit problem, we propose a learning-based agglomerative clustering framework which learns a grouping policy to progressively group small part proposals into bigger ones in a bottom-up fashion. At the core of our approach is to restrict the local context for extracting part-level features, which encourages the generalizability to unseen categories. On the large-scale fine-grained 3D part dataset, PartNet, we demonstrate that our method can transfer knowledge of parts learned from 3 training categories to 21 unseen testing categories without seeing any annotated samples. Quantitative comparisons against four shape segmentation baselines shows that our approach achieve the state-of-the-art performance.
BibTeX
@inproceedings{luo2020learning,
  author = {Tiange Luo and Kaichun Mo and Zhiao Huang and Jiarui Xu and Siyu Hu and Liwei Wang and Hao Su},
  title = {{Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=O_YoaxErfn},
}
Figure