Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories

Tiange Luo, Kaichun Mo, Zhiao Huang, Jiarui Xu, Siyu Hu, Liwei Wang, Hao Su

Introduction

Researchers have developed a novel bottom-up framework capable of discovering 3D object parts in categories it has never encountered before. By abandoning global shape recognition in favor of a learning-based grouping policy focused on local geometry, the model successfully transfers its knowledge of "parts" (such as wheels or legs) from a small set of training categories to dozens of novel ones. This approach achieves state-of-the-art zero-shot performance in 3D shape segmentation.

The Challenge of Zero-Shot Part Discovery

When a standard deep-learning model is trained on thousands of 3D chairs, it becomes highly proficient at segmenting chairs. However, when presented with a structurally different object like a faucet or a bed, it often fails entirely. This limitation arises because most state-of-the-art models rely heavily on global context. They learn the overall structure of a specific category to deduce the location of its constituent parts. Consequently, when the global structure changes, the model's predictive capability breaks down.

The core challenge is zero-shot part discovery: teaching a machine to recognize parts in unseen categories without any new training data. Humans perform this task intuitively, recognizing that a "wheel" retains its identity whether attached to a swivel chair, a car, or a toy. This research bridges that gap by prioritizing local geometric priors over global shape templates.

A Bottom-Up Agglomerative Framework

To address the limitations of global context, the authors propose a learning-based agglomerative clustering framework. Rather than analyzing the entire object and dividing it into pieces (a top-down approach), the system begins with tiny, "superpixel-like" sub-parts and iteratively groups them together (a bottom-up approach).

The system is driven by a Policy Network formulated as a contextual bandit problem. At each step, the model evaluates pairs of sub-parts and decides which merge is most promising.

Network architecture showing the Purity, Rectification, and Verification modules.

The framework relies on three primary modules to execute this policy:

  1. Purity Module: A PointNet-based component that evaluates a potential merged part and predicts the likelihood that it belongs to a single ground-truth part.
  2. Rectification Module: Acting as a structural check, this module compares two sub-parts side-by-side. It prevents the model from erroneously merging a massive part with a tiny, unrelated neighbor simply because the resulting overall "purity" score remains high.
  3. Verification Network: The final gatekeeper that processes the outputs of the previous modules to make the ultimate binary "Yes/No" decision on whether to execute the merge.

The entire system is trained using Reinforcement Learning (RL). By employing an on-policy training scheme with epsilon-greedy exploration, the model develops a robust grouping policy that generalizes effectively across diverse geometries.

Experimental Setup and Datasets

The study leverages PartNet, a comprehensive dataset featuring fine-grained, hierarchical 3D part annotations for over 26,000 models across 24 categories.

To rigorously test generalization, the researchers trained their model on just 3 categories (Chair, Lamp, and Storage Furniture). They then evaluated its zero-shot performance on the remaining 21 unseen categories, which included highly varied objects such as Scissors, Vases, Dishwashers, and Backpacks.

Key Insights

  • Local Context Drives Generalization: A critical insight of this work is that global information can actually hinder zero-shot tasks. By restricting the model's "receptive field" to local part-level features, the model is forced to learn the geometry of parts rather than the topology of specific objects.
  • The "Wheel" Analogy: While airplanes, cars, and chairs are globally distinct, their wheels share similar local structures. Learning the geometric signature of a wheel from one category enables the model to segment it in another.
  • Agglomerative Grouping as a Bandit Problem: Framing the merging process as a sequential decision-making task allows the model to learn a dynamic strategy for building objects, rather than relying on a static mapping of points to labels.

Key Results

The proposed bottom-up method achieved state-of-the-art performance in zero-shot part discovery.

Quantitatively, on the 21 unseen categories, the model significantly outperformed deep learning baselines such as SGPN and GSPN. While traditional global-context methods struggled with novel shapes, this approach maintained a high Mean Recall across three different levels of part granularity. Furthermore, the authors demonstrated a clear context trade-off: adding more global context improved performance on "seen" categories but actively degraded performance on "unseen" ones, validating their hypothesis.

Qualitative comparison of segmentation results on unseen categories (Bed and Faucet) against baseline methods.

Qualitatively, visualizations confirm that the model successfully segments complex, never-before-seen objects like faucets and beds. In contrast, previous methods often produced "blobby" or nonsensical segments due to the absence of a familiar global template.

Limitations and Future Directions

While this framework represents a significant advancement, the authors note several areas for future improvement:

  • Greedy Policy: The current model employs a "greedy" strategy, maximizing the score for the immediate next step. Incorporating long-term planning to look several steps ahead could yield even more accurate groupings.
  • Forward-Only Grouping: The model currently only merges parts and cannot "undo" a mistake. Future iterations could introduce a "split" operation to correct errors made earlier in the sequence.
  • Template Assumptions: The system assumes that parts in unseen categories share some geometric similarity with those in the training set. Completely alien geometric structures might still pose a challenge.
  • 2D Applications: The authors express strong interest in adapting this bottom-up grouping logic to 2D image object detection and segmentation tasks.

Summary updated

Abstract

A zero-shot segmentation framework for 3D shapes. Model the segmentation as a decision-making process, we propose an iterative method to dynamically extend the receptive field for achieving universal shape segmentation.
BibTeX
@inproceedings{luo2020learning,
  author = {Tiange Luo and Kaichun Mo and Zhiao Huang and Jiarui Xu and Siyu Hu and Liwei Wang and Hao Su},
  title = {{Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=O_YoaxErfn},
}
Figure