G-FARS: Gradient-Field-Based Auto-Regressive Sampling for 3D Part Grouping

Junfeng Cheng, Tania StathakiCVPR 2024 · DOI · arXiv

Introduction: The Challenge of 3D Part Grouping

In many real-world scenarios, such as automated recycling, archaeological reconstruction, or industrial assembly, AI systems are presented with a jumbled collection of 3D components. The task of identifying which individual parts belong to the same object is known as 3D part grouping. While significant progress has been made in part assembly—the process of arranging a known set of parts into a coherent whole—the prerequisite task of grouping remains a formidable challenge.

The 3D part grouping task: identifying coherent object groups from a mixed set of parts.

Two primary difficulties hinder traditional approaches:

  1. Complex Geometric Relationships: Determining if a specific leg belongs to a specific chair seat requires an understanding of geometric "affordances" and structural compatibility.
  2. Unknown Cardinality: In a realistic "pile" of parts, the number of objects ($N$) is typically unknown. Standard neural networks often struggle with output sizes that vary between instances.

The G-FARS Framework

To address these issues, we introduce G-FARS (Gradient-Field-based Auto-Regressive Sampling). G-FARS reframes part grouping as a generative process. Instead of attempting to solve the entire set at once, the model adopts an iterative, auto-regressive strategy: it identifies one group of parts, removes them from the set, and repeats the process until the input set is exhausted.

Score-Based Selection

The core innovation of G-FARS lies in its use of a gradient-field-based selection Graph Neural Network (GNN). This approach is inspired by score-based generative models, which have seen immense success in image generation.

The G-FARS architecture: an iterative loop using PointNet for feature extraction and a GNN for gradient-field-based selection.

The framework operates in several stages:

  • Feature Extraction: A PointNet backbone processes the raw 3D point clouds of individual parts into high-dimensional feature vectors ($F_P^n$).
  • Gradient Field Estimation: The GNN learns to estimate the "score"—the gradient of the log-conditional probability density of a selection vector. This effectively creates a "force field" in the selection space that guides a random initial selection toward a valid, coherent combination of parts.
  • Iterative Sampling: Using a Predictor-Corrector sampler, the model refines a selection vector ($c_n$) over multiple steps until it settles on a group.
  • Auto-Regression: Once a group is selected, it is removed from the input set $P_n$, and the remaining parts $P_{n+1}$ become the input for the next iteration.

Experimental Evaluation

G-FARS was evaluated using the PartNet dataset, focusing on three diverse categories: Chairs (6,323 instances), Tables (8,218 instances), and Lamps (2,207 instances). To simulate real-world grouping challenges, a "random mixing method" was employed to create sets containing parts from multiple different shapes.

Key Results

G-FARS demonstrated superior performance compared to existing baselines like GRU-Mask and Comp-Net. On the Chair dataset, G-FARS achieved an F1 score of 0.781, a significant improvement over the 0.678 achieved by Comp-Net.

Beyond quantitative metrics, the model exhibited several key strengths:

  • Generalization: When trained on a mixture of categories, the model's performance improved, suggesting it learns to distinguish parts more effectively when exposed to greater variety.
  • Robustness to Noise: A unique advantage of the gradient-field approach is its ability to perform "noisy part removal." If "junk" parts that do not belong to any valid object are added to the set, the model naturally ignores them because they do not align with the learned gradient fields of valid structures.
  • Cross-Category Transfer: The model maintained reasonable performance even when tested on categories it had never seen during training, indicating it learns general geometric principles rather than just memorizing specific shapes.

Conclusion and Future Directions

G-FARS represents a significant step forward in autonomous 3D part grouping by combining auto-regressive logic with generative score-based modeling. While currently validated on synthetic datasets, future work will focus on adapting the framework for real-world 3D scans, which often include sensor noise and occlusions. Additionally, integrating G-FARS with robotic manipulation systems could enable physical sorting and assembly in complex industrial environments.

Summary updated

BibTeX
@inproceedings{cheng2024g,
  author = {Junfeng Cheng and Tania Stathaki},
  title = {{G-FARS: Gradient-Field-Based Auto-Regressive Sampling for 3D Part Grouping}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02611},
}
Figure