RPG: Learning Recursive Point Cloud Generation

Wei-Jan Ko, Hui-Yu Huang, Yu-Liang Kuo, Chen-Yi Chiu, Li-Heng Wang, Wei-Chen ChiuarXiv 2021 · OpenReview

TL;DR

This paper introduces RPG, a novel recursive point cloud generator that creates detailed 3D models by progressively expanding from a single point. A key innovation is its ability to unsupervisedly discover semantic part segmentations during this generation process, offering a more interpretable understanding of 3D shapes.

What Problem Does This Paper Address?

Understanding and generating 3D point clouds is crucial for applications like robotics and autonomous vehicles. While existing methods focus on either modeling fine geometric details, extracting semantic parts, or finding shape correspondences, this paper aims to unify these three levels of granularity into a single, coherent framework. The challenge lies in generating high-fidelity point clouds that are also semantically meaningful and interpretable, without requiring explicit part annotations during training.

Methods

The core of the proposed method is the Recursive Point Cloud Generator (RPG). It operates in a coarse-to-fine manner through a series of "point expansion stages." Starting with a single point and a latent representation of the target 3D shape, an expansion module (E) recursively duplicates points and their associated structural representations. Each expansion stage generates a denser point cloud and refines the structural information, effectively building a hierarchical representation.

A significant aspect is the unsupervised discovery of semantic parts. The hierarchical parent-child relationships formed during the recursive expansion naturally lead to part segmentation. By tracing a point's lineage back to its ancestors at earlier stages, we can group points belonging to the same "part."

The expansion module itself is designed to be lightweight and efficient by sharing weights across all expansion stages. This recursive application of the same module, along with shared parameters in the underlying functions (F and MLP), contributes to a compact model. The training objective primarily uses Chamfer Distance for reconstruction and incorporates a regularization term to encourage smaller scaling factors, preventing overlapping parts.

Datasets

The experiments are conducted on ShapeNet, a large-scale dataset containing over 51,000 3D shapes across 55 categories. For evaluation, the authors follow standard practices, sampling 2048 points per shape and normalizing them.

Key Insights and Ideas

  • Unified Granularity: The RPG framework successfully integrates geometric detail generation with semantic part discovery in a single, end-to-end process. This is a significant step beyond methods that focus on only one aspect.
  • Unsupervised Semantic Segmentation: The hierarchical nature of the recursive generation inherently provides part segmentation without needing explicit labels. This is a powerful unsupervised learning capability for 3D data.
  • Lightweight and Efficient Architecture: Weight sharing in the expansion module leads to a significantly smaller model size compared to many state-of-the-art methods, making training and inference more efficient.
  • Hierarchical Understanding: The intermediate point clouds generated at each stage offer a coarse-to-fine view of the object, providing insights into the generative process and hierarchical structure of shapes.
  • Co-segmentation: Because the generative process is learned across multiple instances, the model naturally discovers common part segmentations shared across objects of the same category, enabling co-segmentation.

Key Results

  • Reconstruction: The RPG framework achieves comparable or superior reconstruction performance to baselines like AtlasNet, PointFlow, and ShapeGF on the ShapeNet dataset, using significantly fewer parameters (1.8M for RPG vs. 3.5M for ShapeGF).
  • Generation: RPG demonstrates strong performance in point cloud generation, outperforming methods like r-GAN, GCN, TreeGAN, and PF across various metrics (MMD, COV, 1-NNA) for Airplane and Chair categories. It also achieves comparable results to ShapeGF with a much smaller model size.
  • Part Segmentation: The method produces compact and semantically meaningful part segmentations, outperforming TreeGAN in purity (88% vs. 74%) on ShapeNet-Part annotations.
  • Co-segmentation: Qualitative results show successful co-segmentation of common parts across object instances, highlighting intra-class similarities.

Limitations and Future Work

  • Small Sharp Details: While the paper mentions that TreeGAN struggles with small sharp details, it's not explicitly stated as a limitation for RPG, though the qualitative examples might suggest room for improvement in extremely fine details.
  • Scaling Factors: The authors introduce a regularization term to encourage smaller scaling factors, implying that uncontrolled scaling could lead to overlapping parts. This suggests that managing the spatial distribution of points during expansion is an ongoing consideration.
  • Future Work: The paper doesn't explicitly detail future work directions, but the unified framework suggests potential for extending it to other 3D tasks, improving detail generation, and exploring more complex shape structures. The ability to generate intermediate point clouds also opens avenues for interactive 3D modeling and editing.

Summary updated

Abstract

In this paper we propose a novel point cloud generator that is able to reconstruct and generate 3D point clouds composed of semantic parts. Given a latent representation of the target 3D model, the generation starts from a single point and gets expanded recursively to produce the high-resolution point cloud via a sequence of point expansion stages. During the recursive procedure of generation, we not only obtain the coarse-to-fine point clouds for the target 3D model from every expansion stage, but also unsupervisedly discover the semantic segmentation of the target model according to the hierarchical/parent-child relation between the points across expansion stages. Moreover, the expansion modules and other elements used in our recursive generator are mostly sharing weights thus making the overall framework light and efficient. Extensive experiments are conducted to demonstrate that our proposed point cloud generator has comparable or even superior performance on both generation and reconstruction tasks in comparison to various baselines, as well as provides the consistent co-segmentation among 3D instances of the same object class.
BibTeX
@misc{ko2021rpg,
  author = {Wei-Jan Ko and Hui-Yu Huang and Yu-Liang Kuo and Chen-Yi Chiu and Li-Heng Wang and Wei-Chen Chiu},
  title = {{RPG: Learning Recursive Point Cloud Generation}},
  year = {2021},
  eprint = {2105.14322},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.14322},
}
Figure