Point Cloud Completion by Learning Shape Priors

Xiaogang Wang, Marcelo H. Ang, Gim Hee Lee

TL;DR

This paper introduces a novel point cloud completion method that learns shape priors from both complete and incomplete 3D data. By employing feature alignment strategies and a coarse-to-fine pipeline, their approach reconstructs dense and detailed point clouds, outperforming existing methods on various benchmarks.

What Problem Does This Paper Address?

Reconstructing complete and detailed 3D objects from incomplete and sparse point clouds is a significant challenge. Real-world 3D data, such as that from LiDAR, is often characterized by missing information and non-uniform sampling. This sparsity makes it difficult for existing deep learning models, which are often trained on complete and clean synthetic data, to accurately infer the missing geometry and semantic details. While previous methods have shown success in upsampling or completing shapes, they often struggle to recover fine-grained object details, resulting in less realistic outputs. This paper aims to bridge this gap by learning robust shape priors that can be applied to both complete and partial point clouds for more accurate and detailed reconstruction.

Methods

The proposed method utilizes a two-stage approach:

  1. Pre-training an Auto-encoder: An auto-encoder is first trained on a dataset of complete point clouds. This auto-encoder learns to extract latent embeddings (features) that capture the essential shape information of complete objects. Specifically, two feature embeddings, $f_1$ (256-dim) and $f_2$ (1024-dim), are extracted. These learned features serve as strong shape priors.
  2. Completion Network with Feature Alignment: The pre-trained auto-encoder's decoder is then reloaded, and the entire network is trained end-to-end. The core innovation lies in the feature alignment strategies used to learn shape priors from both complete and incomplete point clouds simultaneously.
  3. Feature Matching Loss: This loss directly matches the latent representations of partial and complete point clouds using the L2 distance. This encourages the features extracted from incomplete data to align with those from complete data in the latent space.
  4. Adversarial Loss (MMD-GAN): To further refine the feature alignment, a Maximum Mean Discrepancy Generative Adversarial Network (MMD-GAN) is employed. This loss minimizes the statistical distribution distance between features in a Reproducing Kernel Hilbert Space, ensuring that the learned representations from partial data are statistically similar to those from complete data.

The method also incorporates a coarse-to-fine strategy for reconstruction. A coarse point cloud is first generated, and then the input partial points are concatenated with this coarse output. This combined representation is fed into a finer stage to reconstruct the dense and complete point cloud, preserving input details while adding missing information.

Datasets

  • ShapeNet: Used for training and evaluating the model on seen and unseen categories. The paper uses 30,974 objects from eight categories (airplane, cabinet, car, chair, lamp, sofa, table, vessel) for training and tests on eight additional unseen categories.
  • KITTI Dataset: A real-world dataset used to validate the model's generalization ability. The paper uses Velodyne scans for car objects.

Key Insights and Ideas

  • Learning Shape Priors from Both Complete and Incomplete Data: The key insight is that by explicitly learning shape priors from both complete and partial point clouds, the model can better infer missing geometry. This is achieved through feature alignment.
  • Feature Alignment is Crucial for Detail Recovery: Simply reconstructing a shape is not enough; recovering fine details is essential for realism. The feature matching and MMD-GAN losses are shown to be critical for achieving this.
  • MMD-GAN for Robust Feature Distribution Matching: Using MMD-GAN for adversarial loss is more effective than standard GANs (like LS-GAN) in aligning feature distributions, leading to better completion results.
  • Coarse-to-Fine for Detail Preservation: The coarse-to-fine approach, combined with concatenating partial and coarse outputs, effectively preserves input details while generating missing parts.

Key Results

  • State-of-the-Art Performance: The proposed method achieves state-of-the-art performance on the ShapeNet dataset, outperforming existing methods like PCN and TopNet in terms of Mean Chamfer Distance (CD). For seen categories, improvements of 11.8% and 14.1% on average were reported compared to PCN and TopNet, respectively.
  • Improved Detail Reconstruction: Qualitative results demonstrate that the method generates finer object details, such as the thin legs of chairs and airplane wings, which are often missed by other approaches.
  • Robustness and Generality: The model shows strong performance on unseen categories and on the real-world KITTI dataset, indicating good generalization capabilities. For the KITTI dataset, the proposed feature optimization strategy outperformed other approaches in terms of Fidelity Error (FD).
  • Quantitative Improvements: On a smaller training dataset, the method achieved significant relative improvements over PCN (up to 18.1%) and TopNet.

Limitations and Future Work

  • Computational Cost: While not explicitly stated as a limitation, the use of MMD-GAN and a multi-stage training process likely implies a higher computational cost compared to simpler methods.
  • Future Work: The paper suggests that the feature alignment strategies are beneficial and could be further explored. The authors also mention that they generate four resolutions of output, implying that exploring even higher resolutions or adaptive resolution generation could be a future direction. The robustness on real-world data suggests potential for applications in robotics and autonomous driving.

Summary updated

BibTeX
@inproceedings{wang2020point,
  author = {Xiaogang Wang and Marcelo H. Ang and Gim Hee Lee},
  title = {{Point Cloud Completion by Learning Shape Priors}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  doi = {10.1109/iros45743.2020.9340862},
}

Loading related content...

Figure