Unsupervised Point Cloud Pre-training via Occlusion Completion

Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, Matt J. Kusner

Abstract

We present Occlusion Completion (OcCo), a simple yet effective unsupervised pre-training approach for 3D point clouds. OcCo leverages the natural phenomenon of occlusion to learn robust geometric features. The process involves three steps: masking occluded points from a simulated camera view, training an encoder-decoder to reconstruct the full geometry, and transferring the encoder weights to downstream tasks. Our experiments demonstrate that OcCo significantly improves performance across various datasets and architectures, particularly in few-shot scenarios and real-world noisy environments.

Introduction

Deep learning's success in 3D vision is often hampered by the high cost and complexity of labeling point cloud data. Unlike 2D images, 3D scans can be sparse, massive, and frequently occluded, making manual annotation labor-intensive. While unlabeled 3D data is abundant from LiDAR and depth sensors, effective self-supervised tasks are needed to harness it. Previous methods like orientation prediction or 3D jigsaw puzzles often fail to preserve geometric integrity or generalize to real-world noise. OcCo addresses this by using occlusion—a fundamental aspect of 3D perception—as a supervisory signal.

Methodology: The OcCo Pipeline

The OcCo framework consists of a straightforward three-step pipeline designed to force the model to understand both global and local 3D structures.

The three-step OcCo pipeline: 1) Generate occlusions, 2) Train completion model, 3) Transfer weights.
  1. Generate Occluded Dataset: For a given point cloud (e.g., from ModelNet40), a camera viewpoint is simulated. Using a pinhole camera model and z-buffering via Delaunay triangulation, the method identifies visible points and masks those that would be occluded from that specific perspective.
  2. Train Encoder-Decoder Completion Model: An encoder (such as PointNet, DGCNN, or PCN) processes the partial, occluded cloud into a latent representation. A decoder then attempts to reconstruct the original full point cloud. The training objective is to minimize the Chamfer Distance (CD) between the reconstructed and ground-truth point clouds.
  3. Use Encoded Weights as Initialization: After pre-training, the decoder is discarded. The encoder weights, having learned to "fill in" missing geometry, serve as a powerful initialization for downstream tasks like classification or segmentation.

Experimental Evaluation

The models were pre-trained on the synthetic ModelNet40 dataset. Evaluation spanned a diverse range of downstream tasks and datasets:

  • Object Classification: Tested on ModelNet40, ScanNet10, and ScanObjectNN (real-world scans).
  • Segmentation: Evaluated on ShapeNetPart (part segmentation), S3DIS (indoor semantic segmentation), and SensatUrban (large-scale city scans).

Key Insights

Beyond raw performance, the authors investigated why OcCo works:

  • Architecture Agnostic: OcCo consistently improved performance across different backbones, including PointNet and DGCNN.
  • Loss Landscape Analysis: Visualization revealed that OcCo pre-training leads to "flatter" local minima during fine-tuning. In deep learning, flatter minima are typically associated with superior generalization.
  • Emergent Semantics: Interestingly, the pre-trained features showed high correlation with semantic parts. Specific neurons learned to activate for distinct components like chair legs or airplane wings, despite never seeing a label during pre-training.

Results and Performance

OcCo demonstrates substantial gains over training from scratch and previous self-supervised methods like Jigsaw and cTree.

Relative improvement of OcCo compared to other methods across few-shot learning, object recognition, and segmentation tasks.
  • Few-Shot Learning: In a "10-way 10-shot" setup on ModelNet40, OcCo-pre-trained PointNet reached 83.9% accuracy, a massive leap from the 46.6% achieved with random initialization.
  • Real-World Robustness: On the challenging ScanObjectNN dataset, OcCo improved DGCNN's accuracy to 83.9%, outperforming the Jigsaw baseline.
  • Large-Scale Segmentation: On SensatUrban, OcCo improved mIoU for categories like "cars," showing that object-level pre-training scales to city-level scene understanding.

Conclusion

Occlusion Completion provides a natural and effective self-supervised task for 3D point clouds. By learning to reconstruct missing geometry, models develop a robust spatial understanding that generalizes across architectures and datasets. OcCo's ability to significantly boost performance in data-scarce and real-world scenarios makes it a valuable tool for the 3D computer vision community.

Summary updated

Abstract

We describe a simple pre-training approach for point clouds. It works in three steps: 1. Mask all points occluded in a camera view; 2. Learn an encoder-decoder model to reconstruct the occluded points; 3. Use the encoder weights as initialisation for downstream point cloud tasks. We find that even when we pre-train on a single dataset (ModelNet40), this method improves accuracy across different datasets and encoders, on a wide range of downstream tasks. Specifically, we show that our method outperforms previous pre-training methods in object classification, and both part-based and semantic segmentation tasks. We study the pre-trained features and find that they lead to wide downstream minima, have high transformation invariance, and have activations that are highly correlated with part labels. Code and data are available at: https://github.com/hansen7/OcCo
BibTeX
@inproceedings{wang2021unsupervised,
  author = {Hanchen Wang and Qi Liu and Xiangyu Yue and Joan Lasenby and Matt J. Kusner},
  title = {{Unsupervised Point Cloud Pre-training via Occlusion Completion}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=egyfBwObMa},
}
Figure