Abstract
There has recently been a flurry of exciting advances in deep learning models on point clouds. However, these advances have been hampered by the difficulty of creating labelled point cloud datasets: sparse point clouds often have unclear label identities for certain points, while dense point clouds are time-consuming to annotate. Inspired by mask-based pre-training in the natural language processing community, we propose a pre-training mechanism based point clouds completion. It works by masking occluded points that result from observations at different camera views. It then optimizes a completion model that learns how to reconstruct the occluded points, given the partial point cloud. In this way, our method learns a pre-trained representation that can identify the visual constraints inherently embedded in real-world point clouds. We call our method Occlusion Completion (OcCo). We demonstrate that OcCo learns representations that improve the semantic understandings as well as generalization on downstream tasks over prior methods, transfer to different datasets, reduce training time and improve label efficiency.
Introduction
Deep learning on 3D point clouds has advanced rapidly, yet it remains constrained by the labor-intensive nature of data annotation. Unlike 2D images, 3D point clouds are often sparse, ambiguous, and require significant time to label accurately at the point or bounding-box level. While vast amounts of unlabeled 3D data exist from sensors like LiDAR, leveraging this data effectively for pre-training remains a challenge.
Existing unsupervised methods, such as GANs or self-supervised tasks like voxel shuffling, often disrupt the underlying geometric structure of the objects. This paper introduces Occlusion Completion (OcCo), a pre-training mechanism that preserves geometric integrity by mimicking the natural process of 3D sensing. By learning to "see through" occlusions, the model develops a deep spatial and semantic understanding that significantly boosts performance on downstream tasks.
Methodology: Occlusion Completion (OcCo)
OcCo is inspired by mask-based pre-training in natural language processing (e.g., BERT). Instead of masking words, OcCo masks points that are occluded from a specific camera viewpoint. The process is divided into three main stages:
- Occlusion Generation: A full 3D point cloud is viewed from a random camera position. Using a z-buffering technique, points hidden from that viewpoint are removed, creating a "partial" point cloud.
- Completion Learning: An encoder-decoder model is trained to reconstruct the original full point cloud from the partial input. The model minimizes the Chamfer Distance (CD) between the predicted and ground-truth shapes, forcing the encoder to learn global structure and symmetry.
- Weight Transfer: Once trained, the decoder is discarded. The encoder—which has now learned rich geometric features—is used as a weight initialization for various downstream tasks.
OcCo is architecture-agnostic and has been successfully implemented with popular backbones like PointNet, PCN, and DGCNN.
Experimental Setup
The authors pre-trained the model on the synthetic ModelNet40 dataset. To evaluate generalization, the pre-trained weights were fine-tuned on diverse benchmarks:
- Classification: ModelNet40 (synthetic), ScanNet (real-world indoor), and ScanObjectNN (noisy real-world objects).
- Segmentation: S3DIS (large-scale indoor), ShapeNetPart (object parts), and SensatUrban (urban-scale).
- Few-shot Learning: Evaluated on ModelNet40 and Sydney10.
Results and Discussion
OcCo demonstrates significant performance gains across all evaluated tasks. In few-shot scenarios (10-way 10-shot), OcCo-initialized models achieved approximately 90% accuracy, a massive leap from the 31-52% seen with random initialization.
Key insights from the results include:
- Synthetic-to-Real Transfer: Pre-training on synthetic CAD models effectively transfers to messy, real-world data, suggesting the model learns universal geometric primitives.
- Optimization Benefits: Loss landscape visualizations reveal that OcCo leads to "flatter" minima, which are strongly associated with superior generalization in deep learning theory.
- Emergent Semantics: Visualization of the encoder's internal features shows that specific neurons learn to respond to semantic parts (e.g., chair legs or airplane wings) without any explicit supervision, purely from learning to complete shapes.
- State-of-the-Art Performance: OcCo consistently outperformed previous self-supervised methods like Jigsaw and cTree across almost all benchmarks, including a +7.9% mIoU boost on the S3DIS dataset using PointNet.
Conclusion
Occlusion Completion provides a simple, effective, and geometrically grounded approach to unsupervised 3D pre-training. By leveraging the inherent constraints of camera viewpoints, OcCo enables models to learn rich representations that improve accuracy, reduce training time, and enhance label efficiency. Future work may involve making the model viewpoint-aware or scaling the approach to even larger, more diverse datasets from autonomous driving platforms.
Summary updated