Abstract
We describe a simple pre-training approach for point clouds. It works in three steps:
- Mask all points occluded in a camera view;
- Learn an encoder-decoder model to reconstruct the occluded points;
- Use the encoder weights as initialisation for downstream point cloud tasks. We find that even when we construct a single pre-training dataset (from ModelNet40), this pre-training method improves accuracy across different datasets and encoders, on a wide range of downstream tasks. Specifically, we show that our method outperforms previous pre-training methods in object classification, and both part-based and semantic segmentation tasks. We study the pre-trained features and find that they lead to wide downstream minima, have high transformation invariance, and have activations that are highly correlated with part labels.
Introduction
Deep learning on 3D point clouds—data points representing the surfaces of objects or environments—is critical for applications ranging from autonomous driving to robotics. However, a significant bottleneck exists: the high cost of data labeling. Manually annotating millions of 3D points or defining 3D bounding boxes is far more labor-intensive and error-prone than labeling 2D images.
While massive repositories of unlabeled 3D data from LiDAR and depth cameras are available, effectively leveraging them for pre-training remains a challenge. Existing unsupervised methods often rely on pretext tasks like "Jigsaw" (shuffling voxels) or "cTree" (hierarchical partitioning), which can disrupt the inherent geometric structures the model needs to learn. This paper introduces Occlusion Completion (OcCo), a pre-training task designed to mimic natural sensor perception.
Methodology: Occlusion Completion (OcCo)
The proposed OcCo pipeline consists of three distinct stages designed to instill a deep geometric understanding in the model before it ever sees a labeled example.
- Generate Occlusions: Starting with a complete 3D point cloud, the system "views" it from a random camera angle. Using z-buffering, all points hidden from that specific viewpoint are removed. This creates a "partial" point cloud that realistically represents what a depth sensor would capture.
- The Completion Task: This partial cloud is fed into an encoder-decoder architecture. The encoder (e.g., PointNet, DGCNN) compresses the partial view into a latent vector, while the decoder (e.g., PCN) attempts to reconstruct the original, complete point cloud.
- Transfer Learning: After pre-training, the decoder is discarded. The encoder's weights, having learned to infer missing geometry, serve as the initialization for downstream tasks such as object recognition or semantic segmentation.
Experimental Setup
A key strength of OcCo is its generalization capability. The model was pre-trained on a single synthetic dataset, ModelNet40 (CAD models), and evaluated across a diverse set of benchmarks:
- Classification: ModelNet40, ScanNet (real-world indoor scans), and ScanObjectNN (noisy, occluded real-world objects).
- Segmentation: ShapeNetPart (object parts), S3DIS (indoor rooms), and SensatUrban (large-scale urban scans).
Results and Discussion
OcCo consistently outperforms random initialization and prior state-of-the-art unsupervised methods like Jigsaw and cTree across all tested tasks.
Performance Gains
- Few-Shot Learning: OcCo excels when data is scarce. In a "10-way 10-shot" scenario on ModelNet40, OcCo achieved approximately 90% accuracy, a massive leap from the ~52% achieved by models trained from scratch.
- Real-World Robustness: On the challenging ScanObjectNN dataset, OcCo improved classification accuracy by 4-6% over random initialization across various architectures.
- Semantic Segmentation: On the S3DIS dataset, OcCo-initialized models saw an improvement of nearly 8% in mIoU compared to random starts. Notably, pre-training on clean synthetic data (ModelNet40) provided significant boosts even for messy, urban-scale scans (SensatUrban).
Qualitative Insights
Beyond raw metrics, the authors identified several reasons for OcCo's success:
- Inductive Bias: By mimicking viewpoint-based occlusion, the model learns features that are inherently useful for real-world sensor data.
- Optimization Landscape: Visualization of the loss surface revealed that OcCo leads to "flatter" local minima, which is strongly associated with better generalization to unseen data.
- Emergent Semantics: Analysis of encoder activations showed that specific neurons learned to respond to semantic parts (e.g., chair legs, airplane wings) despite never receiving explicit part-level supervision during pre-training.
Conclusion
Occlusion Completion offers a simple, effective, and "natural" way to pre-train 3D encoders. By forcing models to "fill in the blanks" of occluded objects, OcCo develops robust geometric representations that generalize well from synthetic to real-world data and from small objects to entire urban scenes. Future work may involve making the completion model "occlusion-aware" by incorporating camera viewpoint information directly into the decoder.
Summary updated