Abstract
Masked autoencoding and generative pretraining have achieved remarkable success in computer vision and natural language processing, and more recently, they have been extended to the point cloud domain. Nevertheless, existing point cloud models suffer from the issue of information leakage due to the pre-sampling of center points, which leads to trivial proxy tasks for the models. These approaches primarily focus on local feature reconstruction, limiting their ability to capture global patterns within point clouds. In this paper, we argue that the reduced difficulty of pretext tasks hampers the model's capacity to learn expressive representations. To address these limitations, we introduce a novel solution called the Differentiable Center Sampling Network (DCS-Net). It tackles the information leakage problem by incorporating both global feature reconstruction and local feature reconstruction as non-trivial proxy tasks, enabling simultaneous learning of both the global and local patterns within point cloud. Experimental results demonstrate that our method enhances the expressive capacity of existing point cloud models and effectively addresses the issue of information leakage.
Introduction: The Problem of Prior Shape Bias
In 3D computer vision, point clouds are the standard representation for spatial data. To train models without extensive labels, researchers often use self-supervised "proxy tasks," such as masking parts of an object and requiring the model to reconstruct them. However, a significant flaw exists in current methodologies: the use of Farthest Point Sampling (FPS) to select "center points" for these patches.
FPS is designed to maximize spatial coverage, effectively providing the model with a "skeleton" of the object's global shape before training begins. This phenomenon, termed Prior Shape Bias, makes the reconstruction task trivial. Consequently, models focus on local details but fail to learn the complex global structures necessary for high-level understanding. Furthermore, FPS is non-differentiable, preventing the model from optimizing its sampling strategy during training.
The DCS-Net Framework
To address these limitations, the Differentiable Center Sampling Network (DCS-Net) transforms sampling from a static pre-processing step into a learnable component. The architecture operates in three distinct stages:
- Canonical Sphere Mapping: Input point clouds are mapped onto a canonical sphere to align disparate objects into a unified coordinate space, ensuring the model recognizes semantic similarities across different categories.
- Differentiable Sampling: DCS-Net replaces hard-coded geometric rules with a learned probability map. By utilizing Gumbel-softmax relaxation, the sampling process becomes differentiable, allowing for end-to-end optimization.
- Dual Proxy Tasks: Unlike traditional models that only reconstruct local patches, DCS-Net introduces a Global Feature Reconstruction task. The network must reconstruct the center points themselves, forcing it to master global structure alongside local details.
Experimental Benchmarks
DCS-Net was evaluated across several rigorous standard benchmarks to demonstrate its versatility:
- ShapeNet: Large-scale pretraining across 55 categories.
- ScanObjectNN: Real-world scans featuring backgrounds and occlusions.
- ModelNet40: Synthetic 3D CAD models for classification.
- ShapeNetPart: Part segmentation (e.g., identifying wings vs. fuselage).
- S3DIS: Large-scale indoor semantic segmentation.
- ONCE: Autonomous driving dataset for real-world object detection.
Key Results and Insights
DCS-Net consistently enhances the performance of existing state-of-the-art architectures like Point-BERT, Point-MAE, and PointGPT.
- Classification Accuracy: On ScanObjectNN, integrating DCS-Net into Point-BERT increased accuracy from 87.4% to 88.1%. With PointGPT, it reached a state-of-the-art 92.3%.
- Robustness in Few-Shot Learning: Gains in low-data scenarios (e.g., 5-way 10-shot) indicate that the model learns more generalizable and robust features.
- Real-World Object Detection: On the ONCE dataset, PointRCNN equipped with DCS-Net improved mean Average Precision (mAP) from 28.74% to 32.90%, highlighting its utility for robotics and autonomous systems.
The primary insight is that sampling is a feature, not just a pre-process. By closing the information leak caused by FPS, DCS-Net forces the model to develop a more holistic understanding of 3D geometry through global-local synergy.
Conclusion
DCS-Net effectively mitigates prior shape bias by making center sampling differentiable and introducing non-trivial global proxy tasks. While the method requires a "warm-up" stage for the sampling network, its "plug-and-play" nature allows it to boost the performance of various point cloud models. Future research may extend this differentiable logic to streaming LiDAR data and more complex non-canonical spaces.
Summary updated