DCS-Net: Pioneering Leakage-Free Point Cloud Pretraining Framework with Global Insights

Zhe Li, Zhangyang Gao, Cheng Tan, Stan Z. Li, Laurence T. Yang

Introduction: The Challenge of Prior Shape Bias

In 3D computer vision, self-supervised learning often relies on masking and reconstructing local patches of a point cloud. A critical step in this process is selecting "center points" to define these patches. Traditionally, Farthest Point Sampling (FPS) has been the standard choice due to its ability to cover the spatial extent of an object.

However, FPS introduces a significant "Prior Shape Bias." Because FPS is a fixed, non-differentiable algorithm that effectively outlines the object's skeleton, the model receives a "hint" about the global structure before learning begins. This "shape leakage" simplifies the proxy task, allowing the model to ignore global patterns and focus solely on local details.

Comparison of traditional FPS-based sampling versus the proposed DCS-Net method.

The DCS-Net Framework

To address these limitations, the Differentiable Center Sampling Network (DCS-Net) transforms center selection into a learnable, end-to-end process. By making the sampling differentiable, the framework forces the backbone to learn both global arrangements and local features.

The architecture operates in three distinct stages:

  1. Canonical Sphere Mapping: Input point clouds are mapped to a "canonical sphere." This step ensures semantic consistency, where similar parts of different objects (e.g., chair legs) are projected to the same spherical regions.
  2. Differentiable Sampling: Instead of geometric distance, DCS-Net generates a probability map over the sphere. Using Gumbel-softmax relaxation, it selects centers while maintaining gradient flow.
  3. Dual-Target Reconstruction: The model is tasked with reconstructing both the global center point arrangement and the local patch details, making the pretraining task significantly more challenging and representative.
The three-stage pipeline of DCS-Net, from canonical mapping to task-specific heads.

Experimental Evaluation

The effectiveness of DCS-Net was validated across diverse 3D tasks using datasets like ShapeNet (pretraining), ScanObjectNN (real-world classification), and ONCE (outdoor object detection).

Key Results

  • Classification Performance: Integrating DCS-Net into existing backbones like PointGPT and PointM2AE yielded consistent improvements on the ScanObjectNN benchmark, with PointM2AE's accuracy rising from 86.43% to 87.20%.
  • Robustness in Few-Shot Scenarios: The model demonstrated superior generalization in 10-shot and 20-shot learning tasks, indicating that the learned features are more semantically rich.
  • Large-Scale Object Detection: On the ONCE LiDAR dataset, DCS-Net boosted the mean Average Precision (mAP) of PointRCNN from 28.74% to 32.90%, highlighting its utility for autonomous driving.

Discussion and Future Directions

The core insight of DCS-Net is that "harder is better." By removing the "free" global information provided by FPS, the model is forced to develop a deeper understanding of 3D structures. Furthermore, the "plug-and-play" nature of the sampling module makes it highly compatible with various existing transformer backbones.

Despite its success, some challenges remain. The canonical mapping stage introduces a computational "warm-up" phase, and extending this object-centric approach to massive, unstructured outdoor scenes requires further research. Additionally, performance is sensitive to hyperparameters like network depth, necessitating careful tuning for new architectures.

Summary updated

Abstract

Masked autoencoding and generative pretraining have achieved remarkable success in computer vision and natural language processing, and more recently, they have been extended to the point cloud domain. Nevertheless, existing point cloud models suffer from the issue of information leakage due to the pre-sampling of center points, which leads to trivial proxy tasks for the models. These approaches primarily focus on local feature reconstruction, limiting their ability to capture global patterns within point clouds. In this paper, we argue that the reduced difficulty of pretext tasks hampers the model's capacity to learn expressive representations. To address these limitations, we introduce a novel solution called the Differentiable Center Sampling Network (DCS-Net). It tackles the information leakage problem by incorporating both global feature reconstruction and local feature reconstruction as non-trivial proxy tasks, enabling simultaneous learning of both the global and local patterns within point cloud. Experimental results demonstrate that our method enhances the expressive capacity of existing point cloud models and effectively addresses the issue of information leakage.
BibTeX
@misc{li2024dcs,
  author = {Zhe Li and Zhangyang Gao and Cheng Tan and Stan Z. Li and Laurence T. Yang},
  title = {{DCS-Net: Pioneering Leakage-Free Point Cloud Pretraining Framework with Global Insights}},
  year = {2024},
  eprint = {2402.02088},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.02088},
}
Figure