Self-Supervised Few-Shot Learning on Point Clouds

Charu Sharma, Manohar Kaul

Deep learning on 3D point clouds has unlocked incredible potential for robotics, autonomous driving, and shape synthesis. However, it faces a massive bottleneck: the high cost of data annotation. Manually labeling 3D shapes is far more labor-intensive than annotating 2D images.

To combat this, researchers introduced a novel approach at NeurIPS 2020 that combines self-supervised learning with a clever geometric data structure. By training a model to understand the hierarchical and spatial relationships within a point cloud, they demonstrate how to make models significantly "smarter" even when they have very few labeled examples to learn from—a scenario known as few-shot learning (FSL).

The Challenge: Sample Complexity in 3D

Deep neural networks operating on labeled point clouds have shown promising results on supervised learning tasks like classification and segmentation. Unfortunately, the scarcity of labeled 3D data leads to a "sample complexity" problem: how do we train robust models when we only have a handful of examples?

Existing unsupervised methods attempt to bypass this by relying on Generative Adversarial Networks (GANs) or Auto-encoders. However, these can be computationally expensive. Other methods require converting irregular, continuous point clouds into regular grids (voxels), which inevitably leads to memory bloat and the loss of fine-grained geometric detail. This paper seeks a way to learn meaningful geometric features directly from raw point clouds without needing external labels or heavy generative architectures.

The Solution: Cover-Trees and Pretext Tasks

The core innovation of this paper is the use of a cover-tree to partition the point cloud. A cover-tree is a hierarchical data structure where each level represents the data at a different scale using "balls" (spherical regions) of varying radii. As you go deeper into the tree, the balls get smaller, providing a finer-grained representation of the 3D shape.

Visualization of a point cloud partitioned by spherical regions, representing a level in the cover-tree hierarchy.

Using this hierarchy, the authors designed two novel self-supervised "pretext tasks" to pre-train the network:

  1. Inter-ball Regression (Global): The model is given a pair of balls from the same level of the tree and must predict the $l_2$ distance between their centers. This forces the model to learn global spatial relationships across the point cloud.
  2. Intra-ball Classification (Local): For a parent ball and its child ball (from consecutive levels in the hierarchy), the model must predict which "quadrant" of the parent ball the child's center resides in. This teaches the model local geometric structures.

The Architecture Workflow

The proposed method operates in a distinct pipeline designed to maximize the utility of scarce data.

The proposed self-supervised learning architecture featuring the feature extractor and the two pretext tasks.
  • Feature Extractor: A shared backbone (such as PointNet or DGCNN) processes the raw, unordered points.
  • Self-Supervised Phase: The model trains exclusively on the "support set"—the few labeled examples available for the downstream task. However, it trains using the surrogate labels (distances and quadrants) generated automatically by the cover-tree, requiring no human annotation.
  • Downstream Phase: The pre-trained weights are then used to initialize a network for the actual supervised task, such as identifying if a point cloud is a "table" or segmenting the wings of an "airplane."

Key Insights

  • Self-Supervision as a Data Augmentor: By creating thousands of surrogate tasks from just a few physical samples, the model effectively "sees" the data much more intensely than it would in standard supervised learning.
  • The Power of Hierarchy: The cover-tree allows the model to learn multi-scale representations, understanding both the "big picture" (global distances) and the "fine details" (local quadrants) simultaneously.
  • Breaking the Voxel Dependency: This approach works directly on raw, unordered point clouds, avoiding the quantization artifacts and memory issues inherent to voxel-based methods.
  • Semantic Alignment: A fascinating byproduct of this pre-training is that semantically similar parts (like the two distinct wings of a plane) move closer together in the model's feature space, even if they are physically far apart in the actual 3D input.

Performance and Results

The authors evaluated their method across four major benchmarks: ModelNet40, Sydney Urban Objects, ShapeNet, and S3DIS. The method consistently outperformed both standard supervised models and previous unsupervised state-of-the-art methods in few-shot settings.

  • Classification: On the sparse, real-world LIDAR scans of the Sydney dataset (in a 10-way 20-shot setting), their method achieved 81.50% accuracy. This significantly beat the baseline DGCNN (76.10%) and the previous best unsupervised method, VoxelSSL (69.10%).
  • Segmentation: In part segmentation on ShapeNet, their "Our+PointNet" configuration reached a mean Intersection-over-Union (mIoU) of 36.81, a massive leap compared to just 27.23 for a standard PointNet.

An ablation study revealed that while the global regression task was slightly more impactful than the local classification task, combining both yielded the best overall results.

Limitations and Future Work

While highly effective, the method does have limitations. The "base of expansion constant" for the cover-tree is a critical hyperparameter; if the balls are too large, they overlap excessively, but if they are too small, they lack sufficient points. Finding the optimal constant currently requires a grid search. Additionally, building cover-trees for every object adds a computational preprocessing step to the pipeline.

Future work may look beyond spatial coordinates to incorporate spectral information (like eigenvalues of the point cloud) or more complex geometric properties, further enriching the learned embeddings and pushing the boundaries of what models can learn from limited 3D data.

Summary updated

Abstract

The increased availability of massive point clouds coupled with their utility in a wide variety of applications such as robotics, shape synthesis, and self-driving cars has attracted increased attention from both industry and academia. Recently, deep neural networks operating on labeled point clouds have shown promising results on supervised learning tasks like classification and segmentation. However, supervised learning leads to the cumbersome task of annotating the point clouds. To combat this problem, we propose two novel self-supervised pre-training tasks that encode a hierarchical partitioning of the point clouds using a cover-tree, where point cloud subsets lie within balls of varying radii at each level of the cover-tree. Furthermore, our self-supervised learning network is restricted to pre-train on the support set (comprising of scarce training examples) used to train the downstream network in a few-shot learning (FSL) setting. Finally, the fully-trained self-supervised network's point embeddings are input to the downstream task's network. We present a comprehensive empirical evaluation of our method on both downstream classification and segmentation tasks and show that supervised methods pre-trained with our self-supervised learning method significantly improve the accuracy of state-of-the-art methods. Additionally, our method also outperforms previous unsupervised methods in downstream classification tasks.
BibTeX
@inproceedings{sharma2020self,
  author = {Charu Sharma and Manohar Kaul},
  title = {{Self-Supervised Few-Shot Learning on Point Clouds}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=5isXt9Yyj3},
}
Figure