Few-Shot Point Cloud Semantic Segmentation via Contrastive Self-Supervision and Multi-Resolution Attention

Jiahui Wang, Haiyue Zhu, Haoren Guo, Abdullah Al Mamun, Cheng-Xiang Wang, Tong Heng Lee

This paper, recently appearing on arXiv, tackles a major bottleneck in 3D computer vision: the "class-specific bias" that plagues few-shot learning. By moving away from traditional supervised pretraining and embracing self-supervision, the authors present a more robust way to segment 3D point clouds with minimal data.

TL;DR

The authors propose a few-shot point cloud segmentation framework that replaces biased, fully-supervised pretraining with contrastive self-supervision using a learnable augmentor. Combined with a Multi-Resolution Attention (MRA) module that efficiently captures local and global features, the approach achieves state-of-the-art results and shows significant promise for niche, real-world industrial applications like CAD/CAM segmentation.

What Problem Does This Paper Address?

Semantic segmentation of 3D point clouds usually requires massive, hand-labeled datasets. Few-shot learning (FSL) aims to solve this by learning to identify new classes from just a handful of examples.

However, most FSL methods rely on a "pretrain-then-transfer" paradigm. They pretrain a feature extractor on a large labeled dataset (like S3DIS) using supervised learning. The authors argue this creates a feature extraction bias: the model learns features optimized for the pretrain classes, which don't necessarily generalize well to the novel classes encountered during few-shot tasks. Furthermore, labeling 3D data is notoriously difficult, making large-scale supervised pretraining impractical for many specialized industrial fields.

Methods

The authors introduce three key innovations to solve these issues:

  1. Contrastive Self-Supervision with a Learnable Augmentor: Instead of using labels during pretraining, the model learns by comparing different "views" of the same point cloud. They integrate a trainable augmentor (inspired by PointAugment) that automatically optimizes point-level and shape-level deformations. This forces the encoder to learn universal, point-wise features that are class-agnostic and thus more generalizable.
  2. Multi-Resolution Attention (MRA): Standard attention mechanisms in 3D are computationally expensive and often get "distracted" by redundant background points. The MRA module selectively attends to the $K$-nearest neighbors (for local geometric details) and the farthest points (for global shape context). This dual-focus approach captures essential features while ignoring distractive noise.
  3. Intra-Class Centering Multi-Prototype: In few-shot learning, classes are often represented by "prototypes" (mean feature vectors). To handle the inherent sparsity and complexity of point clouds, the authors use a centering loss to ensure that multiple prototypes for the same class are tightly clustered in the feature space, improving discrimination between similar-looking objects.

Datasets

The model was evaluated on three primary datasets:

  • S3DIS: A standard large-scale indoor scene dataset used for benchmarking.
  • ShapeNet: A dataset of 3D CAD models used primarily to train the learnable augmentor.
  • phcf (Practical CAD/CAM): A specialized dataset containing 3D machining features like "pockets," "holes," "chamfers," and "fillets." This represents a "real-world" small-data scenario where traditional supervised pretraining often fails due to overfitting.

Key Insights and Ideas

  • Self-Supervision > Supervision for FSL: The most striking insight is that class-agnostic features (learned without labels) are actually better for few-shot generalization than features learned via supervised classification. This is because the self-supervised model doesn't "overfit" its perception to a specific set of categories.
  • Selective Attention is Efficient: You don't need to look at every point to understand a scene. By focusing only on the "nearest" and "farthest" points, the MRA module achieves better performance with lower computational overhead than vanilla window-based attention.
  • Domain-Specific Pretraining: The authors found that self-supervised pretraining on the same domain (even without labels) is incredibly powerful. For the CAD/CAM dataset, using self-supervision on the target domain improved results by over 17% compared to supervised pretraining on a different domain.

Key Results

  • S3DIS Performance: The proposed method achieved 54.47% mIoU on 2-way 1-shot tasks, outperforming the previous state-of-the-art (attMPTI) which sat at 53.07%.
  • CAD/CAM Success: In the practical case study (phcf dataset), the model significantly outperformed baselines, proving its effectiveness in industrial settings where data is scarce and shapes are highly similar.
  • Ablation Studies: Removing the contrastive pretraining or the MRA module led to a consistent drop in accuracy, confirming that both the training strategy and the architecture are vital to the performance gains.

Limitations and Future Work

  • Training Speed: The authors acknowledge that their framework is slightly slower to train than some baselines due to the added complexity of feature concatenation in the MRA module.
  • Complexity: The pipeline involves several stages (learnable augmentation, contrastive pretraining, MRA, and graph-based label propagation), which might make it harder to implement or tune compared to simpler end-to-end models.
  • Future Directions: The authors suggest that this self-supervised approach could be extended to other data-efficient paradigms, such as incremental learning, where the model must learn new classes over time without forgetting old ones.

Reference: Wang, J., Zhu, H., et al. "Few-Shot Point Cloud Semantic Segmentation via Contrastive Self-Supervision and Multi-Resolution Attention." arXiv:2302.10501v1 [cs.CV].

Summary updated

Abstract

This paper presents an effective few-shot point cloud semantic segmentation approach for real-world applications. Existing few-shot segmentation methods on point cloud heavily rely on the fully-supervised pretrain with large annotated datasets, which causes the learned feature extraction bias to those pretrained classes. However, as the purpose of few-shot learning is to handle unknown/unseen classes, such class-specific feature extraction in pretrain is not ideal to generalize into new classes for few-shot learning. Moreover, point cloud datasets hardly have a large number of classes due to the annotation difficulty. To address these issues, we propose a contrastive self-supervision framework for few-shot learning pretrain, which aims to eliminate the feature extraction bias through class-agnostic contrastive supervision. Specifically, we implement a novel contrastive learning approach with a learnable augmentor for a 3D point cloud to achieve point-wise differentiation, so that to enhance the pretrain with managed overfitting through the self-supervision. Furthermore, we develop a multi-resolution attention module using both the nearest and farthest points to extract the local and global point information more effectively, and a center-concentrated multi-prototype is adopted to mitigate the intra-class sparsity. Comprehensive experiments are conducted to evaluate the proposed approach, which shows our approach achieves state-of-the-art performance. Moreover, a case study on practical CAM/CAD segmentation is presented to demonstrate the effectiveness of our approach for real-world applications.
BibTeX
@inproceedings{wang2023few,
  author = {Jiahui Wang and Haiyue Zhu and Haoren Guo and Abdullah Al Mamun and Cheng-Xiang Wang and Tong Heng Lee},
  title = {{Few-Shot Point Cloud Semantic Segmentation via Contrastive Self-Supervision and Multi-Resolution Attention}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=CQZOBoqbPx},
}

Loading related content...

Figure