Few-Shot 3D Point Cloud Semantic Segmentation via Stratified Class-Specific Attention Based Transformer Network

Canyu Zhang, Zhenyao Wu, Xinyi Wu, Ziyu Zhao, Song WangAAAI 2023 · DOI

TL;DR

This paper introduces a novel "Stratified Class-Specific Attention Transformer" (SCAT) for few-shot 3D point cloud semantic segmentation. SCAT addresses limitations of prior methods by effectively leveraging limited support data to segment new categories, achieving state-of-the-art performance with significantly reduced inference time.

What Problem Does This Paper Address?

3D point cloud semantic segmentation is crucial for understanding complex 3D environments, but existing supervised methods face two major hurdles: they require vast amounts of meticulously annotated data, which is costly and time-consuming to acquire, and they struggle to segment novel object categories not seen during training. Few-shot learning offers a solution by using a small number of labeled examples (support set) to guide segmentation of new categories. However, prior few-shot 3D segmentation methods, like AttMPTI, often rely on computationally expensive graph networks and can lose fine-grained information due to pooling operations when creating prototypes. This paper aims to bridge this gap by developing a more efficient and effective few-shot segmentation approach for 3D point clouds.

Methods

The core of the proposed method is a Stratified Transformer Architecture. Unlike previous approaches that might use a single prototype or a simple graph, SCAT employs a multi-layer transformer network to capture relationships between query and support point clouds at different scales.

Here's a breakdown of the key components:

  • Feature Extraction: A DGCNN (Dynamic Graph CNN) is used as an encoder to extract initial features from both support and query point clouds.
  • Stratified Transformer: The query point cloud is divided into smaller spatial segments (initially 8, then further divided into top/bottom halves). These segments are then processed by multiple transformer layers.
  • Class-Specific Attention: Crucially, the transformer layers use class-specific support features (masked support features) to condition the query features. This allows the model to focus on relevant semantic information from the support set for each query point. This is achieved by constructing Query (Q), Key (K), and Value (V) matrices where K and V are derived from the class-specific support features.
  • Multi-Scale Aggregation: By processing the query point cloud at different spatial resolutions (through the stratified splitting and multiple transformer layers), the model learns both coarse-grained and fine-grained relationships.
  • Skip Connections: Standard transformer skip connections are incorporated to help propagate features and reduce degradation.
  • Output: The aggregated features from the transformer layers are combined and passed through convolutional layers and a softmax function to produce the final segmentation prediction.

The method is designed to work in a one-shot setting (one support example per class) and can be extended to multi-shot and multi-way settings.

Datasets

The paper evaluates its method on two widely used 3D point cloud semantic segmentation benchmarks:

  • S3DIS (Stanford 3D Indoor Scene Dataset): Contains 12 semantic categories from large-scale indoor scenes across different buildings.
  • ScanNet: A dataset of 1,513 point cloud scans from 707 indoor scenes, offering 20 semantic categories and more varied room types, making segmentation more challenging due to irregular scene structures.

The authors follow standard practice by splitting categories into training and testing sets (Ctrain and Ctest) without overlap to ensure a true few-shot evaluation.

Key Insights and Ideas

  • Stratified Processing is Key: Dividing the query point cloud into spatial segments and processing them through multiple transformer layers allows the model to capture relationships at various scales, leading to more robust segmentation.
  • Class-Specific Attention is More Informative than Prototypes: By directly using masked support features (rather than pooled prototypes), the model retains richer, fine-grained semantic information from the support set, leading to better discrimination.
  • Transformer Efficiency for Point Clouds: Leveraging the self-attention mechanism of transformers, which excels at capturing long-range dependencies, proves effective for point cloud data without the computational overhead of graph construction in prior methods.
  • Efficiency Gains: The transformer-based approach significantly reduces inference time compared to graph-based methods, making it more practical for real-world applications.

Key Results

The proposed SCAT method achieves state-of-the-art performance on both S3DIS and ScanNet datasets for few-shot semantic segmentation.

  • S3DIS: Achieves around 3% improvement in mean-IoU for one-shot settings and 1% for five-shot settings compared to existing methods.
  • ScanNet: Shows improvements of around 2% in one-shot and 1.5% in five-shot settings.
  • Inference Time: The method reduces CPU inference time by approximately 15% compared to previous few-shot segmentation models like MPTI and AttMPTI, which rely on graph networks.

Qualitative results demonstrate SCAT's ability to avoid misclassifications seen in other methods, particularly for challenging object boundaries and smaller objects.

Limitations and Future Work

  • Computational Cost of Transformers: While more efficient than graph methods, transformers can still be computationally intensive, especially for very large point clouds.
  • Future Work: The paper suggests exploring further optimizations for transformer efficiency and potentially extending the approach to other 3D understanding tasks. The authors also highlight the effectiveness of their stratified architecture and class-specific attention mechanism, implying these could be valuable components in future research.

Summary updated

BibTeX
@inproceedings{zhang2023few,
  author = {Canyu Zhang and Zhenyao Wu and Xinyi Wu and Ziyu Zhao and Song Wang},
  title = {{Few-Shot 3D Point Cloud Semantic Segmentation via Stratified Class-Specific Attention Based Transformer Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25449},
}
Figure