Graph Transformer for 3D point clouds classification and semantic segmentation

Wei Zhou, Qian Wang, Weiwei Jin, Xinzhe Shi, Ying HeComput. Graph 2024 · DOI · OpenReview

TL;DR

This paper introduces GTNet, a novel deep learning architecture that fuses graph-based methods and Transformers to improve 3D point cloud understanding. By dynamically constructing graphs and employing both local and global attention mechanisms, GTNet achieves state-of-the-art results in shape classification, part segmentation, and semantic segmentation.

What Problem Does This Paper Address?

3D point clouds are rich in geometric information, but effectively capturing the complex relationships between points remains a challenge. Existing graph-based methods often rely on static graphs, limiting their adaptability, and use simple aggregation techniques that can overlook nuanced feature interactions. Transformer-based methods, while powerful for capturing global context, sometimes neglect crucial local feature extraction. This paper aims to bridge this gap by developing a model that leverages the strengths of both approaches to achieve more robust and accurate 3D point cloud analysis.

Methods

The core innovation is the Graph Transformer block, which combines two key modules:

  • Local Transformer: This module dynamically builds a graph for each layer, considering both coordinate and feature spaces. It employs an intra-domain cross-attention mechanism to learn weighted relationships between a centroid point and its neighbors, capturing fine-grained local features. A novel feature encoding step is introduced to enhance the perception of local shapes.
  • Global Transformer: This module utilizes a global self-attention mechanism to aggregate features from all centroids, capturing broader contextual information.

GTNet integrates these blocks within an Encoder-Decoder architecture. To preserve information during downsampling, all input points are treated as centroids. The model also incorporates a Coordinate Adjustment Network to improve invariance to rotation and translation. The dynamic graph structure is updated iteratively, allowing feature learning to adapt to evolving graph relations.

Datasets

The proposed GTNet is evaluated on three standard 3D point cloud benchmarks:

  • ModelNet40: For shape classification.
  • ShapeNet Part: For part segmentation.
  • S3DIS: For large-scale indoor scene semantic segmentation.

Key Insights and Ideas

  • Dynamic Graphs are Key: The paper strongly advocates for dynamic graph construction that adapts with each layer's feature update, offering greater flexibility than static graphs.
  • Fusion is More Than the Sum of Parts: Combining graph-based local feature extraction with Transformer-based global attention proves more effective than either approach alone.
  • Local Attention Matters: Explicitly modeling local relationships through cross-attention, rather than relying solely on global self-attention, significantly improves performance.
  • Feature Encoding Enhances Local Perception: The proposed feature encoding for edge relations helps the model better understand local shapes by considering both neighbor and centroid features.
  • No Downsampling Needed: Treating all points as centroids and avoiding downsampling helps retain more original input features, leading to better performance.

Key Results

GTNet achieves state-of-the-art results across the evaluated tasks:

  • Shape Classification (ModelNet40): Achieved 93.2% OA and 92.6% mAcc, outperforming previous methods.
  • Part Segmentation (ShapeNet Part): Achieved 85.1% mIoU, showing improvements over strong baselines like PointNet and PointNet++.
  • Semantic Segmentation (S3DIS): Achieved 64.3% mIoU, demonstrating its effectiveness in complex indoor scenes.

Ablation studies confirm the effectiveness of individual components, such as the Local Transformer, Global Transformer, feature encoding, and dynamic graph updates.

Limitations and Future Work

The authors acknowledge that the current model uses a unified K value for neighbors across all categories, which might not be optimal for objects with varying component counts and point densities. Future work includes designing more efficient models, exploring multi-scale neighbor selection, and extending GTNet to other 3D tasks like point cloud registration and 3D reconstruction.

Summary updated

Abstract

Highlights•We propose GTNet, a deep learning model fusing dynamic graph & Transformer.•We design Graph Transformer with 2 sub-structures for feature extraction.•We use feature encoding in Local Transformer for better performance.•We achieve good metrics on ModelNet40, ShapeNet, and S3DIS.
BibTeX
@inproceedings{zhou2024graph,
  author = {Wei Zhou and Qian Wang and Weiwei Jin and Xinzhe Shi and Ying He},
  title = {{Graph Transformer for 3D point clouds classification and semantic segmentation}},
  booktitle = {Comput. Graph},
  year = {2024},
  doi = {10.2139/ssrn.4839921},
}
Figure