PatchFormer: An Efficient Point Transformer with Patch Attention

Cheng Zhang, Haocheng Wan, Xinyi Shen, Zizhao WuCVPR 2022 · DOI · arXiv

Introduction

Transformers have demonstrated remarkable success in natural language processing and 2D computer vision. However, their application to 3D point cloud learning is hindered by the quadratic computational complexity ($O(N^2)$) of standard self-attention mechanisms. As point cloud density increases, memory and processing requirements grow exponentially, making real-time analysis of large-scale 3D scenes nearly impossible. PatchFormer addresses this bottleneck by introducing a novel architecture designed for both efficiency and multi-scale feature extraction.

Methodology

PatchFormer is structured into a multi-stage pipeline that combines local feature extraction with global context modeling. The architecture consists of three primary stages, each utilizing Multi-Scale Attention (MST) and Patch Attention (PAT) blocks to process point features progressively.

The overall architecture of PatchFormer, illustrating the three-stage pipeline and the internal structure of the MST and PAT blocks.

Patch Attention (PAT)

The core innovation of PatchFormer is the Patch Attention (PAT) mechanism. To overcome the $O(N^2)$ complexity, PAT employs a low-rank approximation strategy. Instead of computing attention between every pair of points, the model identifies a small set of representative "patch bases" ($M$) from the input points ($N$).

The Patch Attention (PAT) mechanism, which uses learned bases to achieve linear complexity while maintaining global context.

As shown in the diagram above, the process involves:

  1. Base Estimation: Points are clustered into patches, and their features are aggregated to form $M$ bases ($M \ll N$).
  2. Linear Complexity: By calculating attention between the $N$ input points and the $M$ bases, the complexity is reduced to $O(MN)$, which is effectively linear. This allows the model to capture global geometric structures without the heavy computational cost of traditional self-attention.

Multi-Scale Attention (MST)

To complement the global view of PAT, the Multi-Scale Attention (MST) block focuses on local geometric details. It utilizes a voxel-based pipeline where points are mapped to a regular 3D grid. By employing multiple kernel sizes (e.g., 3x3x3, 5x5x5) simultaneously, the MST block can capture features at various granularities. This hybrid approach—using voxels for local multi-scale kernels and raw points for global attention—allows PatchFormer to maintain high precision across different object scales.

Experimental Results

PatchFormer was evaluated against several state-of-the-art models on industry-standard benchmarks, demonstrating a superior balance of speed and accuracy.

  • Efficiency: PatchFormer achieves a 9.2× speed-up over leading models like Point Transformer (PT). On the ModelNet40 dataset, it processes samples in just 34.3ms, compared to the 530.2ms required by PT2.
  • Classification and Segmentation: The model achieved 93.5% accuracy on ModelNet40 and 86.5% mIoU on ShapeNet Part. For large-scale indoor semantic segmentation (S3DIS), it reached 68.1% mIoU, outperforming significantly heavier architectures.
  • Model Size: With only 2.45M parameters, PatchFormer is highly lightweight, making it an ideal candidate for deployment on resource-constrained edge devices such as drones and mobile robots.

Conclusion

PatchFormer successfully bridges the gap between the high performance of Transformer architectures and the practical requirements of 3D point cloud processing. By leveraging Patch Attention to achieve linear complexity and Multi-Scale Attention for robust local feature extraction, the model provides a scalable and efficient solution for real-world 3D vision tasks.

Summary updated

BibTeX
@inproceedings{zhang2022patchformer,
  author = {Cheng Zhang and Haocheng Wan and Xinyi Shen and Zizhao Wu},
  title = {{PatchFormer: An Efficient Point Transformer with Patch Attention}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01150},
}
Figure