Abstract
Feature learning for 3D object detection from point clouds is very challenging due to the irregularity of 3D point cloud data. In this paper, we propose Pointformer, a Transformer backbone designed for 3D point clouds to learn features effectively. Specifically, a Local Transformer module is employed to model interactions among points in a local region, which learns context-dependent region features at an object level. A Global Transformer is designed to learn context-aware representations at the scene level. To further capture the dependencies among multi-scale representations, we propose Local-Global Transformer to integrate local features with global features from higher resolution. In addition, we introduce an efficient coordinate refinement module to shift down-sampled points closer to object centroids, which improves object proposal generation. We use Pointformer as the backbone for state-of-the-art object detection models and demonstrate significant improvements over original models on both indoor and outdoor datasets.
Introduction
3D object detection is a cornerstone of autonomous driving and augmented reality, but processing raw point cloud data is notoriously difficult. Unlike 2D images, point clouds are irregular, sparse, and unordered. Current approaches generally fall into two categories: voxel-based methods, which gridify space into regular cubes but struggle with resolution trade-offs, and point-based methods, which often rely on simple symmetric functions like max-pooling that limit their ability to capture complex, context-dependent relationships.
Pointformer bridges this gap by utilizing the Transformer architecture's inherent ability to model long-range dependencies and its permutation-invariant nature to learn more expressive 3D features.
The Pointformer Architecture
Pointformer is a U-Net-like architecture composed of three primary Transformer modules designed to handle different scales of context:
- Local Transformer (LT): Focuses on object-level features by modeling interactions between points within a local neighborhood (defined by a ball query), allowing the network to learn detailed geometry.
- Global Transformer (GT): Operates at the scene-level, gathering information from across the entire point cloud to capture relationships between different objects. This is crucial for detecting objects with very few points or those that are partially occluded.
- Local-Global Transformer (LGT): A novel cross-attention module that integrates high-resolution local features with lower-resolution global context, ensuring well-coordinated multi-scale representations.
To manage the computational complexity of standard Transformers, the authors adopt a Linformer-style low-rank approximation, making the model feasible for large-scale point clouds.
Attention-Driven Coordinate Refinement
Beyond feature learning, Pointformer introduces a Coordinate Refinement module. This module uses the attention maps generated by the Transformer to "shift" down-sampled points closer to the actual centroids of objects. By averaging attention weights, the model learns which points are most geometrically significant—typically those near the center of an object—and adjusts the sampled keypoints accordingly. This significantly improves the quality of object proposals.
As shown in the visualization above, the attention mechanism effectively concentrates on the chairs within the scene. As the number of attended points increases (from Top-50 to Top-200), the model maintains a strong focus on the geometric structures, demonstrating the robustness of the attention mechanism in identifying object centers.
Experimental Results
The authors evaluated Pointformer on four major benchmarks, covering both indoor and outdoor environments: SUN RGB-D, ScanNet V2, KITTI, and nuScenes. Pointformer consistently outperformed baseline models when used as a drop-in backbone replacement:
- KITTI: On the "Car" category (Moderate difficulty), it improved PointRCNN by 1.3 AP.
- SUN RGB-D: It achieved a 61.1% mAP, a 2.0% boost over the standard VoteNet implementation.
- ScanNet V2: It reached 64.1% mAP, outperforming the baseline by 1.2%.
- Efficiency: By using the Linformer technique, inference latency was reduced from 0.25s to 0.22s with negligible loss in accuracy, proving that Transformers can be efficient for 3D tasks.
Conclusion
Pointformer demonstrates that Transformer-based backbones can significantly enhance 3D object detection by providing a more generalized and expressive way to model point interactions compared to traditional max-pooling. While the model handles clutter and sparse objects effectively, future work remains in optimizing the architecture for real-time deployment on edge devices and extending its capabilities to tasks like semantic segmentation and shape completion.
Summary updated