Abstract Feature learning for 3D object detection from point clouds is challenging due to the irregular nature of the data. We present Pointformer, a Transformer-based backbone designed to effectively learn features from 3D point clouds. Pointformer employs a hierarchical structure: a Local Transformer models interactions within local regions for object-level features, while a Global Transformer captures scene-level context. A Local-Global Transformer integrates these multi-scale representations. Additionally, a coordinate refinement module shifts down-sampled points toward object centroids to improve proposal generation. Pointformer achieves significant performance gains across major indoor and outdoor benchmarks when used as a backbone for state-of-the-art detection models.
Introduction
Detecting objects in 3D point clouds is a fundamental task in computer vision, yet it remains difficult because point cloud data is irregular, unordered, and sparse. Traditional approaches typically follow two paths: voxel-based methods, which regularize data into 3D grids at the cost of resolution or computation, and point-based methods, which process raw points but often rely on simple symmetric functions like max-pooling that fail to capture complex, long-range dependencies. Pointformer addresses these limitations by leveraging the Transformer architecture to learn context-dependent features directly from raw points without the information loss associated with voxelization.
Methodology
Pointformer introduces a U-Net-like architecture composed of several specialized Transformer modules designed to capture hierarchical context:
- Local Transformer (LT): This module models interactions within local regions (defined by ball queries). It helps the network learn "object-level" features, such as the relationship between the components of a single object (e.g., the legs and back of a chair).
- Global Transformer (GT): This module analyzes the entire scene to capture "scene-level" context, helping the model understand how different objects relate to one another spatially.
- Local-Global Transformer (LGT): Acting as a bridge, this module uses a multi-scale cross-attention mechanism to integrate high-resolution local features with global context from lower resolutions.
- Coordinate Refinement: Since LiDAR only captures object surfaces, sampled points are often far from an object's actual center. This module uses attention maps to "nudge" down-sampled points closer to the object's centroid, significantly improving the quality of object proposals.
- Linear Complexity: To handle the massive number of points in a 3D scene, the authors adopt a "Linformer" approach, projecting keys and values into a lower-dimensional space to achieve linear complexity $O(n)$.
Results and Evaluation
Pointformer was evaluated by using it as a "drop-in" backbone replacement for state-of-the-art detectors like VoteNet and PointRCNN on four major datasets: SUN RGB-D and ScanNet V2 (indoor), and KITTI and nuScenes (outdoor).
On the KITTI dataset, Pointformer improved PointRCNN by 1.2% AP on the "Moderate" car category, with the most significant gains on "Hard" objects (small or occluded). In indoor settings, it boosted VoteNet by 2.0% mAP on SUN RGB-D. On complex categories like "dresser" or "bathtub," the improvement exceeded 5% AP.
Qualitatively, the attention maps show the model "searching" logically: first looking at a chair back, then its legs, then other nearby chairs. This demonstrates the effectiveness of the hierarchical attention mechanism in resolving ambiguity in cluttered scenes.
Conclusion
Pointformer demonstrates that hierarchical attention mechanisms can effectively bridge the gap between local geometric precision and global scene understanding in 3D object detection. By providing a versatile, "plug-and-play" backbone, it offers a robust alternative to traditional voxel and point-based feature extractors. Future work may extend this architecture to other 3D tasks such as semantic segmentation or shape completion.
Summary updated