3D Object Detection With Pointformer

Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li, Gao Huang

Abstract Feature learning for 3D object detection from point clouds is challenging due to the irregular nature of the data. We present Pointformer, a Transformer-based backbone designed to effectively learn features from 3D point clouds. Pointformer employs a hierarchical structure: a Local Transformer models interactions within local regions for object-level features, while a Global Transformer captures scene-level context. A Local-Global Transformer integrates these multi-scale representations. Additionally, a coordinate refinement module shifts down-sampled points toward object centroids to improve proposal generation. Pointformer achieves significant performance gains across major indoor and outdoor benchmarks when used as a backbone for state-of-the-art detection models.

Introduction

Detecting objects in 3D point clouds is a fundamental task in computer vision, yet it remains difficult because point cloud data is irregular, unordered, and sparse. Traditional approaches typically follow two paths: voxel-based methods, which regularize data into 3D grids at the cost of resolution or computation, and point-based methods, which process raw points but often rely on simple symmetric functions like max-pooling that fail to capture complex, long-range dependencies. Pointformer addresses these limitations by leveraging the Transformer architecture to learn context-dependent features directly from raw points without the information loss associated with voxelization.

Methodology

Pointformer introduces a U-Net-like architecture composed of several specialized Transformer modules designed to capture hierarchical context:

  • Local Transformer (LT): This module models interactions within local regions (defined by ball queries). It helps the network learn "object-level" features, such as the relationship between the components of a single object (e.g., the legs and back of a chair).
  • Global Transformer (GT): This module analyzes the entire scene to capture "scene-level" context, helping the model understand how different objects relate to one another spatially.
  • Local-Global Transformer (LGT): Acting as a bridge, this module uses a multi-scale cross-attention mechanism to integrate high-resolution local features with global context from lower resolutions.
  • Coordinate Refinement: Since LiDAR only captures object surfaces, sampled points are often far from an object's actual center. This module uses attention maps to "nudge" down-sampled points closer to the object's centroid, significantly improving the quality of object proposals.
  • Linear Complexity: To handle the massive number of points in a 3D scene, the authors adopt a "Linformer" approach, projecting keys and values into a lower-dimensional space to achieve linear complexity $O(n)$.

Results and Evaluation

Pointformer was evaluated by using it as a "drop-in" backbone replacement for state-of-the-art detectors like VoteNet and PointRCNN on four major datasets: SUN RGB-D and ScanNet V2 (indoor), and KITTI and nuScenes (outdoor).

On the KITTI dataset, Pointformer improved PointRCNN by 1.2% AP on the "Moderate" car category, with the most significant gains on "Hard" objects (small or occluded). In indoor settings, it boosted VoteNet by 2.0% mAP on SUN RGB-D. On complex categories like "dresser" or "bathtub," the improvement exceeded 5% AP.

Visualization of attention maps showing how the model focuses on object parts and scene context across different attention thresholds.

Qualitatively, the attention maps show the model "searching" logically: first looking at a chair back, then its legs, then other nearby chairs. This demonstrates the effectiveness of the hierarchical attention mechanism in resolving ambiguity in cluttered scenes.

Conclusion

Pointformer demonstrates that hierarchical attention mechanisms can effectively bridge the gap between local geometric precision and global scene understanding in 3D object detection. By providing a versatile, "plug-and-play" backbone, it offers a robust alternative to traditional voxel and point-based feature extractors. Future work may extend this architecture to other 3D tasks such as semantic segmentation or shape completion.

Summary updated

Abstract

Feature learning for 3D object detection from point clouds is very challenging due to the irregularity of 3D point cloud data. In this paper, we propose Pointformer, a Transformer backbone designed for 3D point clouds to learn features effectively. Specifically, a Local Transformer module is employed to model interactions among points in a local region, which learns context-dependent region features at an object level. A Global Transformer is designed to learn context-aware representations at the scene level. To further capture the dependencies among multi-scale representations, we propose Local-Global Transformer to integrate local features with global features from higher resolution. In addition, we introduce an efficient coordinate refinement module to shift down-sampled points closer to object centroids, which improves object proposal generation. We use Pointformer as the backbone for state-of-the-art object detection models and demonstrate significant improvements over original models on both indoor and outdoor datasets.
BibTeX
@inproceedings{pan2021d,
  author = {Xuran Pan and Zhuofan Xia and Shiji Song and Li Erran Li and Gao Huang},
  title = {{3D Object Detection With Pointformer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=-ZPjEmIfsX},
}
Figure