Pillartrack: Boosting Pillar Representation for Transformer-Based 3d Single Object Tracking on Point Clouds

Weisheng Xu, Sifan Zhou, Jiaqi Xiong, Ziyu Zhao, Zhihang Yuan

TL;DR

This paper introduces PillarTrack, a novel framework for 3D single object tracking (SOT) on point clouds that addresses the information loss issues of traditional point-based methods. By employing a Pyramid-Encoded Pillar Feature Encoder (PE-PFE) and a modality-aware Transformer backbone, PillarTrack achieves competitive performance with improved efficiency and robustness on challenging datasets like KITTI and nuScenes.

What Problem Does This Paper Address?

3D single object tracking (SOT) is crucial for autonomous systems like robots and self-driving cars, enabling them to accurately follow objects in their environment using LiDAR data. Existing methods often rely on processing individual points, which requires re-sampling. This re-sampling step can lead to the loss or redundancy of valuable information, negatively impacting tracking accuracy and robustness, especially in dynamic scenarios. Furthermore, many current Transformer-based approaches, while powerful, are adapted from image processing and may not optimally leverage the unique geometric properties of point cloud data.

Methods

PillarTrack tackles these challenges with two main innovations:

  • Pyramid-Encoded Pillar Feature Encoder (PE-PFE): Instead of directly processing sparse point clouds, PillarTrack first converts them into dense "pillars." The PE-PFE then encodes the point coordinates within each pillar using a pyramid-like structure. This approach aims to minimize numerical discrepancies between different feature channels, making the network more stable and less sensitive to variations in scale, translation, and rotation. This contrasts with standard Pillar Feature Encoding (PFE) methods that can suffer from large value disparities.
  • Modality-Aware Transformer-based Backbone: The authors propose a Transformer backbone specifically designed for point cloud data. Recognizing that LiDAR inherently captures geometric information, they reallocate computational resources to the earlier stages of the backbone. This allows for more effective integration of the geometric semantics present in the raw point clouds, rather than relying on later stages to extract high-level features as is common in image-based Transformers. This design choice aims to improve performance while reducing computational overhead.

The overall framework then uses these enhanced features within a Siamese-like structure, combining a template and search area to compute similarity and predict the target's 3D bounding box.

Datasets

The paper evaluates PillarTrack on two widely used datasets for 3D object detection and tracking:

  • KITTI: A standard benchmark for autonomous driving, used for evaluating tracking performance.
  • nuScenes: A more challenging dataset with a larger number of scenes and more complex environments, also used for evaluating tracking performance.

Key Insights and Ideas

  • Pillarization is Key for Efficiency: Converting sparse point clouds into dense pillars is a crucial step for achieving real-time performance and enabling seamless integration with existing deep learning architectures.
  • Robustness through Pyramid Encoding: The PE-PFE's pyramid-like encoding effectively stabilizes feature representations against common transformations (scale, translation, rotation), a significant improvement over standard PFE methods.
  • Modality Matters for Backbones: Adapting Transformer architectures from image domains to point clouds requires careful consideration of the data's inherent properties. Reallocating computation to earlier stages of the backbone is more effective for point clouds than for images.
  • Early Stage Computation is Crucial: For point cloud processing, focusing computational power on the initial stages of a Transformer backbone is more beneficial for capturing geometric semantics than distributing it evenly or heavily in later stages.

Key Results

  • KITTI Performance: PillarTrack achieves a mean success rate of 66.8% and a precision of 83.7% on the KITTI dataset, outperforming the baseline SMAT by a significant margin (e.g., +11.4% success and +7.2% precision for mean-by-class). It also shows substantial improvements in specific categories like Truck (+19.6% success) and Trailer (+16.8% success).
  • nuScenes Performance: On the nuScenes dataset, PillarTrack demonstrates comparable performance, achieving the second-best result across five categories.
  • Speed: PillarTrack achieves 27 FPS on a single NVIDIA 3080 GPU, a notable improvement over the baseline SMAT's 18.2 FPS, without introducing extra computational parameters.
  • Robustness: Ablation studies show that PE-PFE maintains stable performance under scale, translation, and rotation disturbances, outperforming the standard PFE.

Limitations and Future Work

  • Car Category NuScenes: While strong overall, PillarTrack's performance in the "Car" category on nuScenes is slightly lower than PTTR++. The authors attribute this to PTTR++'s use of both point-based and pillar-based representations, but note that this hybrid approach might limit onboard device deployment.
  • Further Optimization: The authors suggest that PillarTrack can be further accelerated using quantization techniques, which they plan to explore in future work.
  • Broader Applications: The PE-PFE's effectiveness in enhancing robustness could be explored for other LiDAR-based tasks beyond object tracking.

Summary updated

Abstract

LiDAR-based 3D single object tracking (3D SOT) is a critical issue in robotics and autonomous driving. Existing 3D SOT methods typically adhere to a point-based processing pipeline, wherein the re-sampling operation invariably leads to either redundant or missing information, thereby impacting performance. To address these issues, we propose PillarTrack, a novel pillar-based 3D SOT framework. First, we transform sparse point clouds into dense pillars to preserve the local and global geometrics. Second, we propose a Pyramid-Encoded Pillar Feature Encoder (PE-PFE) design to enhance the robustness of pillar feature for translation/rotation/scale. Third, we present an efficient Transformer-based backbone from the perspective of modality differences. Finally, we construct our PillarTrack based on above designs. Extensive experiments show that our method achieves comparable performance on the KITTI and NuScenes datasets, significantly enhancing the performance of the baseline.
BibTeX
@misc{xu2024pillartrack,
  author = {Weisheng Xu and Sifan Zhou and Jiaqi Xiong and Ziyu Zhao and Zhihang Yuan},
  title = {{Pillartrack: Boosting Pillar Representation for Transformer-Based 3d Single Object Tracking on Point Clouds}},
  year = {2024},
  eprint = {2404.07495},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.07495},
}
Figure