GLT-T++: Global-Local Transformer for 3D Siamese Tracking with Ranking Loss

Jiahao Nie, Zhiwei He, Yuxiang Yang, Xudong Lv, Mingyu Gao, Jing ZhangarXiv 2023 · DOI · OpenReview

TL;DR

This paper introduces GLT-T and GLT-T++, novel Siamese trackers for 3D object tracking in point clouds. They improve proposal generation using a "global-local transformer" and enhance tracking accuracy by reframing the final selection as a ranking problem, achieving state-of-the-art results.

What Problem Does This Paper Address?

Tracking objects in 3D point clouds, particularly from LiDAR sensors, is crucial for applications like autonomous driving. Existing 3D Siamese trackers often rely on 3D region proposal networks (RPNs) that use deep Hough voting. However, these methods have limitations:

  1. Suboptimal Proposal Quality: Using a single seed point feature for voting isn't enough to generate high-quality 3D proposals.
  2. Equal Treatment of Seed Points: Seed points have varying importance based on their proximity to the object, but current methods treat them equally, leading to inaccurate offsets and proposals.
  3. Suboptimal Proposal Selection: The proposal with the highest score isn't always the most accurate, hindering tracking performance. This is due to issues like positive-negative sample imbalance and difficulty distinguishing targets.

Methods

The paper proposes two main innovations:

  1. Global-Local Transformer (GLT) for Voting:
    • Problem Addressed: Improving the quality of 3D proposals generated by the voting mechanism.
  2. Approach: A novel transformer-based voting scheme is introduced. It integrates "object-aware" and "patch-aware" geometric priors into seed point features.
  3. Global Transformer: Encodes object-aware information by considering global geometric shape priors derived from sparse sampling of neighboring points.
  4. Local Transformer: Enhances geometric feature representation by encoding local patch information using K-nearest neighbors.
  5. Importance Prediction Branch: A new branch is added to learn the potential importance weights of seed points. These weights are then used as loss constraints during training, encouraging the model to focus on more informative seed points.
  6. GLT-T: This is the first tracker that incorporates this GLT module into a 3D RPN for Siamese tracking.
  7. Ranking Loss for Proposal Selection:
    • Problem Addressed: Ensuring the most accurate proposal is selected as the tracking result, overcoming the limitations of score prediction.
  8. Approach: The score prediction task is reframed as a ranking problem. Two novel ranking losses are introduced:
  9. Target-Aware Ranking Loss: Addresses positive-negative sample imbalance by pairing foreground and background samples. It assigns higher weights to hard negative samples, effectively widening the prediction boundary.
  10. Localization-Aware Ranking Loss: Addresses the issue of ranking positive samples based on their accuracy (IoU). It pairs positive samples and encourages those with higher IoUs to rank higher.
  11. GLT-T++: This enhanced version of GLT-T integrates these ranking losses, improving tracking accuracy without increasing inference overhead.

Datasets

  • KITTI: Used for training and evaluation. It contains 21 sequences with LiDAR data. The authors split the training sequences into training, validation, and test sets.
  • nuScenes: A larger, more challenging dataset with 1,000 sequences and LiDAR data at a lower frequency (2 Hz). Used for further evaluation.

Key Insights and Ideas

  • Transformer for Geometric Priors: The paper demonstrates that transformers are highly effective at encoding both global object context and local patch details, which are crucial for generating accurate 3D proposals in point clouds.
  • Importance Weighting: Explicitly learning and using seed point importance weights significantly improves the voting process, making it more robust to noisy or less relevant points.
  • Ranking Over Scoring: Reframing the final proposal selection from a classification/scoring problem to a ranking problem is a key insight. This allows the model to learn relative relationships between proposals, leading to more reliable selection, especially in challenging scenarios with distractors or varying proposal quality.
  • No Inference Overhead for Ranking: The ranking losses are applied during training and do not add any computational cost during inference, making GLT-T++ a practical improvement over GLT-T.

Key Results

  • GLT-T: Outperforms existing state-of-the-art methods on KITTI and nuScenes datasets in terms of tracking accuracy (Success and Precision metrics).
  • GLT-T++: Further improves upon GLT-T, achieving significant gains (e.g., +3.6% and +3.9% average performance on KITTI).
  • Real-time Performance: GLT-T runs at 30 FPS on a single NVIDIA 1080Ti GPU, maintaining real-time capabilities. GLT-T++ has the same inference speed.
  • Robustness: The proposed methods show good robustness to sparse point clouds, a common issue in real-world LiDAR data.
  • Ablation Studies: Confirm the effectiveness of individual components: the global transformer, local transformer, and training strategy contribute to performance gains, as do the target-aware and localization-aware ranking losses.

Limitations and Future Work

  • Hyperparameter Sensitivity: The paper notes that optimal performance might require careful tuning of hyperparameters (like $\lambda$ and $\beta$ for ranking losses) for specific categories or datasets.
  • Template Generation: While GLT-T++ shows robustness across different template generation strategies, the "first ground truth & previous result" is the default. Exploring more advanced template generation could be a future direction.
  • Search Region Generation: The paper investigates different search region generation strategies, showing GLT-T++ performs well in both long-term and short-term tracking scenarios. Further research could explore more adaptive search region strategies.
  • Computational Cost: While real-time, the GLT module does introduce extra computation compared to some simpler methods, leading to a slower speed than certain baselines, though this is justified by significantly improved performance.

Summary updated

Abstract

Siamese trackers based on 3D region proposal network (RPN) have shown remarkable success with deep Hough voting. However, using a single seed point feature as the cue for voting fails to produce high-quality 3D proposals. Additionally, the equal treatment of seed points in the voting process, regardless of their significance, exacerbates this limitation. To address these challenges, we propose a novel transformer-based voting scheme to generate better proposals. Specifically, a global-local transformer (GLT) module is devised to integrate object- and patch-aware geometric priors into seed point features, resulting in robust and accurate cues for offset learning of seed points. To train the GLT module, we introduce an importance prediction branch that learns the potential importance weights of seed points as a training constraint. Incorporating this transformer-based voting scheme into 3D RPN, a novel Siamese method dubbed GLT-T is developed for 3D single object tracking on point clouds. Moreover, we identify that the highest-scored proposal in the Siamese paradigm may not be the most accurate proposal, which limits tracking performance. Towards this concern, we approach the binary score prediction task as a ranking problem, and design a target-aware ranking loss and a localization-aware ranking loss to produce accurate ranking of proposals. With the ranking losses, we further present GLT-T++, an enhanced version of GLT-T. Extensive experiments on multiple benchmarks demonstrate that our GLT-T and GLT-T++ outperform state-of-the-art methods in terms of tracking accuracy while maintaining a real-time inference speed. The source code will be made available at https://github.com/haooozi/GLT-T.
BibTeX
@misc{nie2023glt,
  author = {Jiahao Nie and Zhiwei He and Yuxiang Yang and Xudong Lv and Mingyu Gao and Jing Zhang},
  title = {{GLT-T++: Global-Local Transformer for 3D Siamese Tracking with Ranking Loss}},
  year = {2023},
  eprint = {2304.00242},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.00242},
}
Figure