SeqTrack3D: Exploring Sequence Information for Robust 3D Point Cloud Tracking

Yu Lin, Zhiheng Li, Yubo Cui, Zheng Fang

TL;DR

SeqTrack3D introduces a "Sequence-to-Sequence" paradigm for 3D single object tracking, moving beyond simple frame-to-frame matching. By leveraging both historical point clouds and bounding box trajectories through a decoupled local-global Transformer encoder, it achieves new state-of-the-art performance—improving success rates by up to 14% on major datasets while maintaining real-time speeds.

What Problem Does This Paper Address?

3D Single Object Tracking (SOT) is critical for autonomous driving, yet it remains a massive challenge due to the nature of LiDAR data. Point clouds are often sparse, textureless, and prone to occlusions.

Most existing trackers follow a "Two-to-One" paradigm (matching a template to the current frame) or a "Sequence-to-One" approach (using multiple past frames to predict one current box). The authors argue these methods are insufficient because they ignore the motion patterns of the target. When a target becomes extremely sparse or invisible, its current location can still be inferred from its historical trajectory. The gap addressed here is the lack of a framework that explicitly models the relationship between sequences of points and sequences of bounding boxes (BBoxes) to maintain tracking through "blind" spots.

Methods

The core innovation is the Seq2Seq tracking paradigm, implemented via the SeqTrack3D tracker. Its architecture consists of three main stages:

  1. Coarse Box Prediction: A lightweight PointNet-based backbone generates an initial "coarse" bounding box for the current frame to establish a starting point for the sequence.
  2. Local-Global Feature Encoding: Instead of using a single Transformer to process everything, the authors use a "divide-and-conquer" strategy:
  3. Local Encoder: Focuses on the geometry and target location within each individual frame.
  4. Global Encoder: Concatenates features across the entire sequence to capture inter-frame motion dynamics. This decoupling prevents the network from being biased toward one type of feature during optimization.

  5. Box Sequence Decoder: This component takes the historical BBox sequence as a "location prior." It uses these boxes as queries to extract target-specific features from the point cloud sequence, ultimately outputting an updated sequence of BBoxes.

During training, the model is forced to predict the entire sequence of boxes, which acts as a constraint that compels the network to learn realistic motion physics.

Datasets

The model was evaluated on two of the largest and most respected benchmarks in the field:

  • NuScenes: Known for its difficulty due to a low 2Hz annotation frequency and 32-beam LiDAR, which results in sparser data.
  • Waymo Open Dataset: A high-quality dataset with 64-beam LiDAR and a 10Hz annotation frequency.

Key Insights and Ideas

  • BBoxes are Inputs, Not Just Outputs: Most trackers treat previous BBoxes as static history. SeqTrack3D treats them as dynamic "priors" that guide the attention mechanism, allowing the model to "look" where the target is likely to be based on its trajectory, even if the points there are sparse.
  • Decoupling Spatial and Temporal Cues: Spatial features (shape) and temporal features (motion) have distinct attributes. By processing them in separate encoder branches before merging, the model learns more robust representations that don't "wash out" the geometry with motion noise.
  • Sequence-Level Constraints: By predicting a sequence of boxes rather than just the current one, the loss function penalizes trajectories that don't make physical sense, leading to much smoother and more accurate tracking.

Key Results

SeqTrack3D achieved significant gains over previous State-of-the-Art (SOTA) methods:

  • NuScenes: Outperformed STTracker by 6.00% in Success and 2.26% in Precision.
  • Waymo: Outperformed M$^2$-Track by a staggering 14.13% in Success and 8.38% in Precision.
  • Robustness to Sparsity: In extreme cases where the initial template had only 0-15 points, SeqTrack3D maintained high performance while other SOTA methods failed significantly.
  • Efficiency: Despite the complex sequence modeling, the tracker runs at 38 FPS on an NVIDIA RTX 4090, making it suitable for real-time robotics applications.

Limitations and Future Work

The authors noted a "sweet spot" for sequence length. While increasing the time window from 1+1 to 1+3 frames improved results, performance began to degrade at 1+7 frames. They attribute this to two factors:

  1. Error Accumulation: Longer historical sequences contain more cumulative errors in the BBoxes.
  2. Simulation Difficulty: It becomes harder to simulate realistic "noisy" historical data for very long sequences during training.

Future work will likely focus on more sophisticated ways to capture spatial-temporal information in even longer sequences without succumbing to error drift.

Summary updated

Abstract

3D single object tracking (SOT) is an important and challenging task for the autonomous driving and mobile robotics. Most existing methods perform tracking between two consecutive frames while ignoring the motion patterns of the target over a series of frames, which would cause performance degradation in the scenes with sparse points. To break through this limitation, we introduce "Sequence-to-Sequence" tracking paradigm and a tracker named SeqTrack3D to capture target motion across continuous frames. Unlike previous methods that primarily adopted three strategies: matching two consecutive point clouds, predicting relative motion, or utilizing sequential point clouds to address feature degradation, our SeqTrack3D combines both historical point clouds and bounding box sequences. This novel method ensures robust tracking by leveraging location priors from historical boxes, even in scenes with sparse points. Extensive experiments conducted on large-scale datasets show that SeqTrack3D achieves new state-of-the-art performances, improving by 6.00% on NuScenes and 14.13% on Waymo dataset. The code will be made public at https://github.com/aron-lin/seqtrack3d.
BibTeX
@inproceedings{lin2024seqtrack,
  author = {Yu Lin and Zhiheng Li and Yubo Cui and Zheng Fang},
  title = {{SeqTrack3D: Exploring Sequence Information for Robust 3D Point Cloud Tracking}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=PrURfBwE3o},
}

Loading related content...

Figure