SeqTrack3D: Exploring Sequence Information for Robust 3D Point Cloud Tracking

Yu Lin, Zhiheng Li, Yubo Cui, Zheng Fang

Abstract

3D single object tracking (SOT) is a critical yet challenging task for autonomous driving and mobile robotics. Most existing methods perform tracking between two consecutive frames, often ignoring the motion patterns of the target over a series of frames. This limitation leads to significant performance degradation in scenarios with sparse point clouds. To address this, we introduce the Sequence-to-Sequence (Seq2Seq) tracking paradigm and a novel tracker named SeqTrack3D. By capturing target motion across continuous frames and combining historical point clouds with bounding box sequences, SeqTrack3D ensures robust tracking even in challenging environments. Extensive experiments on large-scale datasets demonstrate that SeqTrack3D achieves new state-of-the-art performance, improving success rates by 6.00% on NuScenes and 14.13% on the Waymo Open Dataset while maintaining real-time speeds.

Introduction

3D Single Object Tracking (SOT) is essential for the safe navigation of autonomous vehicles. The goal is to localize a target object in a continuous stream of LiDAR point clouds given its initial state. However, LiDAR data is inherently sparse, textureless, and prone to frequent occlusions.

Traditional trackers typically follow a "Two-to-One" paradigm (matching a template to the current frame) or a "Sequence-to-One" approach (using multiple past frames to predict a single current box). These methods often fail when the target becomes extremely sparse or temporarily invisible because they do not explicitly model the target's motion trajectory. SeqTrack3D breaks this limitation by leveraging historical bounding box (BBox) sequences as location priors, allowing the system to "infer" the target's position even when point cloud data is insufficient.

The SeqTrack3D Framework

SeqTrack3D implements a Sequence-to-Sequence tracking paradigm through a three-stage architecture designed to balance spatial geometry with temporal dynamics.

1. Coarse Box Prediction

The process begins with a lightweight PointNet-based backbone. This component generates an initial "coarse" bounding box for the current frame, establishing a spatial baseline that the subsequent Transformer layers can refine.

2. Decoupled Local-Global Feature Encoding

To prevent the network from becoming biased toward either static geometry or motion noise, SeqTrack3D utilizes a "divide-and-conquer" strategy for feature encoding:

  • Local Encoder: This branch focuses on the specific geometry and target location within each individual frame of the sequence.
  • Global Encoder: This branch concatenates features across the entire sequence to capture inter-frame motion dynamics and temporal consistency.

By decoupling these features, the model learns more robust representations that preserve the target's shape while understanding its movement patterns.

3. Box Sequence Decoder

The decoder treats the historical BBox sequence as a dynamic "location prior." These boxes act as queries to extract target-specific features from the point cloud sequence. Finally, the model outputs an updated sequence of BBoxes. During training, the model is required to predict the entire sequence, a constraint that forces the network to learn physically plausible motion trajectories.

Experimental Results

SeqTrack3D was evaluated on two major benchmarks: NuScenes, known for its sparse 32-beam LiDAR data, and the Waymo Open Dataset, which features high-quality 64-beam LiDAR data.

Performance Gains

SeqTrack3D achieved substantial improvements over previous state-of-the-art (SOTA) methods:

  • NuScenes: Outperformed STTracker by 6.00% in Success and 2.26% in Precision.
  • Waymo: Outperformed M$^2$-Track by 14.13% in Success and 8.38% in Precision.

Robustness to Sparsity

A key finding was the model's resilience to data sparsity. In test cases where the initial template contained only 0-15 points—a scenario where most trackers fail—SeqTrack3D maintained high performance. This is attributed to its ability to rely on the BBox sequence prior when the point cloud signal is weak.

Real-Time Efficiency

Despite the complexity of sequence-to-sequence modeling, SeqTrack3D remains highly efficient. It achieves a processing speed of 38 FPS on an NVIDIA RTX 4090, making it viable for real-time deployment in autonomous systems.

Discussion and Limitations

While SeqTrack3D represents a significant step forward, the authors identified a limit to the benefits of long-term memory. Performance improved when increasing the sequence length from 2 to 4 frames (1+3), but began to degrade at 8 frames (1+7). This degradation is likely due to error accumulation in historical BBoxes and the difficulty of simulating realistic noise for long sequences during training.

Conclusion

SeqTrack3D demonstrates that moving from frame-to-frame matching to a sequence-to-sequence paradigm is vital for robust 3D tracking. By integrating historical motion priors and decoupling spatial-temporal features, the tracker overcomes the inherent challenges of LiDAR sparsity, setting a new benchmark for the field.

Summary updated

Abstract

3D single object tracking (SOT) is an important and challenging task for the autonomous driving and mobile robotics. Most existing methods perform tracking between two consecutive frames while ignoring the motion patterns of the target over a series of frames, which would cause performance degradation in the scenes with sparse points. To break through this limitation, we introduce Sequence-to-Sequence tracking paradigm and a tracker named SeqTrack3D to capture target motion across continuous frames. Unlike previous methods that primarily adopted three strategies: matching two consecutive point clouds, predicting relative motion, or utilizing sequential point clouds to address feature degradation, our SeqTrack3D combines both historical point clouds and bounding box sequences. This novel method ensures robust tracking by leveraging location priors from historical boxes, even in scenes with sparse points. Extensive experiments conducted on large-scale datasets show that SeqTrack3D achieves new state-of-the-art performances, improving by 6.00% on NuScenes and 14.13% on Waymo dataset. The code will be made public at https://github.com/aron-lin/seqtrack3d.
BibTeX
@inproceedings{lin2024seqtrack,
  author = {Yu Lin and Zhiheng Li and Yubo Cui and Zheng Fang},
  title = {{SeqTrack3D: Exploring Sequence Information for Robust 3D Point Cloud Tracking}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=PrURfBwE3o},
}
Figure