RetinaTrack: Online Single Stage Joint Detection and Tracking

Zhichao Lu, Vivek Rathod, Ronny Votel, Jonathan Huang

Introduction

Multi-Object Tracking (MOT) is a critical component for autonomous driving systems, traditionally handled via a "tracking-by-detection" paradigm. This approach involves a two-stage process: first, an object detector identifies instances in a frame, and then a separate association model links these detections across time. While effective, this separation introduces significant computational overhead and often fails to leverage shared features between the two tasks.

RetinaTrack addresses these inefficiencies by merging detection and tracking into a single-stage framework. By extending the RetinaNet architecture to produce instance-level embeddings, the model achieves high-performance tracking with minimal additional latency.

Architectural Innovation: The "Untied" Subnetwork

Standard single-stage detectors like RetinaNet share convolutional weights across all anchor boxes at a given spatial location. While efficient for detection, this "tied" approach struggles when multiple objects overlap, as the shared features lack the specificity needed to distinguish individual instances.

RetinaTrack introduces a modified head architecture to solve this "colliding anchor" problem. As shown in Figure 1, the model splits the post-Feature Pyramid Network (FPN) layers into task-shared ($m_1$) and task-specific ($m_2, m_3$) components.

Comparison of (a) FPN architecture, (b) standard RetinaNet heads, and (c) the proposed RetinaTrack head with task-shared and task-specific layers.

The key modifications include:

  • Task-Shared Layers: Separate parameters are used for each of the $K$ anchor shapes, allowing the model to learn distinct feature representations for different aspect ratios even if they share a center point.
  • Embedding Head: A third parallel subnetwork is added alongside the classification and box-regression heads. This head generates a 256-dimensional embedding for every anchor.
  • Joint Training: The model is optimized using a multi-task loss function combining Focal Loss (classification), Huber Loss (regression), and BatchHard Triplet Loss (embeddings). The triplet loss ensures that embeddings of the same object across frames are similar, while those of different objects are distinct.

Evaluation and Results

The authors evaluated RetinaTrack on the Waymo Open Dataset, a large-scale benchmark for autonomous driving. A key finding was that joint training is symbiotic: the inclusion of the tracking task actually improved detection mean Average Precision (mAP) compared to a detection-only baseline.

Ablation study comparing RetinaNet and RetinaTrack architectures across various configurations, highlighting the trade-off between accuracy and inference speed.

As detailed in the ablation study (Figure 2), RetinaTrack maintains competitive detection performance while enabling efficient tracking. On the Waymo dataset, it achieved a Multi-Object Tracking Accuracy (MOTA) of 39.19 and an mAP of 38.24. Crucially, the model is highly efficient, processing frames in approximately 70ms on an Nvidia V100 GPU. This makes it nearly 38 times faster than multi-stage baselines like Tracktor++, which requires over 2.6 seconds per frame.

Conclusion

RetinaTrack demonstrates that single-stage joint detection and tracking is not only feasible but superior for real-time applications. By "untieing" anchor features and learning instance-specific embeddings, the model effectively handles object occlusions and reduces identity switches. Future work may involve integrating temporal memory or multi-modal data (LiDAR/Radar) to further enhance robustness in complex urban environments.

Summary updated

BibTeX
@inproceedings{lu2020retinatrack,
  author = {Zhichao Lu and Vivek Rathod and Ronny Votel and Jonathan Huang},
  title = {{RetinaTrack: Online Single Stage Joint Detection and Tracking}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  doi = {10.1109/cvpr42600.2020.01468},
}
Figure