FastTrackTr: Real-Time Multiobject Tracking With Transformers for Real World

Pan Liao, Feng Yang, Di Wu, Jinwen Yu, Xingxin Li, Dingwen ZhangIEEE Trans. Industrial Informatics 2025 · DOI

TL;DR

This paper introduces FastTrackTr, a novel Transformer-based multi-object tracking (MOT) framework designed for real-time performance in practical industrial applications. By revisiting the Joint Detection and Tracking (JDT) paradigm and incorporating an efficient inter-frame information transfer mechanism, FastTrackTr achieves competitive accuracy while significantly improving inference speed, making it suitable for deployment on edge devices.

What Problem Does This Paper Address?

Transformer-based multi-object tracking (MOT) methods have shown impressive results, but their complex architectures and dynamic computation patterns often lead to slow inference speeds. This makes them difficult to deploy in real-world industrial scenarios where real-time performance is crucial. Existing methods either sacrifice accuracy for speed (e.g., purely kinematic approaches) or remain too slow despite efforts to optimize them. The challenge is to develop a Transformer-based MOT system that is both fast and accurate, bridging the gap between research prototypes and practical deployment.

Methods

FastTrackTr builds upon the Joint Detection and Tracking (JDT) paradigm, inspired by DETR's object queries which inherently encode appearance features. The core innovations include:

  • Efficient Inter-frame Information Transfer: Instead of relying on separate tracking queries or complex decoders, FastTrackTr uses a novel cross-decoder mechanism. This mechanism replaces the self-attention layer in a standard DETR decoder with a cross-attention layer that directly processes historical query features from previous frames. This reduces computational load and the number of queries needed.
  • Historical Encoder and Decoder: To further refine temporal information, a Historical Encoder processes past frame information, and a Historical Decoder integrates this refined historical context into the current frame's processing. This helps in understanding object trajectories over time.
  • Deterministic Architecture for Acceleration: The framework is designed with a fixed-shape architecture, which is crucial for seamless acceleration using tools like TensorRT. This contrasts with many other Transformer models that have dynamic computation graphs, making them difficult to optimize.
  • Unified Detection and ID Embedding: The model simultaneously predicts bounding boxes and discriminative appearance embeddings (ID embeddings) in a single forward pass. This is achieved through an auxiliary ID embedding head and a joint optimization strategy using a Circle Loss for metric learning, which prioritizes hard samples and improves cross-scene generalizability. The base detection component uses RT-DETR for its efficiency.

Datasets

The paper evaluates FastTrackTr on several standard multi-object tracking benchmarks:

  • DanceTrack [24]
  • SportsMOT [25]
  • MOT17 [26]
  • VisDrone2019 [27]
  • BDD100k [28]

Additionally, they conducted physical experiments on an autonomous vehicle platform.

Key Insights and Ideas

  • JDT Paradigm Revival: The paper demonstrates that the classic Joint Detection and Tracking (JDT) paradigm, when combined with Transformer architectures and efficient information transfer, can be a highly effective approach for real-time MOT.
  • Object Queries as Identity Embeddings: Leveraging DETR's object queries not just for detection but also as a basis for identity embeddings is a key insight, simplifying the architecture and improving feature representation.
  • Cross-Attention for Temporal Fusion: Replacing self-attention with cross-attention in the decoder for historical information processing is a clever way to integrate past context without significantly increasing computational complexity.
  • Deterministic Architecture is Key for Real-Time: The emphasis on a fixed-shape architecture for TensorRT acceleration highlights a critical practical consideration for deploying advanced models in real-world systems.
  • Circle Loss for Robust ID Embeddings: Using Circle Loss for ID embedding learning proves more effective than traditional triplet or one-hot encoding losses, leading to better tracking performance.

Key Results

  • Speed and Accuracy: FastTrackTr achieves state-of-the-art (SOTA) accuracy on several datasets while maintaining real-time inference speeds.
  • On an NVIDIA RTX 4090 with 1333x800 resolution, it achieves 86.6 FPS and 62.4 HOTA on DanceTrack, outperforming other Transformer-based methods.
  • With a 640x640 resolution, it reaches 243.4 FPS with 47.4 HOTA (using a ResNet18 backbone and 3-layer decoder).
  • Edge Deployment: The lightweight variant achieves up to 59.2 FPS on a NVIDIA Jetson AGX Orin with 640x640 images, meeting real-time requirements for industrial applications.
  • Ablation Studies: The paper shows progressive improvements with the addition of each component: Decoder (HOTA 51.6), Decoder + Encoder (HOTA 53.8), and Decoder + Encoder + Mask (HOTA 54.1) on DanceTrack validation.

Limitations and Future Work

  • Accuracy vs. MOTIP: The authors acknowledge that FastTrackTr does not surpass MOTIP in tracking accuracy. They attribute this to MOTIP's specialized network architecture for ID encoding and its sequence-optimized training paradigm, which captures longer-term temporal dependencies, albeit at the cost of slower inference.
  • Future Work: Future work will focus on further optimizing the model structure for more complex scenarios and exploring deployment on different hardware platforms. They aim to bridge the accuracy gap with methods like MOTIP while retaining FastTrackTr's speed advantages.

Summary updated

Abstract

Transformer-based multiobject tracking (MOT) methods have attracted significant attention from researchers. However, these Transformer-based models often suffer from suboptimal inference speeds due to their architectural complexities or other inherent issues, rendering them difficult to deploy in practical industrial applications. To address this challenge, we revisited the classic joint detection and tracking (JDT) paradigm and analyzed existing models. Drawing inspiration from Detection Transformer’s (DETR) object queries, which naturally encode object appearance features, we constructed a fast and novel JDT-type MOT framework named FastTrackTr by implementing an efficient interframe information transfer mechanism. This framework integrates three key technological innovations: a cross-decoder mechanism that implicitly incorporates historical trajectory information without requiring additional queries or decoders, a historical encoder and decoder pair for refining and utilizing historical feature representations, and a deterministic fixed-shape architecture that enables seamless TensorRT acceleration. Benefiting from these designs, our approach not only reduces the number of queries required for tracking but also avoids introducing excessive network structures, ensuring model simplicity while maintaining high accuracy. Experimental results show that our method achieves real-time tracking while maintaining state-of-the-art accuracy. On an NVIDIA RTX 4090 with an image size of 1333 × 800, it reaches 62.4 higher order tracking accuracy (HOTA) at 86.6 frames per second (FPS) on the DanceTrack dataset, outperforming other advanced Transformer-based methods. Furthermore, it excels on edge devices, such as the NVIDIA Jetson AGX Orin, where its lightweight variant achieves up to 59.2 FPS on 640 × 640 images, meeting real-time requirements for practical applications.
BibTeX
@article{liao2025fasttracktr,
  author = {Pan Liao and Feng Yang and Di Wu and Jinwen Yu and Xingxin Li and Dingwen Zhang},
  title = {{FastTrackTr: Real-Time Multiobject Tracking With Transformers for Real World}},
  journal = {IEEE Transactions on Industrial Informatics},
  year = {2025},
  doi = {10.1109/tii.2025.3631698},
}
Figure