Abstract
Multi-Object Tracking (MOT) has traditionally relied on complex heuristics or end-to-end architectures that struggle with long-term dependencies or computational efficiency. This paper introduces PuTR (Pure Transformer), a novel framework that reimagines MOT as a sequence modeling task. By representing trajectories as Directed Acyclic Graphs (DAGs), PuTR maps the association problem directly onto a Transformer’s attention mechanism. Operating within a decoupled tracking-by-detection paradigm, PuTR achieves high-speed inference (up to 77 FPS) and exceptional domain adaptation, requiring only one hour of training on a single GPU.
Introduction
The "tracking-by-detection" paradigm—detecting objects first and then associating them across frames—remains dominant in MOT. However, existing methods often face a trade-off: heuristic-based trackers like SORT are fast but fail during long-term occlusions, while graph-based methods offer robustness but are typically "offline" and unsuitable for real-time applications.
Recent end-to-end Transformer-based trackers attempt to bridge this gap but often treat track propagation as a recurrent process, failing to fully leverage global self-attention for association. PuTR addresses this by asking whether a pure, online Transformer can handle both short- and long-term association by treating the sequence of detections as a unified modeling problem.
Methods: Tracking as Sequence Modeling
The fundamental innovation of PuTR is the observation that trajectory graphs are inherently Directed Acyclic Graphs (DAGs). Since objects move forward in time, the constraints of tracking—specifically the one-in, one-out degree for nodes—align naturally with the attention patterns of a Transformer.
Key Architectural Components
- Frame-Ordered Tokenization: Detections are converted into tokens using image patches and linear layers, arranged chronologically to form the input sequence.
- Frame Causal Masking: While standard Transformers use causal masks to prevent looking at future tokens, PuTR introduces a "frame causal mask." This prevents tokens within the same frame from attending to each other, ensuring the model remains invariant to the detection order within a single image.
- Dual Positional Encodings: To provide necessary context, the model employs absolute temporal encodings for frame identification and spatial encodings for 2D image localization.
- Decoupled Architecture: By separating association from detection (using a frozen detector like YOLOX), PuTR focuses exclusively on the association logic. This makes the model lightweight and facilitates rapid training.
Experimental Results
PuTR was evaluated across diverse benchmarks, including SportsMOT, DanceTrack, MOT17, and MOT20.
Performance and Efficiency
- Speed: The model achieves up to 77 FPS on sparse datasets and maintains 25 FPS in dense environments like MOT20.
- Accuracy: On DanceTrack, characterized by complex non-linear motion, PuTR achieved a 60.6 HOTA, significantly outperforming ByteTrack (47.7) and OC-SORT (55.1). On SportsMOT, extending the temporal window to 120 frames allowed PuTR to reach a state-of-the-art 76.8 HOTA.
- Domain Adaptation: PuTR demonstrates remarkable generalization. While competing models often see performance drops of 35-49% when transferred to new datasets, PuTR’s performance gap was limited to 1.7%.
Ablation Study: Masking and Shuffling
A key technical contribution is the frame causal mask. As shown in the ablation studies, this masking strategy is critical for maintaining high performance across different datasets.
Discussion and Conclusion
PuTR demonstrates that a pure Transformer architecture can effectively solve the MOT association problem with high efficiency and robustness. Its ability to learn the underlying logic of association rather than specific visual features allows for superior domain adaptation.
However, limitations remain. The model's reliance on appearance-based tokens makes it sensitive to very small objects where visual information is sparse. Additionally, the current version does not explicitly integrate motion cues like velocity, which could further resolve ambiguities during occlusions. Future work will explore integrating these cues and investigating the unique scaling laws of Transformer-based association in multi-object tracking.
Summary updated