Rethinking the competition between detection and ReID in Multi-Object Tracking

Chao Liang, Zhipeng Zhang, Yi Lu, Xue Zhou, Bing Li, Xiyong Ye, Jianxiao Zou

Abstract

One-shot Multi-Object Tracking (MOT) models often suffer from performance degradation due to the inherent competition between object detection and re-identification (ReID) tasks. This paper introduces CSTrack, a novel tracker designed to mitigate this "tug-of-war" through a Reciprocal Network (REN) for task decoupling and a Scale-Aware Attention Network (SAAN) for scale-invariant identity embeddings. CSTrack achieves state-of-the-art results on standard benchmarks while maintaining real-time inference speeds, effectively bridging the gap between one-shot efficiency and two-stage accuracy.

Introduction: The One-Shot Dilemma

In Multi-Object Tracking, researchers typically choose between two paradigms: two-stage and one-shot. Two-stage methods detect objects first and then run a separate model for identity feature extraction (ReID). While accurate, they are computationally expensive. One-shot methods integrate both tasks into a single network for speed, but they historically struggle with accuracy.

The authors identify a fundamental conflict in one-shot architectures: Detection requires the network to recognize all pedestrians as similar (high intra-class similarity), whereas ReID requires the network to distinguish between individuals (high intra-class variance). When these tasks share a feature space, they compete, leading to "ambiguous learning."

Figure 1: Visualization of the competition between detection and ReID tasks. Detection similarity (left) seeks commonality across all humans, while ID embedding similarity (right) seeks discriminative features for individuals.

Methods: Decoupling and Scale Awareness

CSTrack addresses these issues through two primary architectural innovations built upon a YOLO-v5-style backbone:

  1. Reciprocal Network (REN): Instead of simply splitting the network head, REN utilizes "self-relation" and "cross-relation" layers. Self-relation allows each branch to specialize in task-specific features, while cross-relation enables the exchange of helpful common information without interference. This effectively decouples the competing objectives.
  2. Scale-Aware Attention Network (SAAN): Standard feature pyramids (FPNs) often assign different-sized objects to different layers, causing "semantic misalignment" for ReID. A person walking toward the camera might look different to the model simply because they moved to a higher-resolution layer. SAAN aggregates features from multiple resolutions using spatial and channel attention, ensuring that identity embeddings remain consistent regardless of the object's scale in the frame.
Figure 5: Comparison between a standard one-shot tracking head (b) and the proposed CSTrack architecture (c), featuring the Reciprocal Network (REN) and Scale-Aware Attention Network (SAAN).

Key Results

CSTrack was evaluated on the MOT16, MOT17, and MOT20 benchmarks, achieving state-of-the-art performance:

  • Accuracy: On MOT17, it achieved a 74.9 MOTA (Multiple Object Tracking Accuracy) and 72.3 IDF1 (Identification F1 score), outperforming previous leading models like FairMOT and JDE.
  • Efficiency: The standard model runs at 16.4 FPS, while a lightweight version (CSTrack-S) reaches 34.6 FPS, making it suitable for real-time deployment.
  • Discriminative Power: In upper-bound tests using ground-truth detections, CSTrack’s ReID features proved significantly more discriminative than prior one-shot methods, demonstrating a cleaner diagonal on cosine similarity matrices.

Conclusion

By rethinking the relationship between detection and ReID, CSTrack proves that one-shot trackers can achieve high accuracy without sacrificing speed. The introduction of REN and SAAN provides the necessary architectural "breathing room" for task specialization and scale invariance. Future work will focus on incorporating temporal information to improve tracking robustness against image blur and long-term occlusions.

Summary updated

Abstract

Due to balanced accuracy and speed, one-shot models which jointly learn detection and identification embeddings, have drawn great attention in multi-object tracking (MOT). However, the inherent differences and relations between detection and re-identification (ReID) are unconsciously overlooked because of treating them as two isolated tasks in the one-shot tracking paradigm. This leads to inferior performance compared with existing two-stage methods. In this paper, we first dissect the reasoning process for these two tasks, which reveals that the competition between them inevitably would destroy task-dependent representations learning. To tackle this problem, we propose a novel reciprocal network (REN) with a self-relation and cross-relation design so that to impel each branch to better learn task-dependent representations. The proposed model aims to alleviate the deleterious tasks competition, meanwhile improve the cooperation between detection and ReID. Furthermore, we introduce a scale-aware attention network (SAAN) that prevents semantic level misalignment to improve the association capability of ID embeddings. By integrating the two delicately designed networks into a one-shot online MOT system, we construct a strong MOT tracker, namely CSTrack. Our tracker achieves the state-of-the-art performance on MOT16, MOT17 and MOT20 datasets, without other bells and whistles. Moreover, CSTrack is efficient and runs at 16.4 FPS on a single modern GPU, and its lightweight version even runs at 34.6 FPS. The complete code has been released at https://github.com/JudasDie/SOTS.
BibTeX
@article{liang2022rethinking,
  author = {Chao Liang and Zhipeng Zhang and Yi Lu and Xue Zhou and Bing Li and Xiyong Ye and Jianxiao Zou},
  title = {{Rethinking the competition between detection and ReID in Multi-Object Tracking}},
  journal = {IEEE Trans. Image Process.},
  year = {2022},
  url = {https://openreview.net/forum?id=YLSFWPeRxg-},
}
Figure