RelationTrack: Relation-aware Multiple Object Tracking with Decoupled Representation

En Yu, Zhuoling Li, Shoudong Han, Hongwei Wang

Abstract: Existing online multiple object tracking (MOT) algorithms often consist of two subtasks, detection and re-identification (ReID). In order to enhance the inference speed and reduce the complexity, current methods commonly integrate these double subtasks into a unified framework. Nevertheless, detection and ReID demand diverse features. This issue would result in an optimization contradiction during the training procedure. With the target of alleviating this contradiction, we devise a module named Global Context Disentangling (GCD) that decouples the learned representation into detection-specific and ReID-specific embeddings. As such, this module provides an implicit manner to balance the different requirements of these two subtasks. Moreover, we observe that preceding MOT methods typically leverage local information to associate the detected targets and neglect to consider the global semantic relation. To resolve this restriction, we develop a module, referred to as Guided Transformer Encoder (GTE), by combining the powerful reasoning ability of Transformer encoder and deformable attention. Unlike previous works, GTE avoids analyzing all the pixels and only attends to capture the relation between query nodes and a few self-adaptively selected key samples. Therefore, it is computationally efficient. Extensive experiments have been conducted on the MOT16, MOT17 and MOT20 benchmarks to demonstrate the superiority of the proposed MOT framework, namely RelationTrack. The experimental results indicate that RelationTrack has surpassed preceding methods significantly and established a new state-of-the-art performance, e.g., IDF1 of 70.5% and MOTA of 67.2% on MOT20.

Introduction: The Optimization Contradiction

Modern multiple object tracking (MOT) frameworks often adopt a "joint detection and embedding" approach to balance speed and complexity. However, this integration introduces a fundamental optimization contradiction. Object detection requires features that emphasize intra-class similarity (e.g., identifying all pedestrians as the same category), whereas re-identification (ReID) requires features that maximize inter-instance differences (e.g., distinguishing between two different pedestrians). When these tasks share a single feature representation, they compete during training, leading to sub-optimal performance.

The RelationTrack Framework

To address these challenges, the authors propose RelationTrack, a framework designed to decouple task-specific features and leverage global semantic relations.

The RelationTrack architecture featuring the Global Context Disentangling (GCD) and Guided Transformer Encoder (GTE) modules.

Global Context Disentangling (GCD)

The GCD module acts as a feature splitter. It extracts a global context vector from the backbone features and uses it to transform the shared representation into two distinct streams: detection-specific and ReID-specific embeddings. This allows the network to satisfy the conflicting requirements of both tasks simultaneously without one compromising the other.

Visualization of feature maps showing how GCD decouples original features into detection-specific and ReID-specific representations.

Guided Transformer Encoder (GTE)

Beyond feature decoupling, RelationTrack addresses the limitation of local-only information. Most trackers focus solely on pixels within a bounding box, ignoring the global context that helps maintain identity. The Guided Transformer Encoder (GTE) combines a Transformer encoder with deformable attention. By self-adaptively selecting a small number of key sampling points rather than processing every pixel, GTE efficiently captures long-range dependencies between targets and their environment.

Experimental Results

RelationTrack was evaluated on the MOT16, MOT17, and MOT20 benchmarks. It established new state-of-the-art (SOTA) performance, particularly in crowded scenarios:

  • MOT20: Achieved an IDF1 of 70.5% and MOTA of 67.2%, significantly outperforming previous methods like FairMOT.
  • MOT17: Reached an IDF1 of 74.7%, a 2.4% improvement over FairMOTv2.

Ablation studies confirmed the effectiveness of the GCD module, which improved the IDF1 score on MOT17 from 73.3% to 74.9% by resolving the feature conflict.

Conclusion

RelationTrack successfully mitigates the optimization contradiction in joint MOT frameworks through its GCD module and enhances association accuracy via the GTE module. While the architecture introduces some computational overhead—running at approximately 7.4 FPS on MOT17—its significant gains in tracking accuracy and robustness to occlusions set a new benchmark for the field. Future work will likely focus on optimizing these modules for real-time deployment.

Summary updated

Abstract

Existing online multiple object tracking (MOT) algorithms often consist of two subtasks, detection and re-identification (ReID). In order to enhance the inference speed and reduce the complexity, current methods commonly integrate these double subtasks into a unified framework. Nevertheless, detection and ReID demand diverse features. This issue would result in an optimization contradiction during the training procedure. With the target of alleviating this contradiction, we devise a module named Global Context Disentangling (GCD) that decouples the learned representation into detection-specific and ReID-specific embeddings. As such, this module provides an implicit manner to balance the different requirements of these two subtasks. Moreover, we observe that preceding MOT methods typically leverage local information to associate the detected targets and neglect to consider the global semantic relation. To resolve this restriction, we develop a module, referred to as Guided Transformer Encoder (GTE), by combining the powerful reasoning ability of Transformer encoder and deformable attention. Unlike previous works, GTE avoids analyzing all the pixels and only attends to capture the relation between query nodes and a few self-adaptively selected key samples. Therefore, it is computationally efficient. Extensive experiments have been conducted on the MOT16, MOT17 and MOT20 benchmarks to demonstrate the superiority of the proposed MOT framework, namely RelationTrack. The experimental results indicate that RelationTrack has surpassed preceding methods significantly and established a new state-of-the-art performance, e.g., IDF1 of 70.5% and MOTA of 67.2% on MOT20.
BibTeX
@article{yu2023relationtrack,
  author = {En Yu and Zhuoling Li and Shoudong Han and Hongwei Wang},
  title = {{RelationTrack: Relation-aware Multiple Object Tracking with Decoupled Representation}},
  journal = {IEEE Trans. Multimedia},
  year = {2023},
  doi = {10.1109/tmm.2022.3150169},
}
Figure