RelationTrack: Relation-Aware Multiple Object Tracking With Decoupled Representation

En Yu, Zhuoling Li, Shoudong Han, Hongwei Wang

TL;DR

RelationTrack introduces a novel multi-object tracking (MOT) framework that solves the "optimization contradiction" between object detection and re-identification (ReID). By decoupling shared features into task-specific embeddings and using a lightweight, Transformer-based global relation module, it achieves state-of-the-art performance on the challenging MOT16, MOT17, and MOT20 benchmarks.

What Problem Does This Paper Address?

In modern "joint detection and embedding" MOT frameworks, a single neural network is asked to do two things at once: locate objects (detection) and distinguish between different individuals (ReID).

The authors identify a fundamental optimization contradiction:

  1. Detection wants the network to emphasize similarities among all objects in the same category (e.g., "all these things are pedestrians").
  2. ReID wants the network to maximize the differences between individuals, even if they belong to the same category (e.g., "this is Person A, not Person B").

When these tasks share the same feature representation, they fight each other during training, leading to sub-optimal tracking. Furthermore, most existing trackers rely on local information within a bounding box, ignoring the global semantic relationships between objects and their background that could help maintain identity over time.

Methods

The authors propose two primary modules to bridge these gaps:

  1. Global Context Disentangling (GCD): This module acts as a "feature splitter." It takes the features from the backbone (DLA-34) and produces a global context vector. It then uses this vector to decouple the representation into two distinct streams: detection-specific and ReID-specific embeddings. This allows the network to satisfy the conflicting requirements of both tasks simultaneously.
  2. Guided Transformer Encoder (GTE): To capture global relations without the massive computational cost of standard Transformers, the authors combined a Transformer encoder with deformable attention. Instead of looking at every single pixel in an image (which is $O(N^2)$ complexity), GTE self-adaptively selects a small number of key sampling points. This allows the model to understand how an object relates to its surroundings and other targets efficiently.

The final architecture, RelationTrack, uses these decoupled features to predict heatmaps for object centers and extract discriminative embeddings for association via the Hungarian algorithm.

Datasets

The model was evaluated on the standard MOTChallenge benchmarks:

  • MOT16 & MOT17: Standard benchmarks for pedestrian tracking in various outdoor/indoor scenarios.
  • MOT20: A significantly more difficult dataset focused on extremely crowded environments (up to 220 pedestrians per frame).

For training, the authors followed standard practice by using a collection of "Extended Datasets," including CrowdHuman, ETH, CityPerson, CalTech, CUHK-SYSU, and PRW.

Key Insights and Ideas

  • The "Aha!" of Decoupling: The authors proved that simply adding more layers isn't enough; you must explicitly separate the feature space. Their ablation study showed that adding GCD improved the IDF1 score (a key tracking metric) from 73.3% to 74.9% on MOT17.
  • Context is King: By using GTE, the model learns that an object's identity is often defined by its relationship to the background and other objects, not just the pixels inside its own box.
  • Efficiency via Deformable Attention: Standard Transformers are often too slow for high-resolution video tasks. By using deformable attention, RelationTrack captures global "long-range" dependencies while remaining computationally viable.

Key Results

RelationTrack established new state-of-the-art (SOTA) results across the board:

  • MOT20: Achieved an IDF1 of 70.5% and MOTA of 67.2%, significantly outperforming previous leaders like FairMOT.
  • MOT17: Reached an IDF1 of 74.7%, a 2.4% improvement over FairMOTv2.
  • Robustness: Qualitative results show the model is exceptionally good at maintaining a target's identity even during partial occlusions where other top-tier trackers (like CSTrack) fail and cause "identity switches."

Limitations and Future Work

While RelationTrack is highly accurate, there is a trade-off in inference speed. On MOT17, RelationTrack runs at approximately 7.4 FPS, which is slower than some "real-time" competitors like FairMOT (25.9 FPS) or CenterTrack (22.0 FPS). The authors note that while GTE is more efficient than a standard Transformer, the overall complexity of the decoupled architecture currently limits its use in ultra-high-speed real-time applications. Future work likely involves further optimizing these modules for mobile or edge deployment.

Summary updated

Abstract

Existing online multiple object tracking (MOT) algorithms often consist of two subtasks, detection and re-identification (ReID). In order to enhance the inference speed and reduce the complexity, current methods commonly integrate these double subtasks into a unified framework. Nevertheless, detection and ReID demand diverse features. This issue results in an optimization contradiction during the training procedure. With the target of alleviating this contradiction, we devise a module named Global Context Disentangling (GCD) that decouples the learned representation into detection-specific and ReID-specific embeddings. As such, this module provides an implicit manner to balance the different requirements of these two subtasks. Moreover, we observe that preceding MOT methods typically leverage local information to associate the detected targets and neglect to consider the global semantic relation. To resolve this limitation, we develop a module, referred to as Guided Transformer Encoder (GTE), by combining the powerful reasoning ability of Transformer encoder and deformable attention. Unlike previous works, GTE avoids analyzing all the pixels and only attends to capture the relation between query nodes and a few self-adaptively selected key samples. Therefore, it is computationally efficient. Extensive experiments have been conducted on the MOT16, MOT17 and MOT20 benchmarks to demonstrate the superiority of the proposed MOT framework, namely RelationTrack. The experimental results indicate that RelationTrack has surpassed preceding methods significantly and established a new state-of-the-art performance, e.g., IDF1 of 70.5% and MOTA of 67.2% on MOT20.
BibTeX
@article{yu2023relationtrack,
  author = {En Yu and Zhuoling Li and Shoudong Han and Hongwei Wang},
  title = {{RelationTrack: Relation-Aware Multiple Object Tracking With Decoupled Representation}},
  journal = {IEEE Trans. Multimedia},
  year = {2023},
  doi = {10.1109/tmm.2022.3150169},
}

Loading related content...

Figure