TGFormer: Transformer with Track Query Group for Multi-Object Tracking

Rui Zeng, Yuanzhou Huang, Songwei PeiAAAI 2025 · DOI

TL;DR

This paper introduces TGFormer, a novel multi-object tracking (MOT) approach that tackles the challenge of occlusions by using a "Track Query Group" for each target. Instead of a single query, TGFormer employs multiple queries per target, each specialized for different occlusion levels, leading to more robust tracking in complex scenarios.

What Problem Does This Paper Address?

Multi-object tracking (MOT) is a fundamental computer vision task with applications ranging from autonomous driving to surveillance. A major hurdle in MOT is accurately tracking objects that become occluded, change appearance, or are in crowded scenes. Traditional methods often struggle with these challenges, leading to identity switches (where a tracker mistakenly assigns a new ID to an already tracked object). While transformer-based trackers have shown promise by integrating detection and tracking, they typically rely on a single "track query" per object. This single query can become brittle when the target's appearance changes due to occlusion, making it difficult to maintain a consistent track. The paper argues that this single-query approach is inadequate for complex, real-world scenarios.

Methods

TGFormer introduces two core innovations:

  • Track Query Group (TG): Instead of a single track query per object, TGFormer maintains a "Track Query Group" for each tracked target. This group consists of up to three track queries, each designed to handle a specific occlusion level: fully exposed, slightly occluded, and heavily occluded. When a new object is detected, a TG is initialized with content and position embeddings. As the target's occlusion level changes, new queries are added to the group if the current occlusion level isn't represented. This allows the model to adapt by selecting the most suitable query for the current situation.
  • Track Query Group Updater & Position Predictor: To ensure long-term tracking and improve localization, TGFormer incorporates two key components:
  • Track Query Group Updater: This module updates the queries within a TG. It uses a Short-Term Memory Aggregator to fuse current frame features with existing queries. An Occlusion-Aware Memory-Attention mechanism then extracts and differentiates features specific to each occlusion level. Finally, a residual connection with long-term memory helps maintain stable general features.
  • Position Predictor: Recognizing that traditional Kalman filters are often overlooked in transformer-based MOT, TGFormer uses a lightweight LSTM network to predict the target's next frame position. This predicted position is combined with content embeddings to form the track query, aiding more accurate localization, especially for fast-moving targets.

Datasets

The paper evaluates TGFormer on two widely used MOT datasets:

  • MOT17: A benchmark with 7 training and 7 testing sequences, known for its crowded scenes.
  • MOT20: A more challenging benchmark with 4 training and 4 testing sequences, featuring even more complex and crowded scenarios.
  • DanceTrack: This dataset is highlighted for its individuals with similar appearances and more complex movements, posing a significant challenge for tracking.

Key Insights and Ideas

  • Occlusion-Aware Queries are Crucial: The core insight is that a single track query is insufficient for handling varying occlusion levels. By creating specialized queries for different occlusion states, the model can adapt more effectively to dynamic scene changes.
  • Leveraging Temporal Information for Prediction: The integration of an LSTM-based Position Predictor addresses a gap in transformer-based MOT, bringing back the benefits of explicit motion prediction that was common in older methods. This is particularly important for fast-moving objects.
  • End-to-End Training with Improved Robustness: TGFormer maintains the end-to-end training paradigm of transformer-based trackers while significantly improving robustness against identity switches, a common failure mode in MOT.

Key Results

TGFormer achieves competitive performance on both MOT17 and DanceTrack datasets.

  • MOT17: The paper reports strong results, with TGFormer outperforming similar transformer-based methods. For instance, it achieves a MOTA of 70.3 and an IDF1 of 67.1 on MOT20, showing significant improvements.
  • DanceTrack: TGFormer leads in most performance metrics compared to other transformer-based trackers on this challenging dataset, demonstrating its effectiveness in handling complex motion and similar appearances.
  • Ablation Studies: These studies confirm the effectiveness of the proposed modules. The Track Query Group Updater alone improved MOTA by 4.7%, and the addition of the Position Predictor further boosted MOTA by 0.6% with 10 input frames and 3 prediction frames.

Limitations and Future Work

  • Computational Cost: While TGFormer improves robustness, the introduction of multiple queries per target might increase computational complexity compared to single-query methods. The paper doesn't explicitly detail the speedup or slowdown.
  • Hyperparameter Tuning: The paper mentions confidence thresholds for occlusion levels (e.g., tupdate = 0.5, t_iou = 0.5) and weights for loss functions, suggesting that careful tuning is required for optimal performance.
  • Future Work: The authors suggest that their method's ability to adapt to different occlusion levels could be further explored. The integration of the Position Predictor also opens avenues for more sophisticated motion modeling in future MOT research.

Summary updated

BibTeX
@inproceedings{zeng2025tgformer,
  author = {Rui Zeng and Yuanzhou Huang and Songwei Pei},
  title = {{TGFormer: Transformer with Track Query Group for Multi-Object Tracking}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.33065},
}
Figure