TAQ-MOTR: Tracking-Aware Detection Query Formation for End-to-End Multiple Object Tracking

Hanyang Liu, Hong Zhang, Ding Yuan, Yang Han, Yifan YangarXiv 2026 · arXiv

TL;DR

TAQ-MOTR improves end-to-end multi-object tracking by making detection queries "tracking-aware" before they even enter the decoder. By fusing current image features with track and learnable memory, the model significantly improves object discovery and identity association, setting new state-of-the-art benchmarks on DanceTrack, SportsMOT, and BFT.

What Problem Does This Paper Address?

In the world of Multi-Object Tracking (MOT), the "Tracking-by-Propagation" (TBP) paradigm has gained traction for its end-to-end simplicity. Unlike traditional methods that detect objects first and link them later, TBP models (like MOTR) propagate "track queries" across frames to maintain identities.

However, these models face a subtle but critical challenge: Detection Query Blindness.

In TBP, detection queries are responsible for finding new objects or providing extra evidence for occluded ones. Most existing methods initialize these detection queries using static learnable embeddings or simple image-based proposals. They have no idea what is already being tracked. This lack of awareness leads to two main issues:

  1. Redundancy & Confusion: Detection queries might compete with existing track queries for the same object, leading to duplicate responses or identity switches.
  2. Supervision Imbalance: During training, track queries usually "claim" the ground-truth labels they are already tracking. This leaves detection queries with very few positive training signals, making the detector component of the model weak and prone to missing new targets.

Methods

The authors introduce TAQ-MOTR (Tracking-Aware Query MOTR), which features two primary innovations:

1. Hybrid Query Generator (HQG)

Instead of starting with "blind" queries, TAQ-MOTR uses the HQG to condition detection queries on three distinct sources of information before they reach the main decoder:

  • Image-based Proposals: Coarse spatial priors and semantic cues from the current frame.
  • Track Memory: Information from existing trajectories (where objects were, what they looked like, and how long they've been missing).
  • Learnable Memory: Dataset-level priors that help the model understand general object characteristics.

As shown in the comparison below, this allows the detection queries to focus specifically on regions that are not currently being tracked or regions where a track is struggling.

Figure 1: Detection query formation before decoding. (a) MOTR: learnable queries; (b) MOTRv2:

2. Auxiliary Detection Branch

To solve the "supervision imbalance," the authors add a training-only branch. This branch reuses the image proposals and HQG layers but is allowed to match against all ground-truth objects in a frame, regardless of whether they are already being tracked. This provides a much denser supervision signal, strengthening the model's ability to localize objects without disrupting the main tracking branch's identity-preserving logic.

Figure 2: Overall architecture of TAQ-MOTR. TAQ-MOTR forms tracking-aware detection queries

Datasets

The model was evaluated on three challenging benchmarks:

  • DanceTrack: Known for uniform appearances and extremely complex, overlapping motion.
  • SportsMOT: Features fast-moving athletes and small targets in wide-field views.
  • Bird Flock Tracking (BFT): A newer benchmark for non-human targets with rapid deformation and occlusion.

Key Insights and Ideas

  • Contextual Awareness Matters: The "aha moment" here is that detection in a video isn't just about finding objects; it's about finding what's missing. By informing the detector about the current state of the world (the track memory), the model avoids redundant work.
  • Decoupled Supervision: The auxiliary branch acts as a "bag of freebies." It improves the shared feature representations for detection during training but is completely removed during inference, meaning it adds zero computational cost to the final tracker.
  • Efficiency via Hybrid Encoding: By using a compact hybrid encoder (inspired by RT-DETR), TAQ-MOTR achieves high accuracy while remaining faster than many of its predecessors.

Key Results

TAQ-MOTR achieves impressive performance across the board, particularly in association metrics:

  • DanceTrack: Achieved 72.0 HOTA and 63.1 AssA, outperforming the previous best TBP method (MOTRv2) by a significant margin in association accuracy.
  • SportsMOT: Reached 72.3 HOTA, showing robustness to fast motion.
  • BFT: Set a new high bar with 73.4 HOTA.
  • Speed: The model runs at 41.1 FPS on an NVIDIA RTX 4090, making it one of the fastest high-performance end-to-end trackers available.

Limitations and Future Work

The authors identify a few areas for improvement:

  • Prior Reliability: The HQG currently treats all track memory as equally reliable. If a track "drifts" or becomes incorrect, it could negatively influence the detection queries. Future work could involve "gating" mechanisms to ignore unreliable priors.
  • Small Target Sensitivity: The use of stride-32 feature maps for proposals might limit the model's ability to find very tiny objects or handle extremely dense overlaps.
  • Automated Scheduling: The auxiliary branch's training schedule (when to turn it off) is currently manually calibrated. Automating this based on training progress could yield even better results.

Summary updated

Abstract

Tracking-by-propagation (TBP) methods perform end-to-end multi-object tracking by propagating track queries across frames, jointly handling detection and identity association without heuristic post-processing while naturally modeling temporal information. However, as detection and association are tightly coupled, detection queries in TBP are expected to discover currently untracked objects and provide complementary current-frame evidence for challenging tracks whose queries and identities are retained. Existing methods largely inherit detection-query formation from single-frame detection and construct such queries before the decoder with little awareness of trajectory states. Consequently, currently untracked objects may be missed, while challenging tracked objects with reduced visibility under occlusion may lack complementary evidence, causing association failures or duplicate responses. To address these issues, we propose TAQ-MOTR, a tracking-aware detection query formation framework that employs a pre-decoder Hybrid Query Generator (HQG) to fuse current-frame image features, track memory, and learnable memory. By conditioning detection queries on trajectory states, TAQ-MOTR adapts them to \rev{the discovery of currently untracked objects and tracked-object complementation We further introduce a training-only auxiliary detection branch whose one-to-one label assignment is decoupled from that of the main tracking branch, providing additional detection supervision to strengthen the shared HQG and decoder for the discovery of currently untracked objects. TAQ-MOTR improves performance on DanceTrack, SportsMOT, and BFT, achieving 72.0 HOTA, 63.1 AssA, and 76.6 IDF1 on the DanceTrack test set. Code is available at https://github.com/Magicmiya/TAQ-MOTR
BibTeX
@misc{liu2026taq,
  author = {Hanyang Liu and Hong Zhang and Ding Yuan and Yang Han and Yifan Yang},
  title = {{TAQ-MOTR: Tracking-Aware Detection Query Formation for End-to-End Multiple Object Tracking}},
  year = {2026},
  eprint = {2611.05205},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05205},
}
Figure