Calibrating Track-Query Updates at Test Time for Multi-Object Tracking

Riku Inoue, Shogo Sato, Yu Mitsuzumi, Kazuhiko Murasaki, Tomoyasu Shimada, Takashi SanoarXiv 2026 · arXiv

TL;DR

Test-time Track-Query Calibration (TQC) is a lightweight, training-free "plug-in" for modern multi-object trackers that prevents identity switches during complex object interactions. By identifying and subtracting "leaking" feature components directed toward nearby objects at inference time, TQC significantly improves tracking consistency with a negligible latency overhead of only ~1ms per frame.

What Problem Does This Paper Address?

Multi-Object Tracking (MOT) is the art of following multiple targets (like pedestrians or players) through a video while keeping their unique IDs consistent. Modern state-of-the-art trackers often use track queries—high-dimensional vectors that represent a specific object's appearance and motion. These queries are "propagated" from one frame to the next and updated with new image features.

However, when objects overlap or cross paths, things get messy. The update process for one object might accidentally "grab" features from a nearby neighbor. This causes the query to drift in the representation space toward the neighbor's identity. Eventually, the tracker gets confused and swaps their IDs—a failure known as an Identity Switch (IDSW).

The authors set out to prove that these failures are preceded by measurable "leaks" in the query update and to fix them without the need for expensive retraining or complex architectural changes.

Figure 1. Standard query propagation and Test-time Track- Query Calibration (TQC). Left: A current-frame query update may contain a component directed toward a nearby active-track representation. TQC calibrates the tracker output before propa- gation using a bounded correction estimated from update compo- nents directed toward nearby active tracks. Right: In this example, standard query propagation leads to an identity switch at frame t + 1, whereas TQC preserves the identities.

Methods

The authors propose Test-time Track-Query Calibration (TQC). It is a gradient-free method that operates entirely during inference. The core idea is to "clean" the query update before it is propagated to the next frame.

The TQC pipeline follows three main steps:

  1. Nearby Track Retrieval: For a target track, it identifies the top-$K$ "neighbor" tracks using simple bounding-box Intersection-over-Union (IoU).
  2. Component Estimation: It calculates the direction from the target track to its neighbors in the query representation space. It then measures how much of the current frame's query update is pointing along these "distractor" directions.
  3. Bounded Calibration: It subtracts a portion of these distractor components from the query update. Crucially, this correction is bounded to ensure that TQC doesn't accidentally remove useful information needed for the object's own appearance adaptation.
Overview of the TQC process: (1) Find nearby tracks via IoU, (2) Estimate update components pointing toward them, and (3) Apply a bounded subtraction to "calibrate" the output.

Datasets

The method was rigorously evaluated across several challenging benchmarks:

  • DanceTrack: Known for objects with very similar appearances but complex, non-linear motion (dancers).
  • SportsMOT: Features fast-moving athletes in various sports settings.
  • MOT17 & MOT20: Standard pedestrian tracking benchmarks.
  • BDD100K: A large-scale autonomous driving dataset.

Key Insights and Ideas

  • Abruptness is a Red Flag: The authors found that track queries change significantly more "abruptly" (larger vector magnitude change) in the frames immediately preceding an identity switch.
  • Feature Leakage is Directional: It’s not just that the queries change a lot; they specifically move toward the representation of a nearby active track. This confirms the intuition that trackers "leak" identity information during close encounters.
  • Spatial Proximity is the Best Proxy: Surprisingly, simple bounding-box IoU was more effective at identifying the "counterpart" track involved in a future ID switch than more complex metrics like query cosine similarity. If two boxes overlap, their features are likely being mixed in the neural network's attention layers.
  • Training-Free Wins: Because TQC is purely geometric (operating on vectors in the latent space), it requires zero retraining. You can take a pre-trained model like MOTR or MeMOTR and see immediate gains just by adding these few lines of calibration code.

Key Results

TQC demonstrated consistent improvements across nine different tracker/dataset configurations:

  • Accuracy Boost: Achieved an average gain of +0.38 HOTA and +0.48 AssA (Association Accuracy). On "association-hard" sequences, HOTA gains reached up to +1.37.
  • IDSW Reduction: Reduced the total number of identity switches in 8 out of 9 test settings.
  • Efficiency: The computational cost is tiny. It adds at most 9.94 MFLOPs to models that already require hundreds of GFLOPs, resulting in a negligible slowdown of ~1%.
  • Qualitative Success: As shown in Figure 5, TQC successfully maintains identities in crowded dance and sports scenes where baseline models frequently swap IDs.
Qualitative examples showing TQC preserving IDs (bottom row) where the baseline fails (top row) in DanceTrack and SportsMOT.

Limitations and Future Work

While highly effective, TQC has a few notable limitations:

  • Continuing Tracks Only: TQC only calibrates tracks that are already active. It does not help with the initial detection of an object or recovering a track that has already been terminated.
  • Reliance on IoU: The method assumes that identity leakage is driven by spatial proximity. While this holds for most current MOT architectures, different types of "identity collisions" in the feature space might require different retrieval strategies.
  • Modest Gains in Some Settings: While the improvements are consistent, they are relatively small in scenarios where the baseline tracker is already highly robust.

Future work could explore more sophisticated ways to estimate the "correction direction" or integrate TQC-like logic directly into the training objective of query-based trackers.

Summary updated

Abstract

Multi-Object Tracking (MOT) requires localizing multiple objects and maintaining their identities over time. Recent query-propagation trackers carry target-specific representations across frames and update them with current-frame image features, yet identity association remains difficult when visually similar targets overlap or cross. During such interactions, current-frame query updates may contain components directed toward nearby active tracks, which may affect identity association in later frames. To investigate this issue, we analyze track-query updates preceding identity switches (IDSWs), where the same target is assigned a different track ID. Our analysis shows that (i) propagated track-query representations change more abruptly before IDSWs, (ii) current-frame updates have larger positive projections along directions toward other active tracks, and (iii) bounding-box intersection over union (IoU) retrieves the counterpart track involved in a subsequent IDSW with high recall when that track remains active before the switch. Motivated by these findings, we propose Test-time Track-Query Calibration (TQC), a training-free and gradient-free method for query-propagation trackers. TQC calibrates the tracker output before propagation using a bounded correction estimated from update components directed toward nearby active tracks. It requires only track-query representations and bounding boxes available during inference, without retraining or architectural modification. Across nine DanceTrack and SportsMOT test settings, TQC improves HOTA and AssA in all settings and IDF1 in eight, with an average HOTA gain of +0.38 and limited inference overhead. Controlled variants on six validation settings further show that TQC's correction direction yields more consistent HOTA improvements than alternative directions with matched correction norms.
BibTeX
@misc{inoue2026calibrating,
  author = {Riku Inoue and Shogo Sato and Yu Mitsuzumi and Kazuhiko Murasaki and Tomoyasu Shimada and Takashi Sano},
  title = {{Calibrating Track-Query Updates at Test Time for Multi-Object Tracking}},
  year = {2026},
  eprint = {2610.38494},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38494},
}
Figure