Abstract
Multi-object tracking (MOT) has traditionally focused on estimating trajectories of all objects in a video, without selectively reasoning about user-specified targets under semantic instructions. In this work, we introduce a query-driven tracking paradigm that formulates tracking as a spatiotemporal reasoning problem conditioned on natural language queries. Given a reference frame, a video sequence, and a textual query, the goal is to localize and track only the target(s) specified in the query while maintaining temporal coherence and identity consistency. To support this setting, we construct RMOT26, a large-scale benchmark with grounded queries and sequence-level splits to prevent identity leakage and enable robust evaluation of generalization. We further present QTrack, an end-to-end vision-language model that integrates multimodal reasoning with tracking-oriented localization. Additionally, we introduce a Temporal Perception-Aware Policy Optimization strategy with structured rewards to encourage motion-aware reasoning. Extensive experiments demonstrate the effectiveness of our approach for reasoning-centric, language-guided tracking.
Introduction: Shifting the Tracking Paradigm
Traditional Multi-Object Tracking (MOT) operates on a "track everything" basis. Standard models detect and follow every instance of a predefined category, such as all pedestrians or all vehicles in a scene. While effective for general surveillance, this approach lacks the semantic depth required for complex real-world interactions where a user might specify a target through natural language: "Track the person in the red hoodie who just picked up a backpack."
Existing frameworks struggle with such queries because they lack semantic reasoning. They can answer "where" an object is but fail to determine "which" specific object matches a complex intent. Moreover, standard vision-language models (VLMs) often suffer from "identity drift," losing targets during occlusions because they rely on static appearance rather than temporal motion dynamics. QTrack addresses these gaps by treating tracking as a spatiotemporal reasoning problem.
Methodology
The QTrack framework integrates multimodal reasoning with tracking-oriented localization through two primary components.
Reasoning-Aware Tracking
Unlike traditional "detect-then-track" pipelines, QTrack utilizes a VLM to generate a Chain-of-Thought (CoT) reasoning trace. The model analyzes the query against the visual scene to identify targets before outputting structured bounding box trajectories. This "Why" component ensures the model maintains a logical link between the semantic query and the physical tracking output.
Temporal Perception-Aware Policy Optimization (TAPO)
To prevent the model from over-relying on static appearance cues—a common failure mode in VLMs—the authors introduced Temporal Perception-Aware Policy Optimization (TAPO). During reinforcement learning, the model is exposed to a "Temporal Corruption" strategy where frames in a sequence are frozen. A Temporal KL Loss penalizes the model if its predictions for moving sequences are identical to those for frozen sequences, forcing it to develop motion awareness.
The model is trained using Group Relative Policy Optimization (GRPO) with three structured rewards:
- Format Reward: Ensures adherence to the CoT and JSON output structure.
- Accuracy Reward: Based on standard Intersection over Union (IoU) for spatial overlap.
- Motion Consistency Precision (MCP): A novel metric evaluating whether the predicted trajectory's direction and speed align with the ground truth.
The RMOT26 Benchmark
To evaluate this new paradigm, the authors constructed RMOT26, a large-scale benchmark for query-driven tracking. It curates 1,000 instances from seven prominent MOT datasets (e.g., DanceTrack, MOT16/17/20). The benchmark includes challenging scenarios such as crowded scenes, partial/complete occlusions, and distractors with similar appearances. Crucially, data splits are performed at the sequence level to prevent identity leakage, ensuring the model generalizes its reasoning capabilities rather than memorizing specific instances.
Key Results and Insights
QTrack (3B) demonstrates that specialized motion supervision can outperform significantly larger models.
- Performance: On the RMOT26 benchmark, QTrack achieved an MCP of 0.30, a 5% absolute improvement over strong baselines. It also significantly reduced the Normalized Distance Error (NDE) to 0.39, compared to 0.82 for standard reasoning models.
- Motion vs. Appearance: The success of TAPO highlights that tracking is fundamentally different from repeated detection. By penalizing "static" reasoning, QTrack learns to utilize temporal dynamics.
- Interpretability: The use of
<think>tags makes the system's target selection more transparent, particularly in crowded scenes with high visual ambiguity.
Conclusion
QTrack introduces a significant shift in MOT by enabling selective, reasoning-centric tracking. By combining the RMOT26 benchmark with the TAPO optimization strategy, the authors provide a robust framework for language-guided tracking that maintains identity consistency through motion awareness. Future work may address spatial grounding sensitivity in initial frames and extend training to longer sequences to further enhance long-term persistence.
Summary updated