Attention to Trajectory: Trajectory-Aware Open-Vocabulary Tracking

Yunhao Li, Yifan Jiao, Dan Meng, Heng Fan, Libo Zhang

Introduction

Multi-Object Tracking (MOT) has traditionally been confined to a narrow set of predefined categories. Open-Vocabulary MOT (OV-MOT) aims to overcome this by tracking arbitrary object classes, including those not encountered during training. However, existing OV-MOT frameworks primarily rely on frame-by-frame instance detection and classification. This approach is highly susceptible to errors caused by temporary occlusions, motion blur, or visual ambiguities in individual frames, leading to frequent identity switches and classification failures.

Motivation for trajectory-aware tracking: (a) Recovering association through trajectory history to prevent ID switches. (b) Improving classification accuracy by leveraging temporal context.

Methodology: The TRACT Framework

To address these challenges, we introduce TRACT (Trajectory-aware OV-Tracker). TRACT shifts the focus from isolated frame analysis to the temporal continuity of an object's entire trajectory. The framework consists of two primary components integrated into standard open-vocabulary detection pipelines.

The TRACT architecture, featuring Trajectory Consistency Reinforcement (TCR) for association and TraCLIP for trajectory-assisted classification.

Trajectory-Enhanced Association (TCR)

The Trajectory Consistency Reinforcement (TCR) module improves object association by maintaining a historical "feature bank" and "category bank" for each active trajectory. Instead of matching detections solely against the preceding frame, TCR compares current detections against the accumulated memory of the trajectory. This historical context allows the tracker to maintain identity and category consistency even when individual frame detections are suboptimal.

Trajectory-Assisted Classification (TraCLIP)

TraCLIP is a plug-and-play module designed to classify entire trajectories rather than single frames. It employs two main strategies:

  1. Trajectory Feature Aggregation (TFA): This sub-module samples visual features across the trajectory and fuses them using a self-attention mechanism. This creates a robust, multi-view visual representation that is resilient to transient noise or occlusions.
  2. Trajectory Semantic Enrichment (TSE): Recognizing that simple category names often lack sufficient semantic depth for vision-language models like CLIP, TSE utilizes a Large Language Model (LLM) to generate detailed visual attribute descriptions. These enriched descriptions provide more discriminative text embeddings for matching against visual trajectory features.

Key Insights

The core philosophy of TRACT is that trajectories provide essential context that single frames lack. A blurred object in one frame may have been clearly visible in previous frames; by aggregating these features, the model can "remember" the object's true identity. Furthermore, the transition from simple labels to descriptive semantic prompts via LLMs significantly enhances zero-shot generalization for novel categories, as it provides the model with visual cues (e.g., "massive throat pouch" for a pelican) rather than just a name.

Experimental Results

TRACT was evaluated on the OV-TAO benchmark, which tests zero-shot generalization by splitting categories into base (seen) and novel (unseen) classes. TRACT achieved state-of-the-art results across various detectors, including ViLD, RegionCLIP, and YOLO-World.

Using the YOLO-World detector, TRACT reached a Tracking-Every-Thing Accuracy (TETA) of 39.4% for base classes and 33.7% for novel classes on the validation set. Notably, the most significant improvements were observed in Classification Accuracy (ClsA), where TRACT outperformed the previous state-of-the-art method, MASA, by +4.6% for novel classes on the test set when using RegionCLIP. Ablation studies confirmed that the combination of TCR, TFA, and TSE is critical for optimal performance.

Conclusion and Future Work

TRACT demonstrates that leveraging temporal trajectory data is a powerful strategy for robust open-vocabulary tracking. While it significantly improves association and classification, challenges remain in bounding box localization (LocA). Future research may explore using trajectory appearance to refine localization. Additionally, addressing dataset limitations, such as incomplete annotations in the TAO dataset, remains a priority for the OV-MOT community to ensure more accurate evaluation of rare category detection.

Summary updated

BibTeX
@inproceedings{li2025attention,
  author = {Yunhao Li and Yifan Jiao and Dan Meng and Heng Fan and Libo Zhang},
  title = {{Attention to Trajectory: Trajectory-Aware Open-Vocabulary Tracking}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure