VOVTrack: Exploring the Potentiality in Videos for Open-Vocabulary Object Tracking

Zekun Qian, Ruize Han, Junhui Hou, Linqi Song, Wei FengSubmission to ICLR 2025 · Rejected

TL;DR

VOVTrack tackles open-vocabulary multi-object tracking (OVMOT) from a video-centric perspective rather than treating it as naive image detection plus tracking. By introducing tracking-state-aware prompt guidance (filtering out heavily occluded/blurred proposals) and a self-supervised object similarity learning framework trained on raw, unannotated video, VOVTrack achieves state-of-the-art results on the TAO benchmark without relying on millions of extra image annotations.

What Problem Does This Paper Address?

Traditional multi-object tracking (MOT) typically restricts itself to fixed closed sets of categories (usually people and vehicles). Open-vocabulary multi-object tracking (OVMOT) broadens this scope by requiring models to detect, classify, and track both pre-seen base classes and unseen novel classes across continuous video streams.

Existing OVMOT systems (like OVTrack) approach the problem with an image-centric mindset: they take an off-the-shelf open-vocabulary detector trained on static images and plug in a tracker trained on hallucinated or augmented image pairs. This approach has two fundamental shortcomings:

  1. Ignoring tracking-specific object degradation: Video frames frequently feature blur, partial occlusion, truncation, and rapid motion. Feeding damaged proposals directly into classification heads degrades feature representations.
  2. Severe video annotation bottleneck: Video tracking annotations are costly to acquire (TAO, for instance, has only 18.1k annotated frames labeled at 1 FPS). Prior methods largely ignore the vast quantities of raw, unlabeled video frames available.

Methods

VOVTrack builds upon a tracking-by-detection architecture using a ResNet-50 backbone, introducing two video-centric innovations:

                      [ Region Proposal (Faster R-CNN) ]
                                      │
            ┌─────────────────────────┴─────────────────────────┐
            ▼                                                   ▼
┌───────────────────────────────┐               ┌───────────────────────────────┐
│     Classification Head       │               │       Association Head        │
│ ∙ Text & Image branches       │               │ ∙ Feature Extraction          │
│ ∙ Prompt-guided attention     │               │ ∙ Long-Short Interval Sampling│
│   (occluded/blur awareness)   │               │ ∙ Category-Aware Clustering   │
│ ∙ Piecewise weight filtering  │               │ ∙ Intra- & Inter-Consistency  │
└───────────────────────────────┘               └───────────────────────────────┘
            │                                                   │
            └─────────────────────────┬─────────────────────────┘
                                      ▼
                      [ Matching & Trajectory Update ]
  1. Tracking-State-Aware Prompt Guided Attention:
  2. The authors define antonym prompt pairs representing tracking states (e.g., complete vs. incomplete, unoccluded vs. occluded, unobscured vs. obscured, recognizable vs. unrecognizable).
  3. Text embeddings of these state descriptors are obtained from a CLIP text encoder. Region candidates are assigned an attention weight $w_r \in [0, 1]$ indicating how "clean" or "favorable" the state is.
  4. A piecewise weighting strategy filters out low-quality candidates ($w_r < 0.3 \rightarrow 0$), retains medium-quality ones as-is ($0.3 \le w_r \le 0.6$), and awards high-quality candidates full weight ($w_r > 0.6 \rightarrow 1$).
  5. Self-Supervised Object Association via Consistency: Instead of requiring ID annotations, VOVTrack exploits cross-frame self-consistency in raw video:

  6. Category-Consistency Constraint: Candidate proposals are grouped using K-means clustering on their classification embeddings, ensuring similarity learning focuses on objects within the same semantic cluster.

  7. Long-Short-Interval Sampling: Concatenates adjacent frames (for small variations) and distant frames (for large visual variations) to train tracking under varying degrees of appearance shift.
  8. Intra-Consistency Loss ($\mathcal{L}_{\text{intra}}$): Enforces pair-wise symmetry (mapping forward from frame $t_i \rightarrow t_j$ and back should return to the original object) and triple-wise cyclicity ($t_i \rightarrow t_j \rightarrow t_k \rightarrow t_i$).
  9. Inter-Consistency Loss ($\mathcal{L}_{\text{inter}}$): Bridges appearance similarity with spatial proximity (spatial bounding-box IoU on adjacent frames) using binary cross-entropy.

Datasets

  • LVIS: Used for pre-training the base-class detector (99.3k base images).
  • TAO (Tracking Any Object): The primary benchmark containing 833 object categories.
  • Annotated subset (18.1k frames, 54.7k bounding boxes) used for benchmark evaluation.
  • Entire raw, unannotated dataset (534.1k frames, yielding ~399.9k self-supervised candidate boxes) used to train VOVTrack’s association head without ground-truth labels.
  • CC3M (Conceptual Captions 3M): A large external text-image dataset (3.3M pairs) used by competitor baselines (e.g., RegionCLIP) for comparison.

Key Insights and Ideas

  • Damaged samples hurt open-vocabulary generalization: Severely occluded or blurry object crops act as noisy distractors when learning alignment with vision-language models. Down-weighting or removing them during training substantially improves novel-class classification.
  • Unlabeled video is a goldmine for tracking: TAO contains ~30× more raw frames than labeled frames. Self-supervised temporal cycle-consistency exploits this unlabelled video effectively, removing the bottleneck of dense manual ID annotation.
  • Cross-task synergy in tracking: High-quality associations directly boost classification accuracy because tracks can pool class predictions via majority voting. Conversely, consistent tracking rescues low-confidence detections from being prematurely discarded.

Key Results

Evaluated on the TAO benchmark using TETA (Tracking-Every-Thing Accuracy, combining Localization LocA, Classification ClsA, and Association AssocA):

  • Validation Set:
  • Novel Classes: VOVTrack achieves 34.4% TETA vs. OVTrack’s 27.8% (+6.6% absolute gain), with novel classification accuracy (ClsA) jumping from 1.5% to 6.0% and association accuracy (AssocA) rising from 33.6% to 39.2%.
  • Base Classes: Reaches 38.1% TETA vs. OVTrack's 35.5%.
  • Test Set:
  • Achieves 29.4% novel TETA (vs. OVTrack’s 24.1%) and 37.0% base TETA (vs. 32.6%).
  • Comparison with Large-Scale Pretraining:
  • Outperforms OVTrack + RegionCLIP across overall TETA on both val (34.4% vs. 32.0%) and test sets (29.4% vs. 25.7%), despite RegionCLIP utilizing 3 million auxiliary image-text pairs from CC3M.

Limitations and Future Work

  • Heavy Occlusion Failures: Extreme occlusion still induces classification failures (misidentifying heavily occluded targets as unrelated background or objects).
  • Visual Distraction: Severe visual ambiguity or fast-moving targets with similar appearances can still lead to identity switches.
  • Two-Stage Overhead: While lighter than foundation-model pretraining pipelines, the training pipeline still relies on multi-step offline stages (detector pretraining, prompt tuning, clustering, self-supervised tracking training).

Summary updated

Abstract

Open-vocabulary multi-object tracking (OVMOT) represents a critical new challenge involving the detection and tracking of diverse object categories in videos, encompassing both seen categories (base classes) and unseen categories (novel classes). This issue amalgamates the complexities of open-vocabulary object detection (OVD) and multi-object tracking (MOT). Existing approaches to OVMOT often merge OVD and MOT methodologies as separate modules, predominantly focusing on the problem through an image-centric lens. In this paper, we propose OVTracker, a novel method that integrates object states relevant to MOT and video-centric training to address this challenge from a video object tracking standpoint. First, we consider the tracking-related state of the objects during tracking and propose a new prompt-guided attention mechanism for more accurate localization and classification (detection) of the time-varying objects. Subsequently, we leverage raw video data without annotations by formulating a self-supervised object similarity learning technique to facilitate temporal object association (tracking). Experimental results underscore that OVTracker outperforms existing methods, establishing itself as a state-of-the-art solution for open-vocabulary tracking tasks.
BibTeX
@misc{qian2024vovtrack,
  author = {Zekun Qian and Ruize Han and Junhui Hou and Linqi Song and Wei Feng},
  title = {{VOVTrack: Exploring the Potentiality in Videos for Open-Vocabulary Object Tracking}},
  year = {2024},
  eprint = {2410.08529},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.08529},
}
Figure