Towards Grand Unification of Object Tracking

Bin Yan, Yi Jiang, Peize Sun, Dong Wang, Zehuan Yuan, Ping Luo, Huchuan Lu

Introduction

In computer vision, object tracking has traditionally been fragmented into four distinct sub-tasks: Single Object Tracking (SOT), Multiple Object Tracking (MOT), Video Object Segmentation (VOS), and Multi-Object Tracking and Segmentation (MOTS). Historically, these tasks have been treated as separate challenges requiring specialized architectures. SOT requires distinguishing a specific, often category-agnostic target from its background, while MOT focuses on detecting and associating multiple instances of specific categories (e.g., pedestrians or vehicles).

This fragmentation has led to a proliferation of redundant models and parameters that fail to generalize across the broader spectrum of tracking. The "Unicorn" framework proposes a grand unification, solving all four tasks using a single network with the exact same parameters, bridging the gap between category-agnostic and category-specific tracking.

The Unicorn Architecture

Unicorn is built upon a simplified, elegant architecture consisting of three primary components designed for multi-task versatility:

1. Unified Inputs and Backbone

Traditional SOT trackers often rely on cropping a small "search region" around a target to simplify the search space. In contrast, Unicorn processes the full image for all tasks. It utilizes a weight-shared backbone (such as ConvNeXt or ResNet) to extract feature pyramids from both a reference frame and the current frame, providing a global context that is often missing in specialized trackers.

2. Unified Embedding: The Correspondence Engine

The heart of the model is a pixel-wise correspondence map. This engine computes a massive similarity matrix between all pixels in the reference and current frames.

  • For SOT and VOS: This map is used to propagate the target's location or mask from the first frame to the current one.
  • For MOT and MOTS: The model extracts instance-level embeddings from this same correspondence map to associate multiple detections over time.

3. Unified Head: The Target Prior

To manage the transition between different tracking tasks, Unicorn introduces a "Target Prior" mechanism within the detection head:

  • SOT/VOS Mode: The prior is the propagated map of the specific target, instructing the head to focus exclusively on that object.
  • MOT/MOTS Mode: The prior is set to zero, allowing the head to function as a standard multi-class detector for all objects of known categories.

Key Insights and Innovations

Unicorn’s success stems from several critical realizations regarding the nature of tracking:

  • Global Search Robustness: The authors found that local search crops are not strictly necessary for SOT. By using a strong Target Prior and global pixel-wise correspondence, the model can locate targets anywhere in the frame, making it significantly more robust to fast motion or temporary occlusions.
  • Mathematical Unification: The research demonstrates that the instance-level matching required for MOT is essentially a sub-matrix of the dense pixel-wise correspondence used for SOT. This mathematical relationship is what enables the "Grand Unification" within a single embedding space.
  • Zero-Prior Versatility: The "Zero-Prior" trick allows the same detection head to seamlessly switch from tracking a user-defined object to detecting all objects of a specific class without changing network weights.

Experimental Results

Unicorn was evaluated across eight major benchmarks, demonstrating that a unified model can outperform specialized SOTA (State-of-the-Art) architectures:

  • Single Object Tracking: Achieved a 68.5% Success rate on LaSOT, surpassing specialized models like STARK.
  • Multiple Object Tracking: Reached 77.2 MOTA on the MOT17 test set.
  • Video Object Segmentation: Set a new high bar for box-initialized VOS on DAVIS16 with an 87.4 J&F score.
  • Efficiency: The standard version runs at 14 FPS, while a "Unicorn-RT" (Real-Time) variant maintains competitive performance at over 20 FPS.

Limitations and Future Directions

Despite its success, Unicorn faces challenges in VOS when compared to models that utilize high-quality mask initialization (like STCN), as Unicorn primarily relies on box-based initialization. Additionally, the current training process requires two stages—joint SOT-MOT training followed by a VOS-MOTS stage—due to the scarcity of mask annotations compared to bounding boxes.

Future work aims to achieve a single, end-to-end training pass and potentially integrate further vision tasks, such as optical flow or depth estimation, into this unified framework, moving closer to a truly general vision model.

Summary updated

BibTeX
@inproceedings{yan2022towards,
  author = {Bin Yan and Yi Jiang and Peize Sun and Dong Wang and Zehuan Yuan and Ping Luo and Huchuan Lu},
  title = {{Towards Grand Unification of Object Tracking}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-19803-8\_43},
}
Figure