Z-GMOT: Zero-shot Generic Multiple Object Tracking

Kim Tran, Anh Duy Le Dinh, Tien-Phat Nguyen, Thinh Phan, Pha A. Nguyen, Khoa Luu, Donald A. Adjeroh, Gianfranco Doretto, Ngan LeNAACL 2024 · DOI · OpenReview

TL;DR

This paper introduces Z-GMOT, a novel zero-shot tracking framework that can identify and follow objects from entirely new categories without prior training or initial bounding boxes. It achieves this by combining an improved language-guided object detector (iGLIP) with a sophisticated object association module (MA-SORT), and also releases a new dataset, Referring GMOT, to facilitate this research.

What Problem Does This Paper Address?

Traditional Multiple Object Tracking (MOT) systems are powerful but often require extensive labeled data and are limited to tracking predefined object categories. Generic Multiple Object Tracking (GMOT) aims to reduce this dependency by focusing on tracking objects of similar generic types, but current GMOT methods still struggle with variations in object appearance (like lighting or occlusion) and often rely on initial bounding boxes to start tracking. This leaves a significant gap in tracking systems that can adapt to completely unseen object categories and track them robustly without any prior examples or manual initialization.

Methods

The paper proposes Z-GMOT, a zero-shot generic multiple object tracking framework built on the tracking-by-detection paradigm. It introduces two main novel components:

  • iGLIP (Improved Grounded Language-Image Pretraining): This is an enhanced object detector that leverages vision-language models. Unlike standard detectors, iGLIP takes natural language descriptions (prompts) as input to identify objects. It features a dual-pathway design: one for general object categories and another for specific attributes (like color). This allows it to detect unseen objects with specific characteristics more accurately, overcoming limitations of existing models that are sensitive to threshold settings and can produce high false positives or negatives. It uses a Query-Guided Matching (QGM) module to refine detections.
  • MA-SORT (Motion-Appearance SORT): This is a novel object association algorithm designed to handle objects with high visual similarity and complex motion patterns, common in GMOT scenarios. It dynamically balances motion and appearance cues for matching. A key insight is that when detections are visually very similar, motion cues should be prioritized. MA-SORT quantifies this visual similarity and adjusts the weighting between motion and appearance costs accordingly.

Datasets

The paper introduces the Referring GMOT dataset, which is built upon existing GMOT datasets (GMOT-40 and AnimalTrack) by adding detailed natural language descriptions to the videos. This dataset is crucial for training and evaluating zero-shot GMOT capabilities.

The framework is also evaluated on:

  • DanceTrack: A dataset for multi-human tracking.
  • MOT20: A benchmark for multi-object tracking in crowded scenes.

Key Insights and Ideas

  • Zero-shot tracking is achievable for generic objects: The core idea is that by combining powerful language-guided detection with robust association, we can track objects from categories never seen during training.
  • Natural language is a powerful tool for object detection: Using descriptive prompts allows for flexible and precise object identification, especially for specific attributes.
  • Balancing motion and appearance is crucial for similar objects: When objects look alike, relying solely on appearance can lead to errors. Prioritizing motion cues in such scenarios significantly improves tracking.
  • New datasets are needed for new capabilities: The Referring GMOT dataset is a key contribution, enabling research into language-guided GMOT.

Key Results

  • Referring GMOT Dataset: The paper introduces this new dataset with natural language descriptions, filling a gap in GMOT research.
  • iGLIP Performance: iGLIP shows significant improvements over GLIP in object detection, achieving higher AP50, AP75, and mAP scores on the Referring GMOT dataset.
  • Z-GMOT Performance: Z-GMOT demonstrates superior performance compared to existing OS-OD (one-shot object detection) methods on the Refer-GMOT40 dataset, achieving substantial gains in HOTA, MOTA, and IDF1 metrics.
  • Generalizability: Z-GMOT shows strong performance on MOT tasks (DanceTrack and MOT20), even outperforming some fully-supervised methods without requiring any training data.
  • MA-SORT Effectiveness: MA-SORT consistently outperforms other trackers on the Refer-GMOT40 dataset, showing significant improvements in HOTA, MOTA, and IDF1.

Limitations and Future Work

  • Reliance on Vision-Language Models (VLMs): While iGLIP leverages VLMs, the paper notes that the field of VLMs is rich and diverse, suggesting that exploring other VLMs could lead to further improvements.
  • Prompt Engineering: The current Z-GMOT implementation primarily uses object and attribute descriptions. The Referring GMOT dataset also contains object synonyms and other attributes, which could be explored for more advanced prompt tuning and engineering.
  • Open-Vocabulary MOT/GMOT: The authors envision their work inspiring future research in open-vocabulary MOT/GMOT paradigms, potentially leading to extensions in other tracking scenarios.

Summary updated

Abstract

Despite recent significant progress, Multi-Object Tracking (MOT) faces limitations such as reliance on prior knowledge and predefined categories and struggles with unseen objects. To address these issues, Generic Multiple Object Tracking (GMOT) has emerged as an alternative approach, requiring less prior information. However, current GMOT methods often rely on initial bounding boxes and struggle to handle variations in factors such as viewpoint, lighting, occlusion, and scale, among others. Our contributions commence with the introduction of the Referring GMOT dataset a collection of videos, each accompanied by detailed textual descriptions of their attributes. Subsequently, we propose Z-GMOT, a cutting-edge tracking solution capable of tracking objects from never-seen categories without the need of initial bounding boxes or predefined categories. Within our Z-GMOT framework, we introduce two novel components: (i) iGLIP, an improved Grounded language-image pretraining, for accurately detecting unseen objects with specific characteristics. (ii) MA-SORT, a novel object association approach that adeptly integrates motion and appearance-based matching strategies to tackle the complex task of tracking objects with high similarity. Our contributions are benchmarked through extensive experiments conducted on the Referring GMOT dataset for GMOT task. Additionally, to assess the generalizability of the proposed Z-GMOT, we conduct ablation studies on the DanceTrack and MOT20 datasets for the MOT task. Our dataset, code, and models are released at: https://fsoft-aic.github.io/Z-GMOT
BibTeX
@inproceedings{tran2024z,
  author = {Kim Tran and Anh Duy Le Dinh and Tien-Phat Nguyen and Thinh Phan and Pha A. Nguyen and Khoa Luu and Donald A. Adjeroh and Gianfranco Doretto and Ngan Le},
  title = {{Z-GMOT: Zero-shot Generic Multiple Object Tracking}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.220},
}
Figure