SLAck: Semantic, Location, and Appearance Aware Open-Vocabulary Tracking

Siyuan Li, Lei Ke, Yung-Hsu Yang, Luigi Piccinelli, Mattia Segù, Martin Danelljan, Luc Van Gool

TL;DR

This paper introduces SLAck, a novel framework for open-vocabulary Multiple Object Tracking (MOT) that unifies semantic, location, and appearance information from the early stages of association. By leveraging a spatial-temporal object graph, SLAck significantly improves tracking accuracy for novel object categories, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Traditional Multiple Object Tracking (MOT) systems are typically trained on a fixed set of object categories (e.g., pedestrians, vehicles). However, real-world applications demand trackers that can generalize to unseen object classes – this is the domain of Open-Vocabulary MOT. The challenge here is twofold: first, accurately classifying novel objects is difficult, and second, their motion patterns can be highly diverse and non-linear, unlike the more predictable movements in traditional MOT datasets. Existing methods often rely heavily on appearance matching, neglecting crucial semantic and motion cues, or they use these cues heuristically in later stages, leading to suboptimal performance, especially for novel classes. SLAck aims to bridge this gap by jointly and effectively integrating all three cues (semantics, location, and appearance) from the outset.

Methods

SLAck's core innovation lies in its unified approach to integrating different object cues. Instead of relying on complex post-processing heuristics, it proposes an early fusion strategy.

  • Feature Extraction: SLAck utilizes pre-trained detectors to extract three types of features for each detected object:
  • Semantic Features: Derived from an adapted classification head, distilled from a CLIP text encoder, to understand the object's category.
  • Location Features: Normalized bounding box coordinates, ensuring scale invariance.
  • Appearance Features: Extracted using a dedicated head, optimized for association.
  • Spatial-Temporal Object Graph (STOG): These features are then fed into a STOG. This graph structure uses:
  • Intra-frame Self-Attention: To model relationships between objects within the same frame, considering their spatial proximity and visual similarity.
  • Inter-frame Cross-Attention: To capture temporal dependencies and object dynamics across consecutive frames. This graph allows for rich information exchange between objects, enabling the model to learn implicit motion patterns and better understand object interactions.

  • Association Loss: The framework uses a differentiable Sinkhorn algorithm to optimize object matching based on a joint similarity matrix derived from the STOG.

  • Detection Aware Training (DAT): To address the issue of sparse annotations in datasets like TAO, SLAck employs DAT. This strategy leverages predicted bounding boxes from the detector during training, effectively simulating the inference process and bridging the gap between training and testing.

Datasets

  • TAO (Tracking Any Object): This is the primary dataset used for training and evaluation. It's a large-scale, open-vocabulary MOT dataset with over 800 categories, featuring diverse object classes and challenging motion patterns.
  • TAO TETA: Used for closed-set MOT evaluation, focusing on association quality.
  • MOT20: Used for comparison to highlight the differences in motion patterns between traditional pedestrian tracking and open-vocabulary tracking.

Key Insights and Ideas

  • Early Fusion is Key: Integrating semantic, location, and appearance cues from the early stages of association, rather than through late-stage heuristics, significantly boosts tracking performance, especially for novel classes.
  • Semantics Inform Motion: Semantic categories often have inherent, predictable motion patterns. By learning these patterns implicitly, trackers can generalize better to semantically similar novel classes.
  • STOG Captures Dynamics: The Spatial-Temporal Object Graph effectively models both spatial relationships within a frame and temporal evolution across frames, crucial for understanding complex object dynamics.
  • DAT Bridges the Annotation Gap: Detection Aware Training is a practical solution for leveraging sparse ground truth annotations in large-scale MOT datasets, enabling end-to-end training.

Key Results

  • State-of-the-Art Performance: SLAck achieves state-of-the-art results on open-vocabulary MOT benchmarks, particularly on the TAO dataset.
  • Significant Improvement in Association: On the open-vocabulary MOT benchmark, SLAck shows a substantial improvement in association accuracy (AssocA), with gains of up to +7.1 when incorporating semantic cues.
  • Outperforming Baselines: SLAck outperforms existing methods like OVTrack by a significant margin, especially in novel class tracking. For instance, on the TAO validation set, SLAck-OV achieves 37.8 AssocA, a +4.2 improvement over OVTrack.
  • Qualitative Success: The qualitative results (Fig. 12) demonstrate SLAck's ability to track objects in diverse real-world scenarios, even with text prompts unseen during testing.

Limitations and Future Work

  • Sensitivity to Frame Rate: SLAck is sensitive to video frame rate changes, as differences between training and inference frame rates can create a domain gap.
  • Need for Video Labels: Despite DAT, the method still requires video labels for training, which are costly and difficult to obtain, limiting scalability.
  • Future Work: The authors suggest exploring more sophisticated methods for temporal encoding and further investigating the relationship between semantic categories and their motion patterns. The limitations also point towards future research in reducing reliance on video annotations and improving robustness to varying frame rates.

Summary updated

Abstract

Open-vocabulary Multiple Object Tracking (MOT) aims to generalize trackers to novel categories not in the training set. Currently, the best-performing methods are mainly based on pure appearance matching. Due to the complexity of motion patterns in the large-vocabulary scenarios and unstable classification of the novel objects, the motion and semantics cues are either ignored or applied based on heuristics in the final matching steps by existing methods. In this paper, we present a unified framework SLAck that jointly considers semantics, location, and appearance priors in the early steps of association and learns how to integrate all valuable information through a lightweight spatial and temporal object graph. Our method eliminates complex post-processing heuristics for fusing different cues and boosts the association performance significantly for large-scale open-vocabulary tracking. Without bells and whistles, we outperform previous state-of-the-art methods for novel classes tracking on the open-vocabulary MOT and TAO TETA benchmarks. Our code is available at \href{https://github.com/siyuanliii/SLAck}{github.com/siyuanliii/SLAck}.
BibTeX
@inproceedings{li2024slack,
  author = {Siyuan Li and Lei Ke and Yung-Hsu Yang and Luigi Piccinelli and Mattia Segù and Martin Danelljan and Luc Van Gool},
  title = {{SLAck: Semantic, Location, and Appearance Aware Open-Vocabulary Tracking}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73383-3\_1},
}
Figure