Systematized event-aware learning for multi-object tracking

Hyemin Lee, Daijin KimUAI 2022

Introduction

Multi-Object Tracking (MOT) involves identifying and maintaining the identities of multiple objects, such as pedestrians or vehicles, across video frames. Despite advancements in deep learning, MOT systems often struggle with "abnormal events"—situations where targets are temporarily occluded (False Negatives), detectors hallucinate non-existent objects (False Positives), or targets exit the scene.

A primary limitation identified by the authors is that most MOT networks are trained on static, frame-by-frame ground-truth (GT) data. This approach fails to account for the temporal "messiness" of real-world tracking, where there is often a significant discrepancy between clean GT boxes and noisy detections. Consequently, tracking logic is frequently not optimized for the errors the system inevitably encounters during inference.

Methods: The SEAT Framework

The core innovation of this work is the Systematized Event-Aware Loss, which enables end-to-end training of the entire tracking pipeline.

System Architecture

The SEAT framework integrates several components to bridge the gap between training and real-world performance. As shown in the system diagram below, the architecture includes a Feature Extraction Network, Classification and Regression Networks, and an Association Network.

The SEAT framework architecture, highlighting the Systematized Event-Aware Network and the integration of the simulation tracker flow.

Key Components

  1. Event Categorization: The system uses a "Target State Table" to categorize interactions between tracked targets and new detections into five states: Tracking Success, Target Missing, Target Disappearance, False Positive, and New Target Appearance.
  2. Simulation Tracker: To expose the network to realistic errors during training, the authors employ a simulation tracker. This generates "actual" tracking candidates, including drifted bounding boxes, mimicking the conditions the model will face during inference.
  3. Loss Function: The total loss ($L$) is a weighted sum of Matching Loss ($L_m$), False Positive Loss ($L_{fp}$), and False Negative Loss ($L_{fn}$). Notably, $L_{fn}$ trains the tracker to locate targets even when the detector fails by forcing the network to regress to the GT box of a missing target.
  4. Deep Hungarian Network (DHN): To maintain end-to-end differentiability, a DHN is used to bridge the discrete association step, allowing gradients to flow back from the association matrix to the underlying networks.

Datasets and Evaluation

The SEAT method was evaluated on three prominent benchmarks: MOT2015, MOT2016, and MOT2017. The latter utilizes MOT16 sequences but with three different detection sets (DPM, SDP, and Faster R-CNN) to evaluate robustness against varying detector quality.

Key Results

The proposed Online Systematized Event-Aware Tracker (SEAT) demonstrated superior performance compared to state-of-the-art methods across all tested datasets.

Visual tracking results on indoor (mall) and outdoor (street) sequences, comparing raw detections with tracked targets.
  • Accuracy Improvements: On MOT2017, SEAT achieved a Multiple Object Tracking Accuracy (MOTA) of 57.3 and an Identity F1 Score (IDF1) of 58.2, outperforming the Tracktor++v2 baseline (56.3 MOTA).
  • Error Reduction: The framework significantly reduced False Negatives (FN). On MOT17, FNs dropped from 235,449 in the baseline to 223,051.
  • Efficiency: The system maintains competitive inference speeds at approximately 8.1 Hz, nearly matching the baseline's 8.6 Hz despite the increased logic complexity.

Limitations and Future Work

While SEAT advances MOT performance, it remains constrained by the "detector bottleneck," as the system's overall quality is still heavily influenced by the initial public detections required for benchmark comparisons. Additionally, the $O(n^3)$ complexity of the Hungarian algorithm may limit scalability in extremely crowded environments. Future research may explore more sophisticated motion models and alternative backbone architectures to further enhance precision.

Summary updated

BibTeX
@inproceedings{lee2022systematized,
  author = {Hyemin Lee and Daijin Kim},
  title = {{Systematized event-aware learning for multi-object tracking}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2022},
}
Figure