Trajectory Factory: Tracklet Cleaving and Re-connection by Deep Siamese Bi-GRU for Multiple Object Tracking

Cong Ma, Changshui Yang, Fan Yang, Yueqing Zhuang, Ziwei Zhang, Huizhu Jia, Xiaodong Xie

Abstract

Multi-Object Tracking (MOT) is a challenging task in complex scenes such as surveillance and autonomous driving. In this paper, we propose a novel tracklet processing method to cleave and re-connect tracklets on crowd or long-term occlusion by Siamese Bi-Gated Recurrent Unit (GRU). The tracklet generation utilizes object features extracted by CNN and RNN to create high-confidence tracklet candidates in sparse scenarios. Due to mis-tracking in the generation process, tracklets from different objects are split into several sub-tracklets by a bidirectional GRU. After that, a Siamese GRU based tracklet re-connection method is applied to link the sub-tracklets which belong to the same object to form a whole trajectory. In addition, we extract the tracklet images from existing MOT datasets and propose a novel dataset to train our networks. Experimental results demonstrate the advantages of our model over state-of-the-art methods on MOT16.

Introduction

Modern Multi-Object Tracking (MOT) systems typically follow the "tracking-by-detection" paradigm. While effective, this approach struggles in crowded environments or during long-term occlusions, leading to two primary failure modes: Identity Switches, where a tracker merges two different individuals into a single tracklet, and Fragmentation, where a single person's path is broken into multiple disconnected segments.

Instead of solely focusing on improving initial association, the "Trajectory Factory" framework acknowledges that errors are inevitable and proposes a post-processing pipeline to identify and rectify these mistakes.

The Trajectory Factory Pipeline

The proposed framework treats tracklets as repairable objects through a three-stage process: generation, cleaving, and re-connection.

Overview of the three-stage Trajectory Factory pipeline: Tracklet Generation, Cleaving, and Re-connection.

1. Tracklet Generation

Initial high-confidence tracklets are generated using appearance features from a ResNet-50 CNN and motion predictions from an LSTM. This stage focuses on creating reliable candidates in sparse scenarios.

2. Tracklet Cleaving

To resolve identity switches, the system employs a Bidirectional Gated Recurrent Unit (Bi-GRU). By analyzing the temporal sequence of features from both forward and backward directions, the model calculates a "feature distance" at every point. A sudden peak in this distance indicates a likely switch between different individuals, triggering a "cleave" at that temporal location.

Visualization of the cleaving mechanism: feature distances peak when the tracker switches from a white-coat individual to a blue-coat individual.

3. Tracklet Re-connection

Fragmented sub-tracklets are linked using a Siamese Bi-GRU architecture. This network determines if two fragments belong to the same identity by optimizing a multi-task loss that combines verification and identification. Polynomial curve fitting is then used to interpolate smooth trajectories across the gaps between re-connected segments.

Experimental Evaluation

The networks were trained using a combination of Market1501, PathTrack, and MARS datasets. Additionally, the authors introduced a specialized dataset containing over 95,160 pedestrian images (793 unique identities) extracted from existing MOT datasets to specifically train the cleaving and re-connection modules.

On the MOT16 benchmark, the "Trajectory Factory" (GCRA) achieved a Multi-Object Tracking Accuracy (MOTA) of 48.2, outperforming established methods like QuadMOT16 (44.1) and EDMT (45.3). The system demonstrated particularly strong performance in static camera sequences, where it frequently ranked first in accuracy.

Conclusion

The "Trajectory Factory" demonstrates that treating tracklets as temporal sequences and explicitly modeling the "cleave then link" process significantly improves tracking robustness. While the system excels in static scenes, performance in moving camera scenarios remains a challenge due to the noise introduced by camera ego-motion. Future research will focus on developing motion models and appearance descriptors that are more invariant to rapid perspective changes.

Summary updated

Abstract

Multi-Object Tracking (MOT) is a challenging task in the complex scene such as surveillance and autonomous driving. In this paper, we propose a novel tracklet processing method to cleave and re-connect tracklets on crowd or long-term occlusion by Siamese Bi-Gated Recurrent Unit (GRU). The tracklet generation utilizes object features extracted by CNN and RNN to create the high-confidence tracklet candidates in sparse scenario. Due to mis-tracking in the generation process, the tracklets from different objects are split into several sub-tracklets by a bidirectional GRU. After that, a Siamese GRU based tracklet re-connection method is applied to link the sub-tracklets which belong to the same object to form a whole trajectory. In addition, we extract the tracklet images from existing MOT datasets and propose a novel dataset to train our networks. The proposed dataset contains more than 95160 pedestrian images. It has 793 different persons in it. On average, there are 120 images for each person with positions and sizes. Experimental results demonstrate the advantages of our model over the state-of-the-art methods on MOT16.
BibTeX
@inproceedings{ma2018trajectory,
  author = {Cong Ma and Changshui Yang and Fan Yang and Yueqing Zhuang and Ziwei Zhang and Huizhu Jia and Xiaodong Xie},
  title = {{Trajectory Factory: Tracklet Cleaving and Re-connection by Deep Siamese Bi-GRU for Multiple Object Tracking}},
  booktitle = {ICME},
  year = {2018},
  doi = {10.1109/icme.2018.8486454},
}
Figure