End-to-End Joint Multi-Object Detection and Tracking for Intelligent Transportation Systems

Qing Xu, Xuewu Lin, Mengchi Cai, Yu-ang Guo, Chuang Zhang, Kai Li, Keqiang Li, Jianqiang Wang, Dongpu CaoChinese 2023 · DOI

Abstract

Environment perception is a critical technology for Intelligent Transportation Systems (ITS). The complex motion interactions between vehicles necessitate robust Multi-Object Tracking (MOT). Traditional MOT algorithms typically follow a "tracking-by-detection" framework, separating detection and tracking into independent segments, which limits global efficiency and requires complex post-processing. This paper introduces the Global Correlation Network (GCNet), an end-to-end framework for joint multi-object detection and tracking. By utilizing a global correlation layer for absolute coordinate regression, GCNet eliminates the need for data association and non-maximum suppression (NMS). Evaluated on the UA-DETRAC dataset, GCNet demonstrates state-of-the-art localization precision and real-time performance.

Introduction: The Challenge of Real-Time MOT

In modern ITS, tracking multiple vehicles simultaneously is essential for safety and traffic management. However, the standard approach—detecting objects in each frame and then linking them using algorithms like the Hungarian method—suffers from high computational costs and sensitivity to hyperparameters.

Most existing detectors rely on local features to predict relative offsets from anchors. These offsets do not translate well across frames, making the subsequent "data association" step difficult, especially during occlusions. GCNet addresses these flaws by unifying detection and tracking into a single network that understands global spatial relationships.

Methods: The Global Correlation Architecture

The core innovation of GCNet is the Global Correlation Layer. Unlike standard convolutions that focus on local neighborhoods, this layer allows the network to relate any specific point to the entire image context, enabling the regression of absolute coordinates rather than relative offsets.

The Q, K, V Mechanism

Borrowing from Transformer architectures, GCNet employs a Query (Q), Key (K), and Value (V) mechanism:

  • Detection: Q and K are derived from the same frame to identify and localize objects.
  • Tracking: Q is pulled from the previous frame, while K comes from the current frame. This allows the network to "query" the current frame to find the new position of a previously tracked object.

Spatial Attention and the Gate Step

To refine accuracy, GCNet uses a classification heatmap (identifying object centers) as a "gate." This spatial attention mechanism ensures the regression branch focuses only on relevant areas, significantly reducing noise in the final coordinates.

The GCNet tracking branch architecture, illustrating the integration of historical track features with current frame detections via a gate mechanism.

Experimental Results

GCNet was rigorously tested on the UA-DETRAC benchmark, which includes over 140,000 frames of vehicle traffic in diverse conditions, including rain, night, and heavy congestion.

Performance Metrics

  • Localization Precision: GCNet achieved a PR-MOTP of 57.3%, nearly doubling the precision of previous challenge winners. This indicates superior ability in pinpointing exact vehicle boundaries.
  • Detection Accuracy: The model reached 74.04% Average Precision (AP), outperforming established detectors like CenterNet.
  • Real-Time Efficiency: On an Nvidia 2080Ti, GCNet maintains 34 FPS for joint detection and tracking, making it suitable for live deployment.
Visual tracking results of GCNet on the UA-DETRAC dataset across various traffic scenarios and lighting conditions.

Key Insights and Future Work

GCNet proves that tracking can be treated as an implicit cross-frame correlation problem rather than a separate logical association task. By mathematically optimizing the global correlation layer, the authors achieved high-resolution processing without the prohibitive memory costs of standard self-attention.

While GCNet excels in spatial precision, future iterations will focus on enhancing long-term occlusion handling by developing more specialized features for the tracking branch. Additionally, further optimization for "sparse mode" processing could enable even higher frame rates on low-power edge devices used in roadside infrastructure.

Summary updated

Abstract

Abstract Environment perception is one of the most critical technology of intelligent transportation systems (ITS). Motion interaction between multiple vehicles in ITS makes it important to perform multi-object tracking (MOT). However, most existing MOT algorithms follow the tracking-by-detection framework, which separates detection and tracking into two independent segments and limit the global efficiency. Recently, a few algorithms have combined feature extraction into one network; however, the tracking portion continues to rely on data association, and requires complex post-processing for life cycle management. Those methods do not combine detection and tracking efficiently. This paper presents a novel network to realize joint multi-object detection and tracking in an end-to-end manner for ITS, named as global correlation network (GCNet). Unlike most object detection methods, GCNet introduces a global correlation layer for regression of absolute size and coordinates of bounding boxes, instead of offsetting predictions. The pipeline of detection and tracking in GCNet is conceptually simple, and does not require complicated tracking strategies such as non-maximum suppression and data association. GCNet was evaluated on a multi-vehicle tracking dataset, UA-DETRAC, demonstrating promising performance compared to state-of-the-art detectors and trackers.
BibTeX
@inproceedings{xu2023end,
  author = {Qing Xu and Xuewu Lin and Mengchi Cai and Yu-ang Guo and Chuang Zhang and Kai Li and Keqiang Li and Jianqiang Wang and Dongpu Cao},
  title = {{End-to-End Joint Multi-Object Detection and Tracking for Intelligent Transportation Systems}},
  booktitle = {Chinese},
  year = {2023},
  doi = {10.1186/s10033-023-00962-x},
}
Figure