End-to-End Multi-Object Tracking with Global Response Map

Xingyu Wan, Jiakai Cao, Sanping Zhou, Jinjun WangarXiv 2020 · OpenReview

Abstract

Most existing Multi-Object Tracking (MOT) approaches follow the Tracking-by-Detection paradigm and the data association framework where objects are firstly detected and then associated. Although deep-learning based method can noticeably improve the object detection performance and also provide good appearance features for cross-frame association, the framework is not completely end-to-end, and therefore the computation is huge while the performance is limited. To address the problem, we present a completely end-to-end approach that takes image-sequence/video as input and outputs directly the located and tracked objects of learned types. Specifically, with our introduced multi-object representation strategy, a global response map can be accurately generated over frames, from which the trajectory of each tracked object can be easily picked up, just like how a detector inputs an image and outputs the bounding boxes of each detected object. The proposed model is fast and accurate. Experimental results based on the MOT16 and MOT17 benchmarks show that our proposed on-line tracker achieved state-of-the-art performance on several tracking metrics.

Introduction

Traditional Multi-Object Tracking (MOT) systems typically rely on the "Tracking-by-Detection" paradigm. This two-step process—detecting objects in individual frames and then associating them across time—often suffers from high computational costs and vulnerability to occlusions. If a detector fails in a single frame due to motion blur or crowding, the entire trajectory can fragment. This paper proposes a unified, end-to-end framework that treats tracking as a saliency detection problem, utilizing a Global Response Map to represent object locations and trajectories directly.

Methodology

The proposed system moves away from discrete bounding box predictions, instead generating a continuous response map where objects appear as Gaussian-like peaks.

System Pipeline

The overall architecture takes a sequence of video frames and outputs tracked object trajectories. As shown in Figure 1, the process involves generating response maps for consecutive frames, calculating optical flow to predict motion, and linking these detections via a data association step.

Overview of the proposed end-to-end tracking framework.

Global Response Map and Temporal Context

The core innovation is the Global Response Map. Each object is represented by a heat peak, with values between 0 and 1 indicating the probability of presence. To improve robustness against occlusion, the model processes a temporal window of frames (typically 5). This allows the network to maintain a "positive response" for an object even if it is temporarily obscured, as the historical context provides implicit motion cues that help "fill in the blanks."

Visualization of the Global Response Map (a) and its evolution over a time window (b) to handle occlusions.

Motion Regression and Data Linking

To associate peaks across frames, the system employs a motion displacement regression sub-network. Using pixel-wise optical flow extracted via FlowNet2, the model predicts the displacement ($\Delta x, \Delta y$) and scale changes ($\Delta w, \Delta h$) for each detected peak. These predictions are then matched with observed peaks in subsequent frames using the Hungarian algorithm based on Intersection-over-Union (IOU) distance.

Experimental Results

The model was evaluated on the MOT16 and MOT17 benchmarks, which feature challenging pedestrian tracking scenarios with heavy crowding and frequent occlusions.

  • Accuracy: The tracker achieved a Multi-Object Tracking Accuracy (MOTA) of 62.0 on MOT16 and 61.2 on MOT17, outperforming contemporary online trackers like Tracktor and Deep SORT.
  • Efficiency: The response map generation is highly efficient, running at 200 FPS. The complete pipeline, including optical flow and association, operates at 6 FPS—significantly faster than many detection-heavy alternatives that require substantial processing time per frame before tracking begins.
  • Robustness: Qualitative analysis confirms that the Global Response Map approach is more resilient to "extremely occluded" scenes compared to traditional bounding-box-based detectors.

Conclusion

By unifying object localization and tracking into a single end-to-end framework based on saliency maps, this approach addresses the core weaknesses of the Tracking-by-Detection paradigm. While optical flow remains a computational bottleneck for real-time performance, the use of a Global Response Map provides a faster, more robust foundation for multi-object tracking. Future work aims to incorporate additional object attributes like orientation and depth directly into the response map representation.

Summary updated

Abstract

Most existing Multi-Object Tracking (MOT) approaches follow the Tracking-by-Detection paradigm and the data association framework where objects are firstly detected and then associated. Although deep-learning based method can noticeably improve the object detection performance and also provide good appearance features for cross-frame association, the framework is not completely end-to-end, and therefore the computation is huge while the performance is limited. To address the problem, we present a completely end-to-end approach that takes image-sequence/video as input and outputs directly the located and tracked objects of learned types. Specifically, with our introduced multi-object representation strategy, a global response map can be accurately generated over frames, from which the trajectory of each tracked object can be easily picked up, just like how a detector inputs an image and outputs the bounding boxes of each detected object. The proposed model is fast and accurate. Experimental results based on the MOT16 and MOT17 benchmarks show that our proposed on-line tracker achieved state-of-the-art performance on several tracking metrics.
BibTeX
@misc{wan2020end,
  author = {Xingyu Wan and Jiakai Cao and Sanping Zhou and Jinjun Wang},
  title = {{End-to-End Multi-Object Tracking with Global Response Map}},
  year = {2020},
  eprint = {2007.06344},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.06344},
}
Figure