CountingMOT: Joint Counting, Detection and Re-Identification for Multiple Object Tracking

Weihong Ren, Bowen Chen, Yuhang Shi, Weibo Jiang, Honghai Liu

Introduction

Multiple Object Tracking (MOT) has evolved significantly with the rise of Joint Detection and Tracking (JDT) frameworks. These models typically integrate object detection and re-identification (reID) into a single network for efficiency. However, tracking in extremely crowded environments—such as busy public squares or transit hubs—remains a formidable challenge. In these scenarios, heavy occlusions lead to frequent missed detections, while visual clutter generates numerous false positives. CountingMOT addresses these issues by introducing a third task: crowd density estimation, which provides a global context to refine local object detections.

Methodology

The core of CountingMOT is an end-to-end framework that jointly models object counting, detection, and re-identification.

Architecture

The model utilizes a shared backbone (e.g., DLA-34 or YOLOX) to extract high-level features. These features are then processed by three parallel branches:

  1. Detection Branch: An anchor-free head that predicts object centers and bounding box scales.
  2. Density Estimation Branch: Generates a density map where the integral over any region represents the estimated count of objects.
  3. reID Branch: Extracts 128-dimensional embedding features used for temporal data association.
The CountingMOT architecture features three parallel branches—Detection, Density Estimation, and reID—linked by mutual constraint losses ($L_{dc}$ and $L_{cd}$) to ensure consistency between global counts and local detections.

Mutual Constraints

The primary innovation lies in the mutual constraints between the detection and density estimation tasks. Instead of training the branches independently, the authors introduce two cross-task losses:

  • Detection with Counting ($L_{dc}$): The density map acts as a teacher, helping the detection branch find occluded people it might have missed.
  • Counting with Detection ($L_{cd}$): Individual detections are used to refine the density map, improving its spatial localization.

To enforce these constraints, the system employs sliding windows across the frame, minimizing the difference between the count derived from the density map and the count of detected bounding boxes within each window. Additionally, uncertainty weighting is used to automatically balance the importance of each loss function during training.

Key Insights: The Synergy of Counting and Detecting

Detection excels at precise localization but struggles with overlapping objects. Conversely, density maps are robust at counting in dense clusters but lack individual bounding boxes. By forcing these two modalities to agree via mutual constraints, CountingMOT achieves a "best of both worlds" effect.

Qualitative comparison showing the input frame, initial object detections, the generated density map, and the final refined detections. The synergy between tasks allows the model to reject false positives and find hidden individuals.

Experimental Results

CountingMOT was evaluated on the MOT Challenge benchmarks, with a focus on the high-density MOT20 dataset.

  • State-of-the-Art Performance: On MOT20, the model achieved a Multiple Object Tracking Accuracy (MOTA) of 78.9%, a significant improvement over the FairMOT baseline (61.8%).
  • Error Reduction: The framework dramatically reduced false positives (FP) on MOT20 from over 103,000 to approximately 33,500.
  • Real-Time Efficiency: Despite the multi-task complexity, the system maintains a processing speed of approximately 25 FPS, making it viable for real-time online tracking.

Conclusion and Future Work

CountingMOT demonstrates that global crowd context is a powerful tool for improving local object tracking in dense scenes. While highly effective for detection, the authors note that re-identification under extreme occlusion remains a challenge. Future research may explore unsupervised reID techniques or more advanced temporal modeling to maintain identities during prolonged occlusions.

Summary updated

Abstract

The recent trend in multiple object tracking (MOT) is jointly solving detection and tracking, where object detection and appearance feature (or motion) are learned simultaneously. Despite competitive performance, in crowded scenes, joint detection and tracking usually fail to find accurate object associations due to missed or false detections. In this paper, we jointly model counting, detection and re-identification in an end-to-end framework, named CountingMOT, tailored for crowded scenes. By imposing mutual object-count constraints between detection and counting, the CountingMOT tries to find a balance between object detection and crowd density map estimation, which can help it to recover missed detections or reject false detections. Our approach is an attempt to bridge the gap of object detection, counting, and re-Identification. This is in contrast to prior MOT methods that either ignore the crowd density and thus are prone to failure in crowded scenes, or depend on local correlations to build a graphical relationship for matching targets. The proposed MOT tracker can perform online and real-time tracking, and achieves the state-of-the-art results on public benchmarks MOT16 (MOTA of 79.7), MOT17 (MOTA of 81.3%) and MOT20 (MOTA of 78.9%).
BibTeX
@misc{ren2022countingmot,
  author = {Weihong Ren and Bowen Chen and Yuhang Shi and Weibo Jiang and Honghai Liu},
  title = {{CountingMOT: Joint Counting, Detection and Re-Identification for Multiple Object Tracking}},
  year = {2022},
  eprint = {2212.05861},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.05861},
}
Figure