Multiple Object Tracking With Correlation Learning

Qiang Wang, Yun Zheng, Pan Pan, Yinghui Xu

Abstract

Recent works have shown that convolutional networks have substantially improved the performance of multiple object tracking by simultaneously learning detection and appearance features. However, due to the local perception of the convolutional network structure itself, the long-range dependencies in both the spatial and temporal cannot be obtained efficiently. To incorporate the spatial layout, we propose to exploit the local correlation module to model the topological relationship between targets and their surrounding environment, which can enhance the discriminative power of our model in crowded scenes. Specifically, we establish dense correspondences of each spatial location and its context, and explicitly constrain the correlation volumes through self-supervised learning. To exploit the temporal context, existing approaches generally utilize two or more adjacent frames to construct an enhanced feature representation, but the dynamic motion scene is inherently difficult to depict via CNNs. Instead, our paper proposes a learnable correlation operator to establish frame-to-frame matches over convolutional feature maps in the different layers to align and propagate temporal context. With extensive experimental results on the MOT datasets, our approach demonstrates the effectiveness of correlation learning with the superior performance and obtains state-of-the-art MOTA of 76.5% and IDF1 of 73.6% on MOT17.

Introduction

Most modern Multiple Object Tracking (MOT) systems follow the "tracking-by-detection" paradigm, where objects are detected in each frame and subsequently linked across time based on appearance features. However, this approach faces significant hurdles in crowded environments. Standard Convolutional Neural Networks (CNNs) possess a "local" focus, meaning they primarily analyze pixels within a small area around a target. This makes it difficult to distinguish between similar-looking individuals (distractors) or maintain tracking during occlusions and complex motion.

To address these limitations, we introduce CorrTracker, a novel framework that moves beyond simple appearance matching by integrating "Correlation Learning." By explicitly modeling spatial relationships and temporal matches, CorrTracker significantly reduces identity switches and handles distractors more effectively.

Methodology

CorrTracker incorporates correlation learning through three primary components designed to capture long-range dependencies that standard CNNs often miss.

Spatial Local Correlation (SLC)

Rather than relying solely on a target's appearance, the SLC module computes a "correlation volume" that records the relationship between a target and its neighbors. This captures the surrounding context—a unique "topological signature." Even if two people look identical, their surrounding environments are likely different, providing the necessary discriminative power to tell them apart.

Overview of correlation learning: (a) Appearance only vs (b) Correlation boost, and (c) the spatial-temporal correlation representation.

Temporal Correlation Learning (TLC)

To handle motion and occlusions, the model establishes frame-to-frame matches over convolutional feature maps. This learnable correlation operator allows the tracker to align and propagate information across time, maintaining a "memory" of an object even during motion blur or partial occlusion. This temporal context is applied across a Multi-scale Pyramid to ensure the model can handle objects of various sizes.

Training and Efficiency

To ensure robust features, the model is trained using self-supervised proxy tasks, including identity-based tracking supervision and a colorization task where the model learns pixel-level correspondences by "coloring" a frame based on a reference. To maintain real-time viability, the system utilizes a local search window (radius $R$) and dilated correlations, increasing the receptive field without the computational overhead of global correlation.

Experimental Results

CorrTracker was evaluated on the MOT15, MOT16, MOT17, and MOT20 benchmarks, following pre-training on massive datasets such as COCO and CrowdHuman.

On the MOT17 benchmark, the model achieved state-of-the-art results with a MOTA (Multiple Object Tracking Accuracy) of 76.5% and an IDF1 (Identity F1 Score) of 73.6%. Notably, it reduced identity switches by approximately 15% compared to baseline models like FairMOT. Despite the added complexity of correlation layers, the model maintains a practical processing speed of 15-18 FPS on a Tesla V100 GPU.

Conclusion and Future Work

The primary insight of CorrTracker is that context is as vital as appearance. By modeling the local structure around an object, the tracker becomes significantly more robust in crowded scenes. While highly effective, future work may address the challenges of extremely long-term re-identification (beyond the current 5-frame memory) and improving accuracy in ultra-dense crowds where anchor-free designs may face trade-offs against slower, graph-based optimization methods.

Summary updated

Abstract

Recent works have shown that convolutional networks have substantially improved the performance of multiple object tracking by simultaneously learning detection and appearance features. However, due to the local perception of the convolutional network structure itself, the long-range dependencies in both the spatial and temporal cannot be obtained efficiently. To incorporate the spatial layout, we propose to exploit the local correlation module to model the topological relationship between targets and their surrounding environment, which can enhance the discriminative power of our model in crowded scenes. Specifically, we establish dense correspondences of each spatial location and its context, and explicitly constrain the correlation volumes through self-supervised learning. To exploit the temporal context, existing approaches generally utilize two or more adjacent frames to construct an enhanced feature representation, but the dynamic motion scene is inherently difficult to depict via CNNs. Instead, our paper proposes a learnable correlation operator to establish frame-to-frame matches over convolutional feature maps in the different layers to align and propagate temporal context. With extensive experimental results on the MOT datasets, our approach demonstrates the effectiveness of correlation learning with the superior performance and obtains state-of-the-art MOTA of 76.5% and IDF1 of 73.6% on MOT17.
BibTeX
@inproceedings{wang2021multiple,
  author = {Qiang Wang and Yun Zheng and Pan Pan and Yinghui Xu},
  title = {{Multiple Object Tracking With Correlation Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=iDDsT1h1Lbe},
}
Figure