DeepMOT: A Differentiable Framework for Training Multiple Object Trackers

Yihong Xu, Yutong Ban, Xavier Alameda-Pineda, Radu Horaud

Abstract

The recent trend in vision-based multi-object tracking (MOT) is heading towards leveraging the representational power of deep learning to jointly learn to detect and track objects. However, existing methods train only certain sub-modules using loss functions that often do not correlate with established tracking evaluation measures such as Multi-Object Tracking Accuracy (MOTA) and Precision (MOTP). As these measures are not differentiable, the choice of appropriate loss functions for end-to-end training of multi-object tracking methods is still an open research problem. In this paper, we bridge this gap by proposing a differentiable proxy of MOTA and MOTP, which we combine in a loss function suitable for end-to-end training of deep multi-object trackers. As a key ingredient, we propose a Deep Hungarian Net (DHN) module that approximates the Hungarian matching algorithm. DHN allows estimating the correspondence between object tracks and ground truth objects to compute differentiable proxies of MOTA and MOTP, which are in turn used to optimize deep trackers directly. We experimentally demonstrate that the proposed differentiable framework improves the performance of existing multi-object trackers, and we establish a new state of the art on the MOTChallenge benchmark.

Introduction

Multi-Object Tracking (MOT) is a critical task in computer vision, requiring systems to detect multiple objects and maintain their identities across video frames. While deep learning has revolutionized object detection and re-identification (ReID), a significant gap remains: the standard metrics for evaluating MOT performance, such as Multi-Object Tracking Accuracy (MOTA) and Precision (MOTP), are non-differentiable. This is because they rely on the Hungarian algorithm for discrete matching between predictions and ground truth. Consequently, trackers are typically trained using proxy losses that do not directly optimize for the final evaluation criteria.

The DeepMOT Framework

To address this, we propose DeepMOT, a framework that enables end-to-end training of multi-object trackers by introducing differentiable proxies for MOTA and MOTP. The core of our approach is the Deep Hungarian Net (DHN), a module designed to approximate the Hungarian matching algorithm.

The proposed DeepMOT training framework (ii) allows gradients to flow through the assignment step, unlike standard baselines (i).

The DHN takes a distance matrix as input and produces a "soft" assignment matrix. By using Bi-directional RNNs to process row-wise and column-wise information, the DHN captures the global context necessary for optimal matching. This soft assignment allows for the formulation of differentiable loss functions that directly penalize tracking errors.

Differentiable MOT Metrics

Using the soft assignments from the DHN, we derive differentiable approximations for the components of the CLEAR-MOT metrics:

  • dMOTP: A proxy for tracking precision based on the weighted distance of matched pairs.
  • dMOTA: A proxy for tracking accuracy that incorporates differentiable counts of False Positives (FP), False Negatives (FN), and ID Switches (IDS).
Detailed breakdown of how the soft assignment matrix is used to compute differentiable proxies for MOT metrics.

By optimizing these proxies, the tracker learns to produce detections and features that are inherently better suited for the final matching process.

Experimental Results

We evaluated DeepMOT on the MOTChallenge benchmarks (MOT15, MOT16, and MOT17). Our results demonstrate that direct optimization of tracking metrics leads to significant performance gains. For instance, applying DeepMOT to the Tracktor baseline improved MOTA on MOT17 from 53.5 to 53.7 and notably reduced ID switches.

A key insight from our ablation studies is the effectiveness of combining different loss components. Training with a combination of $dMOTA$ and $dMOTP$ yielded the best overall performance, achieving a MOTA of 60.66 and reducing IDS to 118 on the MOT17 validation set, compared to 60.20 MOTA and 152 IDS for the vanilla baseline.

Furthermore, we found that DeepMOT allows for more efficient architectures. By penalizing ID switches directly during training, we could replace heavy, externally-trained ReID models with simple, lightweight fully-connected layers while still achieving superior tracking consistency.

Conclusion

DeepMOT bridges the gap between training and evaluation in multi-object tracking. By making the matching process differentiable through the Deep Hungarian Net, we enable trackers to be optimized directly for the metrics that matter most. This framework not only sets a new state-of-the-art but also paves the way for fully end-to-end, differentiable tracking architectures.

Summary updated

Abstract

The recent trend in vision-based multi-object tracking (MOT) is heading towards leveraging the representational power of deep learning to jointly learn to detect and track objects. However, existing methods train only certain sub-modules using loss functions that often do not correlate with established tracking evaluation measures such as Multi-Object Tracking Accuracy (MOTA) and Precision (MOTP). As these measures are not differentiable, the choice of appropriate loss functions for end-to-end training of multi-object tracking methods is still an open research problem. In this paper, we bridge this gap by proposing a differentiable proxy of MOTA and MOTP, which we combine in a loss function suitable for end-to-end training of deep multi-object trackers. As a key ingredient, we propose a Deep Hungarian Net (DHN) module that approximates the Hungarian matching algorithm. DHN allows estimating the correspondence between object tracks and ground truth objects to compute differentiable proxies of MOTA and MOTP, which are in turn used to optimize deep trackers directly. We experimentally demonstrate that the proposed differentiable framework improves the performance of existing multi-object trackers, and we establish a new state of the art on the MOTChallenge benchmark. Our code is publicly available from https://github.com/yihongXU/deepMOT.
BibTeX
@inproceedings{xu2020deepmot,
  author = {Yihong Xu and Yutong Ban and Xavier Alameda-Pineda and Radu Horaud},
  title = {{DeepMOT: A Differentiable Framework for Training Multiple Object Trackers}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  url = {https://openreview.net/forum?id=TErOFVn7-zr},
}
Figure