Introduction
Multi-Object Tracking (MOT) is a fundamental task in computer vision, critical for applications ranging from autonomous driving to human-robot interaction. Despite significant progress, trackers often struggle with "track drift," where occlusions or appearance changes cause the system to lose its target. Furthermore, many existing systems are limited to specific object classes (e.g., pedestrians) and struggle with the dynamic nature of real-world scenes where objects frequently enter and exit the frame.
This paper, presented at the ECCV 2016 workshops, introduces the Multi-Class Multi-Object Tracking (MCMOT) framework. MCMOT addresses these challenges by combining deep learning-based detection with a novel Changing Point Detection (CPD) algorithm to identify and recover from tracking failures.
The MCMOT Framework
The MCMOT framework is built upon a Bayesian filtering foundation, utilizing Markov Chain Monte Carlo (MCMC) sampling for efficient state estimation. The system is designed to be class-agnostic and computationally efficient.
The framework employs an ensemble of four detectors to compute object likelihoods:
- Global Detector (GT): A Faster R-CNN/ResNet model for general object detection.
- Local Detector (LT): A detector fine-tuned on-the-fly using confident track segments to reduce false negatives.
- Color Detector (CT): Uses RGB histograms and Bhattacharyya distance to maintain appearance consistency.
- Motion Detector (MT): A Lucas-Kanade Tracker (KLT) for position prediction.
Changing Point Detection and Validation
The core innovation of MCMOT is the Changing Point Detection (CPD) algorithm. By treating a track as a time series of data attributes, the system can detect abrupt changes that signify potential drift or occlusion.
When a change point is detected, the system performs Forward-Backward (FB) validation. This involves a reverse-sequence check; if the backward track does not align with the forward track, the segment is identified as unstable and is either pruned or corrected. This proactive approach to failure detection allows the system to maintain high-quality trajectories even in cluttered environments.
Evaluation and Results
The authors evaluated MCMOT on two major benchmarks: ImageNet VID and MOT 2016.
- ImageNet VID: MCMOT achieved a mean Average Precision (mAP) of 74.5%, outperforming the previous state-of-the-art T-CNN method (73.8%).
- MOT 2016: The system reached a 62.4% MOTA (Multi-Object Tracking Accuracy), which was a leading score at the time of publication.
Ablation studies confirmed the impact of the new components: adding CPD to the baseline improved mAP by 6.4%, and incorporating FB validation added a further 3.4%.
Conclusion
MCMOT demonstrates that acknowledging and actively detecting tracking failures is a powerful strategy for robust multi-object tracking. By integrating deep learning detectors with CPD and FB validation, the framework achieves state-of-the-art performance across diverse object categories. While the system currently faces challenges with identity switches and fragmentation over long occlusions, it sets a strong foundation for future research in class-agnostic, resilient tracking.
Summary updated