Online learning of robust object detectors during unstable tracking

Zdenek Kalal, Jiri Matas, Krystian MikolajczykWorkshops @ ICCV 2009 · OpenReview

Abstract

Long-term object tracking remains a significant challenge in computer vision due to object disappearances, occlusions, and appearance changes. This paper introduces the Tracking-Modeling-Detection (TMD) framework, which learns a robust object detector online from a single user initialization. By integrating a short-term tracker with a detector that self-corrects through "growing" and "pruning" events, the system maintains stability over long durations. Experimental results demonstrate that TMD effectively handles drift and re-localization, outperforming state-of-the-art methods in challenging real-world sequences.

Introduction

Traditional object tracking methods typically fall into two categories: short-term trackers, which are precise but fail upon target loss, and detectors, which require extensive pre-labeled data. Long-term tracking requires a hybrid approach capable of following an unknown object indefinitely. The primary obstacle is "drift," where adaptive trackers gradually incorporate background information into the object model. The TMD framework addresses this by building a detector on the fly, using the tracker's local continuity to supervise the detector's global learning process.

The TMD Framework

The TMD framework consists of three core components:

  1. Tracking: A Lucas-Kinade (LK) tracker handles frame-to-frame motion estimation.
  2. Detection: A Randomized Forest classifier utilizes 2bit Binary Patterns (2bitBP) to scan for the object. 2bitBP features are designed to be computationally efficient and robust against overfitting by reducing the feature space to just four possible codes.
  3. Modeling: The object model is updated via two complementary processes:
  4. Growing Events: Confident tracker outputs are added to the training set to capture appearance variations.
  5. Pruning Events: Detections that contradict a reliable tracker are identified as false positives and used to refine the model.
Conceptual diagram of the TMD framework showing the interaction between the object model and the unknown object manifold via growing and pruning events.

A critical innovation is the LOOP strategy, which validates learning by identifying "closed loops" in trajectories. If a tracker returns to a previously known appearance after a period of uncertainty, the intervening frames are retroactively used for training.

Experimental Results

The TMD framework was evaluated on diverse sequences including indoor faces, pedestrians with full occlusions, and high-speed motorbikes. A key finding is the necessity of the pruning mechanism. Without pruning, the detector's false positive rate grows rapidly as it incorrectly incorporates background features.

Cumulative false detections over time, showing how pruning events prevent the exponential growth of errors seen in growing-only models.

On a 1000-frame face sequence, TMD achieved 55% recall with only a 10% error rate, significantly more stable than standard trackers that fail during the first occlusion. The system maintains real-time performance, reaching 20 fps in a Matlab implementation.

Conclusion

The TMD framework successfully bridges the gap between short-term tracking and long-term detection through a self-correcting online learning process. While the current model assumes object uniqueness for pruning, it provides a robust foundation for handling complex appearance changes and disappearances. Future work will explore multi-target tracking and more sophisticated learning dynamics.

Summary updated

BibTeX
@inproceedings{kalal2009online,
  author = {Zdenek Kalal and Jiri Matas and Krystian Mikolajczyk},
  title = {{Online learning of robust object detectors during unstable tracking}},
  booktitle = {Workshops @ ICCV},
  year = {2009},
  url = {https://openreview.net/forum?id=hMjS-yWd9Fz},
}
Figure