Deep Learning of Appearance Models for Online Object Tracking

Mengyao Zhai, Mehrsan Javan Roshtkhari, Greg Mori

TL;DR

This paper introduces a novel online object tracking framework that integrates a deep Convolutional Neural Network (CNN) with a Bayesian loss layer. By estimating the probability distributions of target and background features online, the network directly computes tracking scores for candidate locations and adaptively updates its weights frame-by-frame, successfully overcoming the data-scarcity challenge inherent in deep learning-based tracking.

What Problem Does This Paper Address?

Visual object tracking requires a system to follow a target throughout a video sequence given only its initial bounding box in the first frame. While deep learning has revolutionized image classification and object detection, applying it to online tracking is notoriously difficult due to three main challenges:

  1. Data Scarcity: Deep networks typically require thousands of training examples, but a tracker only has access to a single positive instance (the initial target bounding box).
  2. Specificity Dilemma: Pre-trained deep networks excel at learning category-specific features (e.g., distinguishing a dog from a car), whereas tracking requires target-specific features (e.g., distinguishing a specific dog from other dogs or a cluttered background).
  3. Computational Overhead: Deep networks traditionally require long, iterative training processes to converge, making real-time online adaptation computationally prohibitive.

Methods

The authors propose a unified framework that treats tracking as a classification problem, combining a modified AlexNet architecture with a Naive Bayes classifier acting as a custom loss layer.

Fig. 1: Overview of our approach: Given one frame, we sample three batches: positive batch, negative batch, and prediction batch. In the forward procedure, given CNN parameters, we use the positive batch and negative batch to re-estimate Gaussian parameters. Then we search in the prediction batch for the new location with maximum score. In the backward procedure, given Gaussian parameters, we compute gradients with respect to feature nodes and update CNN parameters.

The training and tracking pipeline consists of three distinct phases:

  1. Offline Pre-training for "Objectness" (Phase 1): The network is first trained on ImageNet for classification, and then fine-tuned on 100,000 patches from the ImageNet detection dataset. This teaches all layers of the CNN to extract generic features that distinguish foreground objects from background clutter.
  2. Target-Specific Initialization (Phase 2): In the first frame of a video, the network's final fully connected layer (fc7) is reduced to 500 dimensions. Bounding boxes are sampled around the target to create positive and negative examples, and only the fully connected layers are fine-tuned to capture the specific target's appearance.
  3. Simultaneous Tracking and Online Fine-Tuning: For every subsequent frame, a two-stage iterative process is executed:
  4. Forward Pass (Stage 1): Candidate patches are sampled around the target's previous location. The network extracts features, and the Bayesian layer models their distributions as single Gaussians to compute a tracking score. The location with the highest score is selected as the new target position.
  5. Backward Pass (Stage 2): Using the new target location, positive and negative patches are re-sampled. The Gaussian parameters are updated via maximum likelihood estimation, and the tracking error is backpropagated to update the CNN's fully connected weights. To prevent model drift during occlusions, updates are only performed if the tracking score exceeds a confidence threshold.

Datasets

  • ImageNet 2014 Detection Dataset: Used for the offline "objectness" fine-tuning phase (100k auxiliary image patches).
  • CVPR13 Visual Tracker Benchmark: Used for comprehensive evaluation. This dataset consists of 50 challenging video sequences featuring common tracking hurdles such as illumination changes, occlusions, scale variations, and background clutter.

Key Insights and Ideas

  • Bridging Generative and Discriminative Models: The architecture uniquely combines a generative component (modeling feature distributions using Gaussians) with a discriminative component (updating CNN weights via backpropagation based on classification errors).
  • The Power of "Objectness": Directly fine-tuning a classification network on a specific target often fails because the features are too rigid. Introducing an intermediate "objectness" training step provides a robust middle ground, yielding features that adapt much more effectively to unseen targets.
  • Conservative Online Updates: By gating backpropagation based on the tracking confidence score, the network retains a "memory" of the target's historical appearance, preventing the model from corrupting its weights when the target is temporarily occluded or lost.

Key Results

The proposed tracker, denoted as GDT (Gaussian Density Tracker), was evaluated against eight state-of-the-art methods (including both hand-crafted feature trackers like KCF and Struck, and deep learning trackers like FCNT and SO-DLT).

  • Overall Performance: On the CVPR13 benchmark, GDT achieved a state-of-the-art Success Score of 0.613 (based on bounding box overlap), outperforming all competing deep and shallow trackers.
  • Precision: It achieved a Precision Score of 0.841 (at a 20-pixel location error threshold), proving highly competitive, though slightly behind architectures that utilize features from multiple convolutional layers simultaneously (like HCFT and HDT).
  • Robustness to Scale and Occlusion: GDT demonstrated superior capabilities in accurately localizing and estimating the scale of targets, particularly in scenarios involving out-of-view movements and severe occlusions.
  • Speed: The prototype runs at approximately 1 frame per second (fps) on a PC equipped with an Intel i7-4790 CPU and an Nvidia Titan X GPU.

Limitations and Future Work

  • Processing Speed: At 1 fps, the current implementation is too slow for real-time applications. Optimizing the sampling mechanism and network architecture for speed remains a critical next step.
  • Simplified Density Estimation: The authors assume a single Gaussian distribution for simplicity and efficiency. However, they note that replacing this with a Gaussian Mixture Model (GMM) could better capture complex, multi-modal background distributions and diverse target shapes, potentially boosting tracking accuracy further.

Summary updated

Abstract

This paper introduces a novel deep learning based approach for vision based single target tracking. We address this problem by proposing a network architecture which takes the input video frames and directly computes the tracking score for any candidate target location by estimating the probability distributions of the positive and negative examples. This is achieved by combining a deep convolutional neural network with a Bayesian loss layer in a unified framework. In order to deal with the limited number of positive training examples, the network is pre-trained offline for a generic image feature representation and then is fine-tuned in multiple steps. An online fine-tuning step is carried out at every frame to learn the appearance of the target. We adopt a two-stage iterative algorithm to adaptively update the network parameters and maintain a probability density for target/non-target regions. The tracker has been tested on the standard tracking benchmark and the results indicate that the proposed solution achieves state-of-the-art tracking results.
BibTeX
@inproceedings{zhai2018deep,
  author = {Mengyao Zhai and Mehrsan Javan Roshtkhari and Greg Mori},
  title = {{Deep Learning of Appearance Models for Online Object Tracking}},
  booktitle = {Workshops @ ECCV},
  year = {2018},
  doi = {10.1007/978-3-030-11018-5\_57},
}
Figure