Real-Time Correlation Tracking Via Joint Model Compression and Transfer

Ning Wang, Wengang Zhou, Yibing Song, Chao Ma, Houqiang Li

Deep learning has revolutionized visual object tracking, but it came with a heavy price tag: computational cost. While Correlation Filter (CF) trackers are inherently efficient, their modern "deep" variants usually rely on massive, off-the-shelf networks like VGG-M. This tethers them to high-end GPUs, leaving mobile devices and single-core CPU systems in the dust.

A team of researchers from USTC, Tencent AI Lab, and Shanghai Jiao Tong University has proposed a solution that bridges this gap, achieving real-time speeds on a single-core CPU without sacrificing the accuracy that makes deep tracking so desirable.

TL;DR

The authors introduce a framework to jointly compress heavy pre-trained CNNs and transfer their knowledge into a lightweight "student" network specifically for tracking. By using a dual-loss distillation approach, they reduced model size by 63x and achieved over 20 FPS on a single-core CPU while maintaining state-of-the-art accuracy.

What Problem Does This Paper Address?

The core challenge is the "heavyweight" nature of deep features. Standard deep trackers use models pre-trained for image classification. These models are redundant for tracking and require significant memory and power.

Existing attempts to speed things up usually involve simple model pruning or using hand-crafted features (like HOG), which leads to a significant drop in tracking precision. The authors wanted to find a way to keep the "richness" of deep features while making the architecture lean enough to run on a standard CPU.

Methods

The researchers utilized a Knowledge Distillation framework, treating a pre-trained VGG-M as a "teacher" and a much smaller network as the "student." Their approach is unique because it doesn't just compress the model; it changes its fundamental objective.

  1. Joint Compression and Transfer: They pruned the teacher network by a factor of 8 in both filter numbers and channels, resulting in a 64x reduction in parameters.
  2. Dual Loss Functions:
    • Fidelity Loss: This ensures the student network "mimics" the teacher's ability to represent high-level semantic features.
  3. Correlation Tracking Loss: This is the "transfer" part. It adapts the student's objective from classification (identifying what an object is) to regression/tracking (locating where a specific instance is).
  4. Multi-level Distillation: Instead of just looking at the final layer, they distilled knowledge across low, middle, and high-level convolutional layers to capture a richer variety of features.
  5. Background-Aware Online Adaptation: To handle the specific appearance of a target in a new video, they implemented a fast online fine-tuning step in the first frame. This helps the network distinguish the target from "hard negatives" in the immediate background.

Datasets

The model was distilled offline using the ImageNet ILSVRC 2015 dataset.

For evaluation, the authors tested their "fast" versions of trackers (fECO and fDeepSTRCF) against five major benchmarks:

  • OTB-2013 & OTB-2015: The standard gold-standard benchmarks for object tracking.
  • Temple Color (TC-128): Focused on color-based tracking.
  • VOT-2016 & VOT-2017: Challenges that focus on accuracy and robustness (resetting the tracker upon failure).

Key Insights and Ideas

  • Task-Specific Distillation: The "aha moment" here is that you shouldn't just compress a classification network to do classification better; you should compress it while teaching it a new task (tracking).
  • Semantic vs. Appearance Features: By using two losses, the student network learns to balance general semantic knowledge (from the teacher) with the specific appearance-matching skills needed for tracking.
  • CPU-First Design: Most "real-time" trackers are only real-time on a GPU. This paper specifically targets the single-core CPU bottleneck, which is a much higher bar for efficiency.
  • Pruning is just the start: Simply pruning a network leaves it "dumb." The joint training process is what allows the 1.5MB student to perform nearly as well as the 95MB teacher.

Key Results

  • Massive Compression: The student network (CF-VGG) is only 1.5 MB, compared to the 95 MB VGG-M teacher—a 63x reduction.
  • True Real-Time CPU Speed: On a single-core CPU, the improved fECO tracker reached 27 FPS (up from 5 FPS) and fDeepSTRCF reached 20 FPS (up from 3 FPS).
  • Competitive Accuracy: On OTB-2015, fDeepSTRCF achieved an AUC score of 68.6%, which is actually slightly higher than the original DeepSTRCF (68.5%) despite being much faster.
  • Efficiency: The student network requires only 0.048 Billion FLOPs, compared to the teacher's 1.82 Billion.

Limitations and Future Work

While the results are impressive, the authors noted some "Failure Cases." The compressed model occasionally struggles with:

  • Drastic Appearance Changes: If an object rotates or changes shape suddenly (e.g., the Ironman sequence), the lightweight features might not be robust enough.
  • Frequent Occlusions: In low-resolution videos with many overlapping objects (e.g., Freeman4), the student network can lose the target.

The authors suggest that future work could involve using more diverse training data and exploring even more efficient network architectures to further close the gap with massive GPU-bound models.

Summary updated

Abstract

Correlation filters (CF) have received considerable attention in visual tracking because of their computational efficiency. Leveraging deep features via off-the-shelf CNN models (e.g., VGG), CF trackers achieve state-of-the-art performance while consuming a large number of computing resources. This limits deep CF trackers to be deployed to many mobile platforms on which only a single-core CPU is available. In this paper, we propose to jointly compress and transfer off-the-shelf CNN models within a knowledge distillation framework. We formulate a CNN model pretrained from the image classification task as a teacher network, and distill this teacher network into a lightweight student network as the feature extractor to speed up CF trackers. In the distillation process, we propose a fidelity loss to enable the student network to maintain the representation capability of the teacher network. Meanwhile, we design a tracking loss to adapt the objective of the student network from object recognition to visual tracking. The distillation process is performed offline on multiple layers and adaptively updates the student network using a background-aware online learning scheme. The online adaptation stage exploits the background contents to improve the feature discrimination of the student network. Extensive experiments on six standard datasets demonstrate that the lightweight student network accelerates the speed of state-of-the-art deep CF trackers to real-time on a single-core CPU while maintaining almost the same tracking accuracy.
BibTeX
@article{wang2020real,
  author = {Ning Wang and Wengang Zhou and Yibing Song and Chao Ma and Houqiang Li},
  title = {{Real-Time Correlation Tracking Via Joint Model Compression and Transfer}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=7YDlm9H8hHP},
}

Loading related content...

Figure