Real-Time Correlation Tracking via Joint Model Compression and Transfer

Ning Wang, Wengang Zhou, Yibing Song, Chao Ma, Houqiang Li

Abstract

Correlation filters (CF) have received considerable attention in visual tracking because of their computational efficiency. Leveraging deep features via off-the-shelf CNN models (e.g., VGG), CF trackers achieve state-of-the-art performance while consuming a large number of computing resources. This limits deep CF trackers to be deployed to many mobile platforms on which only a single-core CPU is available. In this paper, we propose to jointly compress and transfer off-the-shelf CNN models within a knowledge distillation framework. We formulate a CNN model pretrained from the image classification task as a teacher network, and distill this teacher network into a lightweight student network as the feature extractor to speed up CF trackers. In the distillation process, we propose a fidelity loss to enable the student network to maintain the representation capability of the teacher network. Meanwhile, we design a tracking loss to adapt the objective of the student network from object recognition to visual tracking. The distillation process is performed offline on multiple layers and adaptively updates the student network using a background-aware online learning scheme. Extensive experiments on five challenging datasets demonstrate that the lightweight student network accelerates the speed of state-of-the-art deep CF trackers to real-time on a single-core CPU while maintaining almost the same tracking accuracy.

Introduction

Deep learning has revolutionized visual object tracking, but the computational cost remains a significant barrier. While Correlation Filter (CF) trackers are inherently efficient, their modern "deep" variants usually rely on massive, off-the-shelf networks like VGG-M. This dependency tethers them to high-end GPUs, leaving mobile devices and single-core CPU systems unable to run state-of-the-art models in real-time. This paper presents a framework that bridges this gap by achieving real-time speeds on a single-core CPU without sacrificing tracking accuracy.

Methodology: Joint Compression and Transfer

The core of the proposed approach is a Knowledge Distillation framework. A pre-trained VGG-M network acts as a "teacher," which is distilled into a much smaller "student" network (CF-VGG) specifically optimized for tracking tasks.

The offline learning process for the lightweight backbone and the online tracking pipeline.

Dual Loss Distillation

Unlike standard model compression, which focuses solely on maintaining classification performance, this framework employs two distinct loss functions during distillation:

  1. Fidelity Loss: This ensures the student network mimics the teacher's ability to represent high-level semantic features.
  2. Correlation Tracking Loss: This adapts the student's objective from general object recognition to instance-level visual tracking (regression).

Multi-level and Online Adaptation

The distillation process is performed across low, middle, and high-level convolutional layers to capture a rich variety of features. To handle the specific appearance of a target in a new video, the system includes a background-aware online adaptation step in the first frame, helping the network distinguish the target from immediate background distractors.

Experimental Results

The student network (CF-VGG) achieved a massive 63x reduction in model size, shrinking from the 95 MB VGG-M teacher to just 1.5 MB. This reduction is reflected in computational efficiency, with the student requiring only 0.048 Billion FLOPs compared to the teacher's 1.82 Billion.

Comparison of the proposed fECO and fDeepSTRCF trackers against state-of-the-art real-time and non-realtime trackers.

Performance on CPU

On a single-core CPU, the improved trackers—fECO and fDeepSTRCF—reached real-time speeds of 27 FPS and 20 FPS, respectively. This is a significant leap from the original versions, which ran at 5 FPS and 3 FPS.

Accuracy Benchmarks

Despite the drastic compression, accuracy remained competitive. On the OTB-2015 benchmark, fDeepSTRCF achieved an AUC score of 68.6%, which is slightly higher than the original GPU-bound DeepSTRCF (68.5%). The framework was further validated on TC-128, VOT-2016, and VOT-2017, consistently demonstrating that the lightweight features maintain high robustness and precision.

Conclusion

By jointly compressing and transferring knowledge from heavy pre-trained models, this research enables high-performance deep tracking on resource-constrained CPU platforms. The dual-loss distillation approach successfully balances general semantic knowledge with the specific matching skills required for visual tracking, providing a viable path for deploying sophisticated AI on mobile and embedded devices.

Summary updated

Abstract

Correlation filters (CF) have received considerable attention in visual tracking because of their computational efficiency. Leveraging deep features via off-the-shelf CNN models (e.g., VGG), CF trackers achieve state-of-the-art performance while consuming a large number of computing resources. This limits deep CF trackers to be deployed to many mobile platforms on which only a single-core CPU is available. In this paper, we propose to jointly compress and transfer off-the-shelf CNN models within a knowledge distillation framework. We formulate a CNN model pretrained from the image classification task as a teacher network, and distill this teacher network into a lightweight student network as the feature extractor to speed up CF trackers. In the distillation process, we propose a fidelity loss to enable the student network to maintain the representation capability of the teacher network. Meanwhile, we design a tracking loss to adapt the objective of the student network from object recognition to visual tracking. The distillation process is performed offline on multiple layers and adaptively updates the student network using a background-aware online learning scheme. Extensive experiments on five challenging datasets demonstrate that the lightweight student network accelerates the speed of state-of-the-art deep CF trackers to real-time on a single-core CPU while maintaining almost the same tracking accuracy.
BibTeX
@article{wang2020real,
  author = {Ning Wang and Wengang Zhou and Yibing Song and Chao Ma and Houqiang Li},
  title = {{Real-Time Correlation Tracking via Joint Model Compression and Transfer}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=7YDlm9H8hHP},
}
Figure