Abstract
Correlation filters (CF) have received considerable attention in visual tracking because of their computational efficiency. Leveraging deep features via off-the-shelf CNN models (e.g., VGG), CF trackers achieve state-of-the-art performance while consuming a large number of computing resources. This limits deep CF trackers to be deployed to many mobile platforms on which only a single-core CPU is available. In this paper, we propose to jointly compress and transfer off-the-shelf CNN models within a knowledge distillation framework. We formulate a CNN model pretrained from the image classification task as a teacher network, and distill this teacher network into a lightweight student network as the feature extractor to speed up CF trackers. In the distillation process, we propose a fidelity loss to enable the student network to maintain the representation capability of the teacher network. Meanwhile, we design a tracking loss to adapt the objective of the student network from object recognition to visual tracking. The distillation process is performed offline on multiple layers and adaptively updates the student network using a background-aware online learning scheme. Extensive experiments on five challenging datasets demonstrate that the lightweight student network accelerates the speed of state-of-the-art deep CF trackers to real-time on a single-core CPU while maintaining almost the same tracking accuracy.
Introduction
Deep learning has revolutionized visual object tracking, but the computational cost remains a significant barrier. While Correlation Filter (CF) trackers are inherently efficient, their modern "deep" variants usually rely on massive, off-the-shelf networks like VGG-M. This dependency tethers them to high-end GPUs, leaving mobile devices and single-core CPU systems unable to run state-of-the-art models in real-time. This paper presents a framework that bridges this gap by achieving real-time speeds on a single-core CPU without sacrificing tracking accuracy.
Methodology: Joint Compression and Transfer
The core of the proposed approach is a Knowledge Distillation framework. A pre-trained VGG-M network acts as a "teacher," which is distilled into a much smaller "student" network (CF-VGG) specifically optimized for tracking tasks.
Dual Loss Distillation
Unlike standard model compression, which focuses solely on maintaining classification performance, this framework employs two distinct loss functions during distillation:
- Fidelity Loss: This ensures the student network mimics the teacher's ability to represent high-level semantic features.
- Correlation Tracking Loss: This adapts the student's objective from general object recognition to instance-level visual tracking (regression).
Multi-level and Online Adaptation
The distillation process is performed across low, middle, and high-level convolutional layers to capture a rich variety of features. To handle the specific appearance of a target in a new video, the system includes a background-aware online adaptation step in the first frame, helping the network distinguish the target from immediate background distractors.
Experimental Results
The student network (CF-VGG) achieved a massive 63x reduction in model size, shrinking from the 95 MB VGG-M teacher to just 1.5 MB. This reduction is reflected in computational efficiency, with the student requiring only 0.048 Billion FLOPs compared to the teacher's 1.82 Billion.
Performance on CPU
On a single-core CPU, the improved trackers—fECO and fDeepSTRCF—reached real-time speeds of 27 FPS and 20 FPS, respectively. This is a significant leap from the original versions, which ran at 5 FPS and 3 FPS.
Accuracy Benchmarks
Despite the drastic compression, accuracy remained competitive. On the OTB-2015 benchmark, fDeepSTRCF achieved an AUC score of 68.6%, which is slightly higher than the original GPU-bound DeepSTRCF (68.5%). The framework was further validated on TC-128, VOT-2016, and VOT-2017, consistently demonstrating that the lightweight features maintain high robustness and precision.
Conclusion
By jointly compressing and transferring knowledge from heavy pre-trained models, this research enables high-performance deep tracking on resource-constrained CPU platforms. The dual-loss distillation approach successfully balances general semantic knowledge with the specific matching skills required for visual tracking, providing a viable path for deploying sophisticated AI on mobile and embedded devices.
Summary updated