Robust and real-time deep tracking via multi-scale domain adaptation

Xinyu Wang, Hanxi Li, Yi Li, Fumin Shen, Fatih Porikli

Abstract

Visual tracking using deep learning often faces a trade-off between high accuracy and real-time performance. This paper introduces the Multi-Scale Domain Adaptation Tracker (MSDAT), which leverages channel reduction and a "lazy feed-forward" strategy to achieve state-of-the-art precision at approximately 35 FPS. By adapting pre-trained classification features for instance-specific tracking, MSDAT bridges the gap between general visual recognition and robust object following.

Introduction

Visual tracking—the task of following a specific object across a video sequence—has seen a massive performance boost thanks to Deep Convolutional Neural Networks (CNNs). However, this accuracy comes at a high computational cost. Most high-performing deep trackers are slow (often 1–10 FPS), making them impractical for real-world use in robotics or surveillance. Conversely, existing "real-time" trackers often sacrifice significant precision to maintain speed.

Furthermore, most deep trackers use features pre-trained for image classification (e.g., distinguishing a "car" from a "dog"). Tracking requires a different focus: distinguishing a specific instance of an object from its immediate background, even if the background contains other objects of the same category. This "domain gap" necessitates specialized adaptation techniques.

Methodology

The authors introduce MSDAT, which builds upon the Hierarchical Convolutional Features (HCF) framework but introduces several clever optimizations to achieve real-time efficiency without losing robustness.

Convolutional Channel Reduction and Multi-Scale Adaptation

The core of MSDAT is a specialized tracking branch "grafted" onto the feature layers of a pre-trained VGG-19 network. This branch consists of convolutional layers that reduce the number of feature channels by a factor of 8 (e.g., from 256 channels down to 32). This domain adaptation strips away redundant classification data and focuses on features essential for identifying individual objects. To handle objects of varying sizes, the adaptation layer uses multiple filter sizes (3x3 and 5x5), inspired by Inception architectures, which are concatenated to provide a richer representation.

The MSDAT architecture showing the integration of multi-scale adaptation layers with the VGG-19 backbone and KCF response.

Lazy Feed-Forward Strategy

In standard tracking, the CNN typically runs twice per frame: once for prediction and once for model updates. MSDAT employs a "Lazy Feed-Forward" strategy based on the observation that objects usually move very little between consecutive frames. By cropping a search window slightly larger than the target, the system can reuse feature maps for the update step if the object remains within that window, resulting in a nearly 50% speed gain.

KCF Integration

The reduced, multi-scale features are fed into Kernelized Correlation Filters (KCF). KCF is a highly efficient classical tracking method that benefits immensely from the lower dimensionality of the adapted features, as its computational complexity is heavily tied to the number of input channels.

Experimental Results

MSDAT was evaluated on the industry-standard OTB-50 and OTB-100 (Object Tracking Benchmark) datasets. The base VGG-19 network was pre-trained on the ILSVRC (ImageNet) dataset for general feature extraction.

Precision and Success plots on the OTB-100 benchmark comparing MSDAT (ours) against state-of-the-art trackers.

Performance and Speed

  • Speed: MSDAT runs at 34.8 FPS, meeting the threshold for real-time applications. This is roughly 3x faster than its prototype (HCF).
  • Accuracy: On the OTB-100 benchmark, MSDAT achieved a Distance Precision (DP) rate of 83.0% and an Area Under Curve (AUC) of 0.567.
  • Competitive Edge: While the "MD-Net" tracker is more accurate (90.9% DP), it runs at only 1 FPS. Compared to other real-time trackers like GOTURN (165 FPS but 56.39% DP) and SiamFC (58 FPS and 75.2% DP), MSDAT provides a superior balance of high precision and usable speed.

Conclusion

MSDAT successfully addresses the accuracy-speed dilemma in deep visual tracking. By utilizing channel reduction to bridge the domain gap between classification and tracking, and implementing a lazy feed-forward strategy to exploit temporal redundancy, the tracker achieves state-of-the-art results at real-time speeds. Future work will explore online updating of the neural network weights to further boost robustness without compromising the real-time efficiency established in this study.

Summary updated

Abstract

Visual tracking is a fundamental problem in computer vision. Recently, some deep-learning-based tracking algorithms have been achieving record-breaking performances. However, due to the high complexity of deep learning, most deep trackers suffer from low tracking speed, and thus are impractical in many real-world applications. Some new deep trackers with smaller network structure achieve high efficiency while at the cost of significant decrease on precision. In this paper, we propose to transfer the feature for image classification to the visual tracking domain via convolutional channel reductions. The channel reduction could be simply viewed as an additional convolutional layer with the specific task. It not only extracts useful information for object tracking but also significantly increases the tracking speed. To better accommodate the useful feature of the target in different scales, the adaptation filters are designed with different sizes. The yielded visual tracker is real-time and also illustrates the state-of-the-art accuracies in the experiment involving two well-adopted benchmarks with more than 100 test videos.
BibTeX
@inproceedings{wang2017robust,
  author = {Xinyu Wang and Hanxi Li and Yi Li and Fumin Shen and Fatih Porikli},
  title = {{Robust and real-time deep tracking via multi-scale domain adaptation}},
  booktitle = {ICME},
  year = {2017},
  doi = {10.1109/icme.2017.8019450},
}
Figure