Video Tracking Using Learned Hierarchical Features

Li Wang, Ting Liu, Gang Wang, Kap Luk Chan, Qingxiong Yang

Abstract

Visual object tracking remains a challenging task due to non-rigid deformations, rotations, and lighting changes. This paper proposes a novel framework to learn hierarchical features specifically for tracking. By combining offline learning on auxiliary video sequences with a temporal slowness constraint and online domain adaptation for specific targets, the model achieves state-of-the-art performance. The learned features are robust to complex motion transformations and appearance shifts, significantly improving tracking accuracy and overlap rates.

Introduction

Visual object tracking is often hindered by the constant evolution of a target's appearance. Traditional methods rely on hand-crafted features like HOG or features learned from static image datasets. However, these fail to capture temporal invariance—the principle that an object’s features should transition smoothly over time. This paper addresses this gap by introducing a two-layer Convolutional Neural Network (CNN) that learns motion-robust features offline and adapts them to specific objects online.

Methodology: Offline Learning and Online Adaptation

The core of the proposed approach is a two-stage learning process designed to separate generic motion patterns from target-specific appearance details.

Offline Learning with Temporal Slowness

The network is first trained on auxiliary video sequences. To ensure the features are robust to motion, the authors employ a temporal slowness constraint. This constraint penalizes large fluctuations in feature representations between consecutive frames. Consequently, the first layer learns to handle simple translations, while the second layer captures more complex transformations such as non-rigid warping and out-of-plane rotations.

Online Domain Adaptation

When a tracking task begins, the generic features are tuned using a domain adaptation module. This module balances the preservation of robust pre-learned features with the incorporation of the specific target's appearance from the initial video frames. This adaptation occurs in both layers of the CNN.

Overview of the proposed framework showing offline feature learning and online adaptation for a specific object sequence.

As seen in the figure above, the adaptation process makes the filters visually more relevant to the target. For instance, in face tracking, the second-layer filters evolve to represent semantic, face-like parts.

Experimental Results

The method was evaluated on 15 challenging video sequences featuring complex motion and 8 standard benchmark sequences. The learned features were integrated into the Adaptive Structural Local sparse Appearance (ASLA) model.

Quantitative Performance

The proposed tracker significantly outperformed baselines. On the 15 challenging sequences, it achieved an average overlap rate of 63%, compared to just 36% for the raw-pixel baseline (ASLA_RAW).

Average overlap rates for various tracking methods across 15 challenging video sequences.

The table above highlights that the proposed method ("Ours") consistently achieves the highest overlap rates across nearly all test sequences. Furthermore, the average center location error was reduced to 12.2 pixels, a massive improvement over the HOG-based tracking error of 89.3 pixels.

Discussion and Conclusion

The primary innovation of this work is the successful separation of "how things move" (learned offline) from "what a specific object looks like" (adapted online). While the current implementation faces computational challenges—running at approximately 0.8 frames per second—the robustness of the hierarchical features is clear. Future work focusing on GPU acceleration and parallel computing could bring this high-accuracy approach closer to real-time performance.

Reference: Wang, L., Liu, T., Wang, G., Chan, K. L., & Yang, Q. (2015). Video Tracking Using Learned Hierarchical Features. arXiv:1511.07940v1.

Summary updated

Abstract

In this paper, we propose an approach to learn hierarchical features for visual object tracking. First, we offline learn features robust to diverse motion patterns from auxiliary video sequences. The hierarchical features are learned via a two-layer convolutional neural network. Embedding the temporal slowness constraint in the stacked architecture makes the learned features robust to complicated motion transformations, which is important for visual object tracking. Then, given a target video sequence, we propose a domain adaptation module to online adapt the pre-learned features according to the specific target object. The adaptation is conducted in both layers of the deep feature learning module so as to include appearance information of the specific target object. As a result, the learned hierarchical features can be robust to both complicated motion transformations and appearance changes of target objects. We integrate our feature learning algorithm into three tracking methods. Experimental results demonstrate that significant improvement can be achieved using our learned hierarchical features, especially on video sequences with complicated motion transformations.
BibTeX
@article{wang2015video,
  author = {Li Wang and Ting Liu and Gang Wang and Kap Luk Chan and Qingxiong Yang},
  title = {{Video Tracking Using Learned Hierarchical Features}},
  journal = {IEEE Transactions on Image Processing},
  year = {2015},
  doi = {10.1109/tip.2015.2403231},
}

Loading related content...

Figure