Adaptive Hyper-Feature Fusion for Visual Tracking

Zhi Chen, Yongzhao Du, Jianhua Deng, Jiafu Zhuang, Peizhong LiuIEEE Access 2020 · OpenReview

Abstract

In this work, we propose a robust tracking algorithm based on a context-aware correlation filter framework. To improve the richness of the feature representation, we introduce a "hyper-feature" consisting of a linearly weighted mixture of hand-crafted features (such as HOG and color histograms) and hierarchical deep convolutional features from VGGNet. The final output response map is refined using a Gaussian constrained optimization (GCO) method to ensure the response follows a Gaussian distribution, thereby increasing robustness to target appearance variations. Furthermore, an efficient adaptive model updating method is proposed to suppress model noise by selectively updating based on tracking confidence. Extensive experimental results on well-known tracking benchmark datasets demonstrate that the proposed algorithm performs favorably against many state-of-the-art methods in terms of success rate, accuracy, and robustness.

Introduction

Visual tracking—the process of locating a specific object across a sequence of video frames—remains a fundamental yet challenging task in computer vision. Trackers must navigate a variety of "appearance variations," including illumination changes, partial occlusions, and target deformations.

Historically, tracking research has been divided into two primary approaches. Hand-crafted features, such as Histogram of Oriented Gradients (HOG), are computationally efficient and capture fine-grained textures but often fail when the object's appearance changes drastically. Conversely, deep features extracted from Convolutional Neural Networks (CNNs) offer high-level semantic robustness but frequently lack the spatial resolution necessary for precise localization. This paper bridges these two paradigms through an adaptive fusion strategy.

Proposed Methodology

The core of the proposed system is a multi-faceted approach to feature representation and model maintenance. The architecture is designed to balance semantic understanding with spatial precision while actively preventing "model drift."

1. Hyper-Feature Fusion

Instead of relying on a single feature type, the tracker utilizes a "hyper-feature" that adaptively blends hand-crafted features (HOG and Color Names) with deep features from hierarchical layers of VGGNet (specifically conv4 and conv5 layers). This fusion allows the tracker to leverage semantic information for robustness during complex scenes and texture information for pinpoint accuracy.

Overview of the proposed tracking framework featuring hyper-feature fusion and adaptive model updating.

2. Gaussian Constrained Optimization (GCO)

Correlation filters typically produce a "response map" where the peak indicates the target's predicted location. However, background clutter can introduce noise into this map. The authors apply GCO to regularize the output, forcing the response map to adhere to a clean Gaussian distribution. This optimization makes the tracker significantly less susceptible to being distracted by similar-looking objects in the background.

3. Adaptive Model Update Strategy

A common failure point in visual tracking is the continuous updating of the model even when the target is occluded or blurred. This causes the tracker to "learn" noise, leading to failure. The proposed method introduces a gatekeeper: the Peak-to-Sidelobe Ratio (PSR). By measuring the sharpness and confidence of the tracking peak, the system determines whether to update the model. If the PSR is too low, the system pauses updates, effectively "holding its breath" until the target reappears clearly.

Experimental Evaluation

The algorithm was rigorously tested against state-of-the-art (SOTA) methods, including ECO-HC, SRDCF, and MCPF, across four major benchmarks:

  • OTB-50 & OTB-100: The tracker achieved a Distance Precision (DP) of 86.6% and an Overlap Success (OS) rate of 64.2% on OTB-100.
  • UAV123: In aerial tracking scenarios involving fast motion and scale changes, the method reached a DP of 74.3%.
  • Temple Color-128: The tracker demonstrated superior performance in color-intensive sequences.

Attribute-based analysis revealed that the tracker's greatest strengths lie in handling "fast motion," "occlusion," and "out-of-view" scenarios, validating the effectiveness of the adaptive update logic.

Conclusion and Future Work

The proposed adaptive hyper-feature fusion tracker provides a robust solution to the problem of model drift and feature insufficiency. By combining hierarchical deep features with hand-crafted textures and employing a smart update strategy, it achieves competitive results on standard benchmarks.

While the tracker is highly resilient, challenges remain in environments with extreme background clutter (e.g., the Ironman sequence) and significant target rotation. Future research will focus on enhancing differentiation in cluttered scenes and optimizing the CNN feature extraction process to improve real-time performance on mobile hardware.

Summary updated

Abstract

In this work, we propose a robust tracking algorithm based on context-aware correlation filter framework. In order to improve the richness of the feature representation, the proposed hyper-feature which contains linearly weighted mixture of hand-crafted features (such as HOG, color histogram) and hierarchical deep convolutional features (such as VGGNet). The final output response map is optimized by the Gaussian constrained optimization method to control the response map follow the Gaussian distribution, which gain the robustness to target appearance variations. In addition, in terms of model update, an efficient adaptive model updating method is proposed to suppress the model noises significantly. Extensive experimental results on well-known tracking benchmark datasets to evaluate the proposed algorithm. Experimental results demonstrate that the proposed algorithm performs favorably against many state-of-the-art methods in terms of success rate, accuracy, and robustness.
BibTeX
@inproceedings{chen2020adaptive,
  author = {Zhi Chen and Yongzhao Du and Jianhua Deng and Jiafu Zhuang and Peizhong Liu},
  title = {{Adaptive Hyper-Feature Fusion for Visual Tracking}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=H9tpy4ayHd},
}
Figure