Learning Adaptive Discriminative Correlation Filters via Temporal Consistency Preserving Spatial Feature Selection for Robust Visual Object Tracking

Tianyang Xu, Zhen-Hua Feng, Xiao-Jun Wu, Josef Kittler

TL;DR

This paper introduces LADCF, a novel visual object tracking method that tackles the spatial boundary effect and temporal filter degradation issues in existing Discriminative Correlation Filter (DCF) trackers. By jointly learning adaptive spatial feature selection and enforcing temporal consistency, LADCF achieves more robust and accurate tracking, especially in challenging scenarios.

What Problem Does This Paper Address?

Visual object tracking is crucial for many applications, but real-world scenarios present significant challenges like illumination changes, background clutter, and object deformation. Discriminative Correlation Filter (DCF) based trackers have shown great success due to their efficiency and accuracy. However, they suffer from two main problems:

  1. Spatial Boundary Effect: The cyclic nature of DCF's correlation operations can lead to distortions at the image patch boundaries, impacting the quality of training data and thus the tracker's performance.
  2. Temporal Filter Degradation: Standard DCF methods often update filters based on single frames or with fixed rates, making them vulnerable to rapid appearance changes and leading to a degradation of the filter's effectiveness over time.

This paper aims to address these limitations by proposing a new DCF framework that integrates adaptive spatial feature selection with temporal consistency constraints.

Methods

The proposed LADCF method introduces several key innovations:

  • Adaptive Spatial Feature Selection: Instead of using all available features, LADCF employs a structured spatial sparsity constraint (approximated by L1-norm regularization, similar to Lasso) to adaptively select the most discriminative spatial features. This helps to preserve the underlying structure of the target's appearance manifold and reduces the impact of irrelevant or noisy features.
  • Temporal Consistency Preservation: To combat filter degradation, the method enforces temporal consistency by restricting the learned filter to remain close to its historical value. This is achieved through an L2-norm relaxation of the temporal consistency constraint, which encourages local changes in selected features while maintaining overall stability.
  • Joint Spatial-Temporal Optimization: A unified optimization framework, built upon the Augmented Lagrangian Method (ALM), is used to jointly learn the discriminative filters and select the temporal consistency-preserving spatial features. This allows for an integrated approach to address both spatial and temporal challenges.
  • Multi-channel Feature Integration: The framework is designed to effectively utilize multi-channel features (e.g., HOG, Color-Names, and deep CNN features) by considering them in a group structure, enabling joint sparsity across channels and spatial locations.

Datasets

The proposed LADCF method was evaluated on several widely-used visual object tracking benchmarks:

  • OTB2013
  • OTB50
  • OTB100
  • Temple-Colour
  • UAV123
  • VOT2018

The paper does not introduce any new datasets.

Key Insights and Ideas

  • Feature Selection is Key: The paper strongly emphasizes that not all features are equally useful for tracking. Adaptively selecting a small subset of discriminative spatial features significantly improves robustness and reduces computational load.
  • Temporal Consistency is Crucial for Long-Term Tracking: Simply updating filters frame-by-frame or with a fixed rate is insufficient for handling appearance variations. Enforcing temporal consistency helps maintain a stable and adaptive appearance model over time.
  • Joint Optimization Unlocks Synergies: By jointly optimizing spatial feature selection and temporal consistency, the method creates a more powerful and cohesive tracking system than addressing these issues in isolation.
  • Low-Dimensional Manifold Learning: The approach effectively learns a low-dimensional discriminative manifold for the target's appearance, which enhances its ability to generalize and adapt to changes.

Key Results

  • State-of-the-Art Performance: LADCF consistently outperformed existing state-of-the-art trackers on multiple challenging datasets, particularly on OTB100, achieving the best results in terms of precision (DP score) and success (AUC score) when using both hand-crafted and deep features.
  • Significant Feature Reduction: The method achieved superior performance by selecting only about 5% of hand-crafted features and 20% of deep features, demonstrating the efficiency of its feature selection mechanism.
  • Robustness Across Attributes: LADCF showed strong performance across various challenging attributes like low resolution, background clutter, out-of-view, and illumination variations, as evidenced by success plots on OTB100.
  • Competitive Speed: While not the primary focus, the method achieved competitive frame rates, making it suitable for real-time applications.

Limitations and Future Work

  • Parameter Sensitivity: While the paper analyzes parameter sensitivity, the performance can still be influenced by the choice of regularization parameters ($\lambda_1, \lambda_2$) and the learning rate ($\alpha$).
  • Long-Term Tracking: The paper notes that while LADCF performs well, trackers like ECO* are more sophisticated in dealing with very long-term tracking scenarios by collecting clusters of historical training samples. This suggests a potential avenue for future work to incorporate more advanced long-term memory mechanisms.
  • Feature Representation: The paper highlights the importance of feature representation and shows that combining different types of features (hand-crafted and deep) yields better results. Future work could explore more advanced feature fusion techniques.

Summary updated

Abstract

With efficient appearance learning models, discriminative correlation filter (DCF) has been proven to be very successful in recent video object tracking benchmarks and competitions. However, the existing DCF paradigm suffers from two major issues, i.e., spatial boundary effect and temporal filter degradation. To mitigate these challenges, we propose a new DCF-based tracking method. The key innovations of the proposed method include adaptive spatial feature selection and temporal consistent constraints, with which the new tracker enables joint spatial-temporal filter learning in a lower dimensional discriminative manifold. More specifically, we apply structured spatial sparsity constraints to multi-channel filters. Consequently, the process of learning spatial filters can be approximated by the lasso regularization. To encourage temporal consistency, the filter model is restricted to lie around its historical value and updated locally to preserve the global structure in the manifold. Last, a unified optimization framework is proposed to jointly select temporal consistency preserving spatial features and learn discriminative filters with the augmented Lagrangian method. Qualitative and quantitative evaluations have been conducted on a number of well-known benchmarking datasets such as OTB2013, OTB50, OTB100, Temple-Colour, UAV123, and VOT2018. The experimental results demonstrate the superiority of the proposed method over the state-of-the-art approaches.
BibTeX
@article{xu2019learning,
  author = {Tianyang Xu and Zhen-Hua Feng and Xiao-Jun Wu and Josef Kittler},
  title = {{Learning Adaptive Discriminative Correlation Filters via Temporal Consistency Preserving Spatial Feature Selection for Robust Visual Object Tracking}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=508EU2r0lF},
}
Figure