Progressive Multistage Learning for Discriminative Tracking

Weichao Li, Xi Li, Omar El Farouk Bourahla, Fuxian Huang, Fei Wu, Wei Liu, Hongmin Liu

TL;DR

This paper introduces a novel "progressive multi-stage learning" approach for visual object tracking. It tackles the challenge of adapting to changing object appearances by intelligently selecting training samples in an easy-to-hard progression, guided by both the tracking error and detection confidence. This leads to more robust and accurate tracking, especially in complex scenarios.

What Problem Does This Paper Address?

Visual tracking aims to follow a specific object through a video sequence. A major hurdle is that objects can change their appearance due to factors like lighting, pose, and occlusion. To keep up, tracking algorithms often adapt their internal models using samples from the video. However, these samples are derived from the tracker's own predictions, which can be imperfect. If the tracker makes mistakes, it can learn from bad data, leading to a gradual drift away from the true object. This paper addresses the critical problem of how to effectively select and weigh these training samples to build a robust appearance model that can handle significant intra-class variations while maintaining clear distinction from the background.

Methods

The core of this paper is a "progressive multi-stage learning" strategy for discriminative tracking. Instead of learning from all available samples at once, it employs a staged approach:

  • Self-Paced Learning (SPL): This is the foundation, inspired by how humans learn. It involves gradually introducing more complex samples into the training process. The learning pace is controlled by a parameter ($\lambda$), which increases over stages.
  • Multi-Stage Optimization: The learning process is divided into multiple stages. At each stage, the tracker refines its appearance model and sample weights. The output of one stage serves as the initialization for the next, allowing for a progressive refinement.
  • Easy-to-Hard Sample Selection: The key idea is to start with the "easiest" samples (those with low tracking error) and gradually incorporate "harder" samples (those with higher errors or more complex variations). This is achieved by adjusting the learning pace ($\lambda$) across stages.
  • Time-Weighted Regularizer: To account for the temporal nature of tracking, samples from recent frames are given more importance. This is done by assigning higher weights ($p_k$) to more recent samples, causing older samples' influence to decay exponentially.
  • Detection Confidence Integration: Beyond just tracking error, the quality of the object detection itself is used to guide sample selection. Samples with sharper, more confident detection responses are favored. This is incorporated as an additional term in the objective function, controlled by a parameter ($\xi$).
  • Joint Optimization: The appearance model parameters ($\theta$) and sample weights ($v$) are optimized jointly in an alternating convex search manner.

Datasets

The paper evaluates its approach on several standard visual tracking benchmarks:

  • OTB-2015: A widely used dataset for evaluating tracking algorithms.
  • Temple Color: Another benchmark dataset, often used for color-based tracking.
  • VOT-2015, VOT-2016, VOT-2017: Datasets from the Visual Object Tracking challenge, known for their challenging sequences.

The paper does not introduce any new datasets.

Key Insights and Ideas

  • Progressive Learning is Key: The paper strongly suggests that a staged, easy-to-hard learning approach is more effective than a single-pass method for handling complex object appearance changes.
  • Beyond Tracking Error: Relying solely on tracking loss for sample selection can be insufficient. Incorporating detection confidence provides a crucial complementary signal for robust sample weighting.
  • Temporal Dynamics Matter: Giving more weight to recent samples is essential for adapting to the evolving nature of object appearance in a video sequence.
  • Multi-Stage vs. Single-Stage: The multi-stage approach allows for a more nuanced and robust learning process, effectively filtering out corrupted samples and gradually incorporating challenging ones.

Key Results

The proposed "Progressive Multi-Stage Learning" (SPL) tracker, especially with the time-weighted and detection-guided controller (SPL-DCI), demonstrates significant improvements:

  • OTB-2015: SPL-DCI achieved a relative gain of 5.8% in Precision Score (PS) and 6.4% in Area Under Curve (AUC) compared to the baseline SRDCF. It outperformed other variants like SPL and SPL-TKI.
  • Attribute-Based Performance: The tracker showed particular strength in challenging attributes like occlusion, low resolution, scale variation, deformation, and out-of-plane rotation.
  • Temple Color Dataset: The approach performed best among the evaluated state-of-the-art methods, achieving a mean AUC of 54.0%.
  • VOT Datasets: The tracker consistently outperformed the baseline SRDCF on VOT-2015, VOT-2016, and VOT-2017, showing gains of 2.1%, 2.5%, and 3.2% in EAO metric, respectively.
  • Speed: The method integrates efficiently with existing trackers like SRDCF, maintaining competitive tracking speeds (around 5 frames per second with model updates every 6 frames).

Limitations and Future Work

  • Parameter Sensitivity: The paper notes that the initial learning pace ($\lambda$) and scale ratio ($\mu$) are important and their optimal settings may depend on the specific video sequence.
  • Number of Stages: While the paper shows that increasing the number of learning stages ($N$) generally improves performance, it also highlights the trade-off with computational cost. The authors found $N=3$ to be a good balance.
  • Future Work: The paper suggests that the proposed framework is flexible and can be integrated with various discriminative tracking methods. Future work could explore more sophisticated ways to integrate temporal importance and detection confidence, or investigate different loss functions and regularization terms.

Summary updated

BibTeX
@article{li2022progressive,
  author = {Weichao Li and Xi Li and Omar El Farouk Bourahla and Fuxian Huang and Fei Wu and Wei Liu and Hongmin Liu},
  title = {{Progressive Multistage Learning for Discriminative Tracking}},
  journal = {IEEE Trans. Cybernetics},
  year = {2022},
  url = {https://openreview.net/forum?id=n6UR9WEIVTl},
}
Figure