AFAT: Adaptive Failure-Aware Tracker for Robust Visual Object Tracking

Tianyang Xu, Zhen-Hua Feng, Xiao-Jun Wu, Josef KittlerarXiv 2020 · OpenReview

TL;DR

This paper introduces AFAT, an adaptive visual object tracker that addresses the issue of failures in existing Siamese-based trackers by incorporating a novel Quality Prediction Network (QPN). AFAT leverages spatio-temporal information from response maps to predict potential tracking failures online, enabling adaptive correction and significantly improving robustness.

What Problem Does This Paper Address?

Visual object tracking aims to follow a target object through a video sequence. Siamese-based trackers have become very popular due to their impressive performance, learning to match a target template with candidate regions in subsequent frames. However, a key limitation of these trackers is their "one-shot" learning approach during the online tracking phase. This means they don't adapt to changing conditions or learn from their mistakes in real-time. Consequently, when the target's appearance changes drastically or it becomes occluded, these trackers can fail without any indication, leading to tracking shifts or complete loss of the target. The paper addresses this by proposing a system that can predict and react to these potential failures during the tracking process.

Methods

The core of AFAT is the Quality Prediction Network (QPN). This network is designed to predict the likelihood of a successful track. Here's how it works:

  • Input: Instead of raw image patches, QPN takes the response maps generated by a base Siamese tracker. Response maps are essentially heatmaps indicating where the tracker believes the target is located.
  • Spatial Feature Extraction: Convolutional Neural Networks (CNNs) are used to extract spatial features from these response maps. The paper mentions a preprocessing step where the response map is "circularly shifted and scatter the maximal peak into the four corners" to enhance context.
  • Temporal Feature Fusion: To capture the dynamics of tracking, Long Short-Term Memory (LSTM) networks are employed. These LSTMs process a sequence of response maps from recent frames, allowing the QPN to understand the temporal evolution of the tracking confidence.
  • Failure Prediction: The QPN, trained on labeled data, outputs a quality prediction (e.g., "success" or "lost"). This prediction acts as an "alarm" for potential tracking failures.
  • Adaptive Tracker (AFAT): The QPN is integrated with a state-of-the-art Siamese tracker (specifically, SiamRPN++). When the QPN detects a potential failure, AFAT can switch to a more robust (though potentially slower) model or trigger a correction mechanism. The paper uses two backbones for SiamRPN++: a faster mobilev2 for general tracking and a more powerful resnet50 for correction.

Datasets

  • Training: The QPN is trained on data generated from the TrackingNet dataset. Response maps and corresponding ground truth (used to derive IOU scores) from 20,000 videos were collected.
  • Evaluation: AFAT is evaluated on several standard benchmarks:
  • VOT2016, VOT2018, VOT2019
    • OTB2015
    • UAV123
    • LaSOT

Key Insights and Ideas

  • Response Maps as Rich Information: The paper highlights that response maps, often used for basic localization, contain rich spatio-temporal information that can be leveraged for failure prediction.
  • Spatio-Temporal Fusion is Crucial: Simply looking at the current frame's response map isn't enough. Combining information across multiple frames (temporal) and within the map itself (spatial) is key to accurately predicting tracking quality.
  • Online Failure Detection is Necessary: The "one-shot" nature of Siamese trackers is a bottleneck. Introducing an online mechanism to detect and flag potential failures is essential for robust tracking in challenging scenarios.
  • Adaptive Model Selection: By predicting failures, the tracker can intelligently switch between different models or apply corrective measures, balancing speed and accuracy.

Key Results

  • AFAT demonstrates outstanding and balanced performance in both accuracy and speed across various challenging datasets.
  • On VOT datasets, AFAT achieves superior EAO (Expected Average Overlap) and Accuracy scores compared to state-of-the-art trackers. For example, on VOT2018, AFAT achieved the best EAO score of 0.419 and Accuracy of 0.605.
  • The QPN's forward pass is very efficient, taking only 1.2-1.5 ms, making it feasible for real-time integration.
  • AFAT shows improved performance on datasets like UAV123 in terms of AUC, DP, and speed.
  • While AFAT sometimes detects more "failure alarms" on datasets like VOT (which are designed to be challenging), it also shows better performance on datasets like OTB2015 and UAV123, indicating its effectiveness in handling diverse tracking difficulties.

Limitations and Future Work

  • False Positives: The paper acknowledges that there can be false positive alarms from the QPN, although they aim for a conservative detection strategy.
  • Long-Term Robustness: On datasets like LaSOT, which have very long sequences, AFAT did not outperform SiamRPN++ in AUC, suggesting that further improvements in long-term robustness might be needed.
  • Future Work: The paper implicitly suggests that further refinement of the QPN and its integration with even more advanced Siamese trackers could lead to even better performance. The adaptive nature of AFAT also opens avenues for more sophisticated failure correction strategies.

Summary updated

Abstract

Siamese approaches have achieved promising performance in visual object tracking recently. The key to the success of Siamese trackers is to learn appearance-invariant feature embedding functions via pair-wise offline training on large-scale video datasets. However, the Siamese paradigm uses one-shot learning to model the online tracking task, which impedes online adaptation in the tracking process. Additionally, the uncertainty of an online tracking response is not measured, leading to the problem of ignoring potential failures. In this paper, we advocate online adaptation in the tracking stage. To this end, we propose a failure-aware system, realised by a Quality Prediction Network (QPN), based on convolutional and LSTM modules in the decision stage, enabling online reporting of potential tracking failures. Specifically, sequential response maps from previous successive frames as well as current frame are collected to predict the tracking confidence, realising spatio-temporal fusion in the decision level. In addition, we further provide an Adaptive Failure-Aware Tracker (AFAT) by combing the state-of-the-art Siamese trackers with our system. The experimental results obtained on standard benchmarking datasets demonstrate the effectiveness of the proposed failure-aware system and the merits of our AFAT tracker, with outstanding and balanced performance in both accuracy and speed.
BibTeX
@misc{xu2020afat,
  author = {Tianyang Xu and Zhen-Hua Feng and Xiao-Jun Wu and Josef Kittler},
  title = {{AFAT: Adaptive Failure-Aware Tracker for Robust Visual Object Tracking}},
  year = {2020},
  eprint = {2005.13708},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.13708},
}
Figure