Keyfilter-Aware Real-Time UAV Object Tracking

Yiming Li, Changhong Fu, Ziyuan Huang, Yinqiang Zhang, Jia Pan

TL;DR

This paper introduces "Keyfilter-Aware Object Tracking" (KAOT), a novel approach for real-time object tracking in Unmanned Aerial Vehicle (UAV) footage. KAOT addresses key limitations of existing methods, namely boundary effects and filter corruption, by employing a "keyfilter" mechanism inspired by SLAM techniques, leading to improved accuracy and efficiency.

What Problem Does This Paper Address?

Tracking objects in UAV footage presents unique challenges: the rapid motion, frequent appearance changes, occlusions, and the limited computational resources onboard UAVs. Traditional correlation filter (CF) based trackers, while efficient, suffer from two main issues:

  1. Boundary Effect: When training CFs, artificial "circulant" samples are used, which can lead to a loss of discriminative power due to the periodic nature of these samples.
  2. Filter Corruption: The learned filters can become corrupted by the constant appearance variations in the scene, especially in dynamic environments like those captured by UAVs.

Existing solutions, like enlarging the search area, can introduce background distractions, while frame-by-frame context learning, though helpful for reducing distractions, significantly slows down the tracking speed. This paper aims to develop a tracker that is both accurate and efficient enough for real-time UAV applications, overcoming these specific limitations.

Methods

The core innovation of KAOT is the introduction of the keyfilter concept, inspired by keyframe techniques in Simultaneous Localization and Mapping (SLAM). Here's how it works:

  • Keyframes and Keyfilters: Periodically, the tracker selects "keyframes" from the video stream. These keyframes are used to generate "keyfilters."
  • Keyfilter Restriction: These keyfilters learn the contextual information intermittently. During the training of the regular tracking filters for subsequent frames, the keyfilter acts as a constraint or "restriction." This means the regular filter's learning process is guided by the keyfilter, ensuring that context awareness is propagated and helping to prevent the regular filter from becoming corrupted by transient appearance changes.
  • Intermittent Context Learning: Instead of learning context from every single frame (which is computationally expensive and often redundant), KAOT learns context only from keyframes. This significantly reduces computational load while still capturing essential contextual information.
  • Optimization: The tracking problem is formulated as an optimization problem, and the authors utilize the Alternating Direction Method of Multipliers (ADMM) to efficiently solve it in the frequency domain.
  • Context Patches Scoring: A simple yet effective scheme is used to score context patches based on their Euclidean distance to the object, ensuring that closer patches have a stronger influence on penalization.

Datasets

The KAOT tracker was evaluated on two challenging UAV datasets:

  • DTB70: A dataset known for its frequent and drastic displacements.
  • UAV123@10fps: Another benchmark dataset for UAV tracking.

The paper does not introduce new datasets but uses these established benchmarks to compare its performance against state-of-the-art trackers.

Key Insights and Ideas

  • Keyfilters as Contextual Anchors: The idea of using periodically updated "keyfilters" to guide the learning of regular filters is a novel way to inject long-term contextual understanding without the overhead of per-frame context learning.
  • Mitigating Filter Corruption: By restricting the learning of the main filter with a more stable keyfilter, the tracker becomes more resilient to rapid appearance changes and noise, a common problem in UAV tracking.
  • Balancing Speed and Accuracy: The intermittent nature of keyfilter updates and context learning strikes a crucial balance, allowing for real-time performance without sacrificing significant accuracy.
  • SLAM Inspiration for Tracking: Borrowing the concept of keyframes from SLAM for visual tracking is an innovative cross-domain application that proves effective.

Key Results

  • Performance on Benchmarks: KAOT demonstrated competitive performance on both DTB70 and UAV123@10fps datasets, often ranking among the top trackers.
  • Speed: The tracker achieves a speed of approximately 15 FPS on a GPU, making it suitable for real-time UAV applications. It is noted to be significantly faster (e.g., 27% faster than ECO on DTB70) while maintaining high precision.
  • Attribute-Based Performance: KAOT showed particular strength in scenarios with background clutter, in-plane rotation, and deformation, outperforming other trackers in several of these challenging attributes.
  • Robustness to Motion Blur and Fast Motion: The tracker performed well in sequences with fast camera motion and motion blur, which are critical for UAV applications.

Limitations and Future Work

  • Keyframe Selection: The paper acknowledges that the current periodic keyframe selection mechanism is simple and could be improved. If keyframes are unreliable, it might introduce distractions. Future work could involve developing more adaptive strategies for selecting keyframes.
  • Re-detection and Rotation: The current KAOT framework has limitations in re-detecting objects that have disappeared for extended periods and in handling significant in-plane rotations. The authors suggest incorporating re-detection and rotation-aware modules to address these issues.
  • Speed Optimization: While the tracker is real-time, it was implemented in MATLAB and not fully optimized. Further speed improvements are expected with optimized code.

Summary updated

Abstract

Correlation filter-based tracking has been widely applied in unmanned aerial vehicle (UAV) with high efficiency. However, it has two imperfections, i.e., boundary effect and filter corruption. Several methods enlarging the search area can mitigate boundary effect, yet introducing undesired background distraction. Existing frame-by-frame context learning strategies for repressing background distraction nevertheless lower the tracking speed. Inspired by keyframe-based simultaneous localization and mapping, keyfilter is proposed in visual tracking for the first time, in order to handle the above issues efficiently and effectively. Keyfilters generated by periodically selected keyframes learn the context intermittently and are used to restrain the learning of filters, so that 1) context awareness can be transmitted to all the filters via keyfilter restriction, and 2) filter corruption can be repressed. Compared to the state-of-the-art results, our tracker performs better on two challenging benchmarks, with enough speed for UAV real-time applications.
BibTeX
@inproceedings{li2020keyfilter,
  author = {Yiming Li and Changhong Fu and Ziyuan Huang and Yinqiang Zhang and Jia Pan},
  title = {{Keyfilter-Aware Real-Time UAV Object Tracking}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  doi = {10.1109/icra40945.2020.9196943},
}
Figure