Onboard Robust Visual Tracking for UAVs Using a Reliable Global-Local Object Model

Changhong Fu, Ran Duan, Doğan Kırcalı, Erdal KayacanSensors 2016 · DOI

TL;DR

This paper introduces a novel, robust, and real-time visual tracking algorithm for Unmanned Aerial Vehicles (UAVs). It tackles the challenges of tracking arbitrary 2D/3D objects in dynamic environments by employing a reliable global-local object model, an efficient geometric filter for outlier removal, and a heuristic local outlier factor for enhanced robustness.

What Problem Does This Paper Address?

Unmanned Aerial Vehicles (UAVs) are increasingly used for autonomous tasks like inspection and surveillance, which often require them to track moving objects in real-time. However, visual tracking for UAVs is a significant challenge due to various factors: object appearance changes (shape deformation, illumination variations, rotation), occlusions, and the inherent limitations of onboard processing power. Existing methods often struggle with robustness, efficiency, or the ability to adapt to these dynamic conditions, leading to tracking failures. This paper aims to develop a visual tracking system that is robust, accurate, and efficient enough to operate onboard a UAV in real-world scenarios.

Methods

The proposed algorithm is built upon three main modules:

  • Global Matching and Local Tracking (GMLT): This module initiates tracking by finding feature correspondences. It uses the FAST feature detector and an improved BRIEF descriptor for global matching, enabling re-detection of lost objects. For tracking between consecutive frames, it employs an iterative Lucas-Kanade optical flow algorithm. A forward-backward consistency evaluation is used to refine these local correspondences. The scale of the target object is estimated using a median of feature distance ratios, which is robust to outliers.
  • Local Geometric Filter (LGF): This module addresses the issue of outlier feature correspondences that can arise from the GMLT stage. It introduces a novel forward-backward pairwise dissimilarity measure and utilizes a hierarchical agglomerative clustering (HAC) approach with a single-link method. This helps maintain pairwise geometric consistency and filter out erroneous correspondences.
  • Local Outlier Factor (LOF) Module: This is presented as a novel contribution for visual tracking applications. It further refines the feature set by removing outliers that might have been missed by the LGF, particularly addressing issues like "chaining" (features stretching too far) and matching confusion caused by multiple object parts. The LOF module formulates outlier detection as a binary classification problem based on local density comparisons.

Datasets

The paper does not explicitly mention training datasets. Instead, it focuses on evaluating the proposed tracker using real-world UAV flight experiments. Six recorded image sequences were selected from various challenging environments at Nanyang Technological University, totaling 11,646 image frames. These sequences were manually labeled for ground truth. The challenging factors included mechanical vibration, aggressive flight, illumination variation, occlusion, scale variation, deformation, in-plane and out-of-plane rotation, out-of-view, and cluttered backgrounds.

Key Insights and Ideas

  • Global-Local Synergy: Combining global matching (for re-detection and initial correspondence) with local tracking (for fine-grained movement) is crucial for robust tracking, especially in dynamic UAV scenarios.
  • Outlier Filtering is Paramount: The paper emphasizes that simply finding features isn't enough; effectively filtering out incorrect correspondences is key to maintaining an accurate object model. The LGF and LOF modules are central to this.
  • LOF for Tracking is Novel: Applying the Local Outlier Factor (LOF) algorithm, typically used in data mining, to visual tracking for outlier feature removal is a novel contribution that helps resolve issues like feature chaining and matching confusion.
  • Real-World UAV Performance: The algorithm is designed and tested for onboard UAV applications, highlighting the importance of real-time performance and robustness against the specific challenges faced by aerial platforms.

Key Results

The proposed visual tracker demonstrated strong performance across various metrics:

  • Real-time Performance: Achieved frame rates of over 35 frames per second on an Intel i7 processor with a 640x512 image resolution.
  • Robustness and Accuracy: Outperformed several popular state-of-the-art trackers in terms of robustness, efficiency, and accuracy across extensive UAV flight tests.
  • Quantitative Comparison: In terms of average Center Location Error (CLE), the proposed tracker, KCF, and Frag were ranked 1st, 2nd, and 3rd, respectively. For average Frames Per Second (FPS), KCF, the proposed tracker, and CT achieved 149.8, 38.9, and 28.7 FPS, ranking 1st, 2nd, and 3rd. The average success rate (SR) was highest for the proposed tracker (95.9%).
  • Qualitative Results: Figures and tables show the tracker successfully handling challenging scenarios like scale variation, out-of-plane rotation, and partial occlusions, often outperforming other methods in difficult sequences.

Limitations and Future Work

The authors acknowledge two main limitations:

  • Strong Motion Blur: The tracker struggles with significant motion blur, as it can prevent the feature detector from reliably detecting enough features.
  • Large Out-of-Plane Rotation: Extreme out-of-plane rotations can also lead to imprecise bounding box estimations due to feature detection issues.

Future work could focus on addressing these limitations, potentially by incorporating more advanced feature descriptors or motion models that are more resilient to blur and extreme rotations. The paper also suggests that the UAV can achieve good control performance with more than 20 FPS in real flights, implying that the tracker's performance is sufficient for such applications. The authors also note that the tracker does not require external stabilization systems like gimbals.

Summary updated

Abstract

In this paper, we present a novel onboard robust visual algorithm for long-term arbitrary 2D and 3D object tracking using a reliable global-local object model for unmanned aerial vehicle (UAV) applications, e.g., autonomous tracking and chasing a moving target. The first main approach in this novel algorithm is the use of a global matching and local tracking approach. In other words, the algorithm initially finds feature correspondences in a way that an improved binary descriptor is developed for global feature matching and an iterative Lucas-Kanade optical flow algorithm is employed for local feature tracking. The second main module is the use of an efficient local geometric filter (LGF), which handles outlier feature correspondences based on a new forward-backward pairwise dissimilarity measure, thereby maintaining pairwise geometric consistency. In the proposed LGF module, a hierarchical agglomerative clustering, i.e., bottom-up aggregation, is applied using an effective single-link method. The third proposed module is a heuristic local outlier factor (to the best of our knowledge, it is utilized for the first time to deal with outlier features in a visual tracking application), which further maximizes the representation of the target object in which we formulate outlier feature detection as a binary classification problem with the output features of the LGF module. Extensive UAV flight experiments show that the proposed visual tracker achieves real-time frame rates of more than thirty-five frames per second on an i7 processor with 640 × 512 image resolution and outperforms the most popular state-of-the-art trackers favorably in terms of robustness, efficiency and accuracy.
BibTeX
@inproceedings{fu2016onboard,
  author = {Changhong Fu and Ran Duan and Doğan Kırcalı and Erdal Kayacan},
  title = {{Onboard Robust Visual Tracking for UAVs Using a Reliable Global-Local Object Model}},
  booktitle = {Sensors},
  year = {2016},
  doi = {10.3390/s16091406},
}
Figure