Siamese Object Tracking for Unmanned Aerial Vehicle: A Review and Comprehensive Analysis

Changhong Fu, Kunhan Lu, Guangze Zheng, Junjie Ye, Ziang Cao, Bowen Li

Abstract

Unmanned aerial vehicle (UAV)-based visual object tracking has enabled a wide range of applications in intelligent transportation systems. Siamese networks have emerged as a leading approach due to their balance of accuracy and speed. However, limited onboard computational resources and complex aerial environments pose significant challenges. This work presents a comprehensive review of state-of-the-art Siamese trackers, evaluates them on typical UAV onboard processors, and analyzes their performance in real-world scenarios, including low-illumination conditions.

Introduction: The Challenges of Aerial Tracking

Visual object tracking from a UAV platform is significantly more demanding than standard tracking tasks. Drones operate under a "triple threat" of constraints:

  1. Computational Scarcity: Onboard processors must be lightweight and power-efficient, unlike high-end server GPUs.
  2. Dynamic Environments: Rapid maneuvers and viewpoint changes lead to motion blur and scale variations.
  3. Environmental Factors: Aerial footage often suffers from low resolution and poor lighting, particularly during nighttime operations.

While Siamese networks are popular for their efficiency, their deployment on resource-constrained UAV hardware requires careful optimization to maintain real-time performance.

Methodology and Taxonomy

The Siamese tracking framework typically follows a dual-branch architecture consisting of a template branch and a search branch.

General pipeline of a Siamese network-based tracker for UAV applications.

As shown in the figure above, the process involves feature extraction from both branches, followed by feature refinement and cross-correlation. The resulting response maps are then processed through classification and regression heads to localize the target. The authors categorize the evolution of these trackers into a clear taxonomy based on their architectural innovations:

Taxonomy of leading-edge Siamese trackers categorized by feature modeling and target localization strategies.
  • Feature Modeling: This includes "Pioneers" like SiamFC, innovations in feature extraction (e.g., deeper backbones), and feature refinement techniques (e.g., attention mechanisms in SiamAttn).
  • Target Localization: Trackers are divided into Anchor-based methods (using predefined bounding boxes) and Anchor-free methods (directly regressing coordinates). Anchor-free models often prove more robust for the small, varying scales typical of UAV footage.
  • Extra Types: Specialized models like LightTrack (using Neural Architecture Search) or GlobalTrack for long-term tracking.

Comprehensive Benchmarking

To bridge the gap between theoretical performance and real-world deployment, 19 state-of-the-art trackers were integrated into a unified library and tested on an NVIDIA Jetson AGX Xavier, a standard UAV onboard processor.

Key Insights

  • The 30 FPS Barrier: For effective UAV control, a tracker must exceed 30 FPS. Many high-accuracy models fail this on embedded hardware.
  • Efficiency vs. Depth: While deep backbones (ResNet-50) dominate desktop benchmarks, shallower architectures (like AlexNet-based SiamRPN++ or SiamAPN) often perform better on UAVs by preserving resources for flight-critical tasks.
  • Anchor-Free Superiority: Models like SiamBAN and SiamAPN++ handle extreme scale variations better than rigid anchor-based systems.
  • The Nighttime Gap: Performance often drops by over 40% in low-light conditions, highlighting a major area for improvement.

Results and Performance

SiamAPN and SiamAPN++ emerged as the top performers for UAV deployment, offering the best trade-off between precision and speed. On the Jetson AGX Xavier, SiamAPN achieved approximately 35 FPS with a precision score of 0.773, while the AlexNet-based SiamRPN++ reached a blistering 52 FPS.

Future Directions

The study identifies several critical areas for future research:

  • Low-Light Robustness: Developing trackers that can handle noise and low contrast without heavy active sensors.
  • Latency Compensation: Integrating Predictive Visual Tracking (PVT) to account for computational lag.
  • Security: Addressing adversarial vulnerabilities where small image perturbations can cause tracking failure.
  • Automated Design: Utilizing Neural Architecture Search (NAS) to optimize models specifically for embedded hardware constraints.

The unified code library and evaluation results are available at https://github.com/vision4robotics/SiameseTracking4UAV.

Summary updated

Abstract

Unmanned aerial vehicle (UAV)-based visual object tracking has enabled a wide range of applications and attracted increasing attention in the field of intelligent transportation systems because of its versatility and effectiveness. As an emerging force in the revolutionary trend of deep learning, Siamese networks shine in UAV-based object tracking with their promising balance of accuracy, robustness, and speed. Thanks to the development of embedded processors and the gradual optimization of deep neural networks, Siamese trackers receive extensive research and realize preliminary combinations with UAVs. However, due to the UAV's limited onboard computational resources and the complex real-world circumstances, aerial tracking with Siamese networks still faces severe obstacles in many aspects. To further explore the deployment of Siamese networks in UAV-based tracking, this work presents a comprehensive review of leading-edge Siamese trackers, along with an exhaustive UAV-specific analysis based on the evaluation using a typical UAV onboard processor. Then, the onboard tests are conducted to validate the feasibility and efficacy of representative Siamese trackers in real-world UAV deployment. Furthermore, to better promote the development of the tracking community, this work analyzes the limitations of existing Siamese trackers and conducts additional experiments represented by low-illumination evaluations. In the end, prospects for the development of Siamese tracking for UAV-based intelligent transportation systems are deeply discussed. The unified framework of leading-edge Siamese trackers, i.e., code library, and the results of their experimental evaluations are available at https://github.com/vision4robotics/SiameseTracking4UAV .
BibTeX
@inproceedings{fu2023siamese,
  author = {Changhong Fu and Kunhan Lu and Guangze Zheng and Junjie Ye and Ziang Cao and Bowen Li},
  title = {{Siamese Object Tracking for Unmanned Aerial Vehicle: A Review and Comprehensive Analysis}},
  booktitle = {Artif. Intell. Rev},
  year = {2023},
  url = {https://openreview.net/forum?id=kKKHLcyde7},
}
Figure