Spatio-Temporal Feature Aware Vision Transformers for Real-Time Unmanned Aerial Vehicle Tracking

Hao Zhang, Hengzhou Ye, Xiaoyu Guo, Xu Zhang, Yao Rong, Shuiwang LiDrones 2025 · DOI

TL;DR

This paper introduces NT-Track, a novel single-stream UAV tracking system that significantly improves real-time object tracking accuracy and robustness. By effectively integrating spatial neighborhood features and temporal information using a Transformer backbone, NT-Track achieves state-of-the-art performance, notably reaching 90% precision on the challenging VisDrone2018 dataset.

What Problem Does This Paper Address?

Unmanned Aerial Vehicles (UAVs) are increasingly used for applications like surveillance, agriculture, and disaster management, necessitating robust and efficient object tracking from an aerial perspective. However, UAV tracking presents unique challenges: extreme viewpoints, motion blur, and severe occlusions, which traditional trackers struggle with. Existing methods often fail to adequately capture the crucial spatio-temporal relationships between consecutive frames, leading to tracking drift and reduced accuracy, especially under dynamic conditions. Furthermore, many advanced deep learning trackers are computationally too expensive for real-time deployment on resource-constrained UAV platforms. This paper aims to bridge this gap by developing a highly accurate and efficient real-time UAV tracker that leverages both spatial context and temporal dynamics.

Methods

NT-Track is built upon a single-stream Vision Transformer (ViT) architecture. Its core innovations lie in two key modules:

  • Spatial Neighborhood Feature Extraction Module: This module enhances the capture of local contextual information by analyzing features from the target's immediate surroundings in consecutive frames. It effectively leverages the similarity of these surrounding areas to extract more robust features, especially when target movement is minimal.
  • Spatio-Temporal Fusion Module: This module integrates information from consecutive frames (t and t-1) using a transformer-based architecture with multi-head self-attention. This allows the model to learn spatio-temporal coherence, improving its ability to adapt to changes in target appearance, scale, and orientation, and thus enhancing robustness against occlusions and lighting variations.

The framework uses a unified approach to integrate these spatial and temporal features, feeding them into a prediction head for bounding box estimation. The training process employs a weighted focal loss for classification and L₁ and GIoU losses for bounding box regression, along with a background suppression loss to mitigate the influence of irrelevant background features.

Datasets

The paper evaluates NT-Track on four widely recognized aerial tracking benchmark datasets:

  • UAVDT: Designed for vehicle tracking under various conditions.
  • VisDrone2018: Focuses on object tracking on aerial platforms.
  • UAV123: A large-scale dataset with challenging sequences.
  • UAV123@10fps: A downsampled version of UAV123 to study frame rate effects.

For training, the authors utilized datasets from GOT-10k, LaSOT, COCO, and TrackingNet.

Key Insights and Ideas

  • Leveraging Neighborhood Similarity is Crucial: The paper highlights that simply focusing on the target itself is insufficient. Analyzing the similarity of the target's surrounding areas across frames provides vital contextual information for robust tracking.
  • Temporal Context is More Than Just Motion: While motion is important, NT-Track's spatio-temporal fusion goes beyond simple motion prediction by using attention mechanisms to learn complex temporal dependencies between frames.
  • Single-Stream Transformer for Efficiency: By adopting a single-stream ViT architecture, NT-Track achieves a good balance between computational efficiency and performance, making it suitable for real-time UAV applications.
  • Balancing Precision and Speed: The research demonstrates that it's possible to achieve both high precision and high frame rates, a critical requirement for practical UAV tracking.

Key Results

NT-Track demonstrates state-of-the-art performance across multiple benchmarks:

  • VisDrone2018: Achieved a groundbreaking precision of 90.0% and a success rate of 68.2%, outperforming other trackers by significant margins.
  • UAV123: Showcased the highest precision among all trackers.
  • Speed: Operates efficiently at approximately 112 FPS on a standard GPU, and even 132 FPS with TensorRT acceleration.
  • Comparison with Deep Trackers: NT-Track achieved the highest precision and success rate, while also being significantly faster than many competitive deep learning trackers. For instance, it was over 2 times faster than ROMTrack and over 10 times faster than SeqTrack.
  • Qualitative Results: NT-Track successfully tracked targets in challenging scenarios involving pose variations, background clutter, and scale changes, outperforming other state-of-the-art trackers.

Limitations and Future Work

  • Computational Cost of Temporal Fusion: While NT-Track is efficient, the paper notes that temporal fusion processes can still be computationally intensive. Future work could focus on further optimizing this aspect.
  • Generalizability to Extremely Diverse Scenarios: Although NT-Track performs well on challenging benchmarks, its performance in extremely diverse or unseen environments might require further investigation.
  • Future Directions: The authors suggest exploring additional approaches to leverage contextual information and further optimizing computational efficiency for temporal fusion processes.

Summary updated

Abstract

Driven by the rapid advancement of Unmanned Aerial Vehicle (UAV) technology, the field of UAV object tracking has witnessed significant progress. This study introduces an innovative single-stream UAV tracking architecture, dubbed NT-Track, which is dedicated to enhancing the efficiency and accuracy of real-time tracking tasks. Addressing the shortcomings of existing tracking systems in capturing temporal relationships between consecutive frames, NT-Track meticulously analyzes the positional changes in targets across frames and leverages the similarity of the surrounding areas to extract feature information. Furthermore, our method integrates spatial and temporal information seamlessly into a unified framework through the introduction of a temporal feature fusion technique, thereby bolstering the overall performance of the model. NT-Track also incorporates a spatial neighborhood feature extraction module, which focuses on identifying and extracting features within the neighborhood of the target in each frame, ensuring continuous focus on the target during inter-frame processing. By employing an improved Transformer backbone network, our approach effectively integrates spatio-temporal information, enhancing the accuracy and robustness of tracking. Our experimental results on several challenging benchmark datasets demonstrate that NT-Track surpasses existing lightweight and deep learning trackers in terms of precision and success rate. It is noteworthy that, on the VisDrone2018 benchmark, NT-Track achieved a precision rate of 90% for the first time, an accomplishment that not only showcases its exceptional performance in complex environments, but also confirms its potential and effectiveness in practical applications.
BibTeX
@inproceedings{zhang2025spatio,
  author = {Hao Zhang and Hengzhou Ye and Xiaoyu Guo and Xu Zhang and Yao Rong and Shuiwang Li},
  title = {{Spatio-Temporal Feature Aware Vision Transformers for Real-Time Unmanned Aerial Vehicle Tracking}},
  booktitle = {Drones},
  year = {2025},
  doi = {10.3390/drones9010068},
}
Figure