Abstract
While discriminative correlation filters (DCF)-based trackers prevail in UAV tracking for their favorable efficiency, lightweight convolutional neural network (CNN)-based trackers using filter pruning have also demonstrated remarkable efficiency and precision. However, the use of pure vision transformer models (ViTs) for UAV tracking remains unexplored. In this paper, we propose an efficient ViT-based tracking framework, Aba-ViTrack, for UAV tracking. In our framework, feature learning and template-search coupling are integrated into an efficient one-stream ViT to avoid an extra heavy relation modeling module. The proposed Aba-ViT exploits an adaptive and background-aware token computation method to reduce inference time. This approach adaptively discards tokens based on learned halting probabilities, which a priori are higher for background tokens than target ones. Extensive experiments on six UAV tracking benchmarks demonstrate that the proposed Aba-ViTrack achieves state-of-the-art performance in UAV tracking.
Introduction
Visual tracking for Unmanned Aerial Vehicles (UAVs) requires a delicate balance between high precision and computational efficiency. Drones operate in resource-constrained environments with limited battery life and onboard processing power, yet they must navigate complex scenarios involving extreme viewing angles, motion blur, and fast-moving targets.
Historically, the field has been divided between Discriminative Correlation Filters (DCF), which offer high speed but struggle with appearance changes, and Convolutional Neural Networks (CNNs), which provide high accuracy at the cost of significant computational overhead. While Vision Transformers (ViTs) have set new benchmarks in general computer vision, their application to real-time UAV tracking has been hindered by their "token-heavy" nature. Processing every image patch through every layer is inefficient, especially when large portions of the search area consist of irrelevant background. Aba-ViTrack addresses this by introducing a "smart" ViT that adaptively allocates computation.
Aba-ViTrack Framework
The Aba-ViTrack framework introduces two primary innovations to make ViTs viable for real-time aerial tracking: a one-stream backbone and background-aware token halting.
One-Stream ViT Backbone
Unlike traditional Siamese trackers that process the target template and search area in separate branches before fusion, Aba-ViTrack utilizes a one-stream architecture. The template and search patches are concatenated and fed into the transformer simultaneously. This allows the model to perform feature extraction and relation modeling (matching the target to the scene) concurrently, eliminating the need for heavy cross-correlation or "neck" modules.
Background-Aware Token Halting (Aba-ViT)
The core of the efficiency gain comes from the Aba-ViT mechanism, which predicts a halting probability for every token at each layer. If a token is identified as background noise, the model "halts" its processing early to save computational cycles.
To train this mechanism, the authors use a modified Ponder Loss that specifically penalizes the model for spending excessive time on background tokens. By leveraging prior knowledge of the target's location, the model learns to "glance" at the background while "pondering" the target, effectively acting as a regularizer that prevents the tracker from being distracted by noise.
Experimental Results
Aba-ViTrack was evaluated on six rigorous UAV-specific benchmarks, including UAV123, DTB70, and VisDrone2018, after training on large-scale datasets like GOT-10k and LaSOT.
Performance and Speed
On the UAV123 benchmark, Aba-ViTrack achieved a state-of-the-art precision of 0.864, outperforming previous CNN-based leaders like TCTrack. In terms of speed, the model reaches approximately 180 FPS on a desktop GPU (NVIDIA TitanX). Crucially for UAV applications, it maintains a real-time speed of 35.6 FPS on an NVIDIA Jetson TX2, an embedded edge device commonly used in drone payloads.
Ablation Studies
Ablation experiments confirmed that the "Background-Aware" weighting is vital, providing a 1.8% precision boost over standard adaptive ViT models. This demonstrates that targeted computational savings on background regions are more effective than uniform token reduction.
Conclusion
Aba-ViTrack represents the first pure Vision Transformer framework optimized for real-time UAV tracking. By integrating feature learning and relation modeling into a single stream and employing an intelligent token-halting mechanism, it achieves a superior balance of speed and accuracy. Future work may involve refining the distributional priors for token exiting and extending this background-aware logic to other computer vision tasks like object detection.
Summary updated